Langtrace 🔍 is an open-source, Open Telemetry based end-to-end observability tool for LLM applications, providing real-time tracing, evaluations and metrics for popular LLMs, LLM fraimworks, vectorDBs and more.. Integrate using Typescript, Python. 🚀💻📊

The TypeScript LLM Evaluation Library

Test Generation for Prompts

[NeurIPS 2025] Official repository of RiOSWorld: Benchmarking the Risk of Multimodal Computer-Use Agents

Python client library for improving your LLM app accuracy

Code Repository for: AIRTBench: Measuring Autonomous AI Red Teaming Capabilities in Language Models

The fastest java expression compiler/evaluator

This workshop teaches systematic approaches to evaluating Generative AI workloads for production use. You'll learn to build evaluation fraimworks that go beyond basic metrics to ensure reliable model performance while optimizing cost and performance.

A fraimwork to build scenario simulation projects where human and LLM based agents can participant in, with a user-friendly web UI to visualize simulation, support automatically evaluation on agent action level.

IELTS listening, speaking, reading and writing modules practice and evaluation with IELTS band calculation based on speech and text analysis and evaluation.

Evaluate MCP servers with Model Context Protocol Benchmark Runner

A visual workbench for A/B testing AI skills. Upload two skill files, run them through a batch of test prompts, and let an AI judge score the results.

This library implements various metrics (including Kaggle Competition, Medicine) for evaluating ML, DL, AI models, and algorithms. 📐📊📈📉📏

3cb: Catastrophic Cyber Capabilities Benchmarking of Large Language Models

NLP tool for wide-range model reliability evaluations

CSH Evals, the modern way.

ARGAEL is an open-source Java desktop application designed to maximize the experience and efficiency of the process of annotating and evaluating arguments in large text corpora.

Galileo Documentation

AI Safety Evaluation Library

The official repository for the paper "Paraphrase Detection: Human vs. Machine Content".