RAGeval
by Community
LLM and RAG evaluation library: multi-judge consensus with a disagreement alarm, bootstrap confidence intervals and OpenTelemetry tracing.
OSS
RAGeval
Added 16 Oct 2026
Overview
RAGeval is a Python library for evaluating LLM and RAG systems. It uses multiple judges to reach consensus, flags disagreements with an alarm, computes bootstrap confidence intervals, and integrates OpenTelemetry tracing.
Best for
Best for
Developers building RAG pipelines who want statistically grounded, traceable evaluation.
Use cases
- Evaluate RAG pipeline responses with multi-judge consensus
- Detect judge disagreement to flag low-confidence evaluations
- Trace evaluation runs via OpenTelemetry for observability
Notes
0 stars on GitHub. Last updated 2026-10-14. Licensed AGPL-3.0.
Pros
- Multi-judge consensus reduces reliance on a single judge
- Disagreement alarm surfaces uncertain evaluations
- Bootstrap confidence intervals quantify evaluation reliability
Cons
- Community project with no official support
- Python-only, not language-agnostic
- Multi-judge setup adds configuration overhead
Get the free Developer’s Field Guide
A 27-page field guide to the AI coding workflow with Claude. Claude Code, MCP servers, the prompt patterns that work, and what to delegate. Free.
Enter your work email. We send it straight over, plus a few short notes worth knowing. Unsubscribe any time.