lm-evaluation-harness

A high-throughput and memory-efficient inference and serving engine for LLMs

Evals is a framework for evaluating LLMs and LLM systems, and an open-source registry of benchmarks.

Track, rank and evaluate open LLMs and chatbots

List of papers on hallucination detection in LLMs.

This paper list focuses on the theoretical and empirical analysis of language models, especially large language models (LLMs). The papers in this list investigate the learning beha

A curated list of Human Preference Datasets for LLM fine-tuning, RLHF, and eval.

Explore The Berkeley Function Calling Leaderboard (also called The Berkeley Tool Calling Leaderboard) to see the LLM

Beyond the Imitation Game collaborative benchmark for measuring and extrapolating the capabilities of language models

CompMix: A Benchmark for Heterogeneous Question Answering.

[ACL 2024] An Easy-to-use Knowledge Editing Framework for LLMs.

An open platform for training, serving, and evaluating large language models. Release repo for Vicuna and Chatbot Arena.

FELM: Benchmarking Factuality Evaluation of Large Language Models

IInfiBench: Evaluating the Question-Answering Capabilities of Code LLMs

2020-05

LawBench

LLMEval is a research series dedicated to building comprehensive, fair, and robust evaluation frameworks for large language models.

Leaderboard | M 3 CoT

a comprehensive benchmarking platform designed to evaluate large models' mathematical abilities across 20 fields and nearly 30,000 math problems.

A simple, performant and scalable Jax LLM!

Leaderboard for the MMToM-QA benchmark (Jin et al., ACL 2024).

NeurIPS Tutorial Foundational Robustness of Foundation Models

Evaluation suite for LLMs

PubMedQA Homepage

The hub for EleutherAI's work on interpretability and learning dynamics

Open weights language model from Google DeepMind, based on Griffin.

a Swedish language understanding benchmark that evaluates natural language processing (NLP) models on various tasks such as argumentation analysis, semantic similarity, and textual

Breaking Common Sense: WHOOPS! A Vision-and-Language Benchmark of Synthetic and Compositional Images

Official github repo for ACLUE, an evaluation benchmark focused on ancient Chinese language comprehension

AlpacaEval Leaderboard

Benchmarking large language models' complex reasoning ability with chain-of-thought prompting

非线智能 NoneLinear - ReLE评测：中文AI大模型能力评测（持续更新）：目前已囊括374个大模型，覆盖chatgpt、gpt-5.4、谷歌gemini-3.1-pro、Claude-4.6、文心ERNIE-X1.1、ERNIE-5.0、qwen3.6-max、qwen3.6-plus、百川、讯飞星火、商汤senseChat等商用模型， 以及st

Holistic Evaluation of Language Models (HELM) is an open source Python framework created by the Center for Research on Foundation Models (CRFM) at Stanford for holistic, reproducib

This repository contains code to quantitatively evaluate instruction-tuned models such as Alpaca and Flan-T5 on held-out tasks.

Lighteval is your all-in-one toolkit for evaluating LLMs across multiple backends

Evaluation suite for LLMs

Evals is a framework for evaluating LLMs and LLM systems, and an open-source registry of benchmarks.

Test your prompts, agents, and RAGs. Red teaming/pentesting/vulnerability scanning for AI. Compare performance of GPT, Claude, Gemini, DeepSeek, and more. Simple declarative config