lm-evaluation-harness

Tensors and Dynamic neural networks in Python with strong GPU acceleration

A high-throughput and memory-efficient inference and serving engine for LLMs

LLM inference in C/C++

Track, rank and evaluate open LLMs and chatbots

Official github repo for ACLUE, an evaluation benchmark focused on ancient Chinese language comprehension

Aligning Large Language Models with Human: A Survey

👨‍💻 An awesome and curated list of best code-LLM for research.

List of papers on hallucination detection in LLMs.

This paper list focuses on the theoretical and empirical analysis of language models, especially large language models (LLMs). The papers in this list investigate the learning beha

LLM hallucination paper list

A curated list of Human Preference Datasets for LLM fine-tuning, RLHF, and eval.

非线智能 NoneLinear - ReLE评测：中文AI大模型能力评测（持续更新）：目前已囊括374个大模型，覆盖chatgpt、gpt-5.4、谷歌gemini-3.1-pro、Claude-4.6、文心ERNIE-X1.1、ERNIE-5.0、qwen3.6-max、qwen3.6-plus、百川、讯飞星火、商汤senseChat等商用模型， 以及st

CompMix: A Benchmark for Heterogeneous Question Answering.

Emergent Abilities

2021-08

This paper explores a simple method for improving the zero-shot learning abilities of language models. We show that instruction tuning—finetuning language models on a collection

IInfiBench: Evaluating the Question-Answering Capabilities of Code LLMs

LawBench

LLMEval is a research series dedicated to building comprehensive, fair, and robust evaluation frameworks for large language models.

Meta Lingua: a lean, efficient, and easy-to-hack codebase to research LLMs.

Medical Multilingual Benchmark

Leaderboard for the MMToM-QA benchmark (Jin et al., ACL 2024).

Large language models have recently been shown to attain reasonable zero-shot generalization on a diverse set of tasks (Brown et al., 2020). It has been hypothesized that this is

NeurIPS Tutorial Foundational Robustness of Foundation Models

PubMedQA Homepage

GITHUB HUGGING FACE MODELSCOPE DISCORD 🚨 This model mainly supports English. We will release bilingual (English and Chinese) math models soon. Introduction Over the past year, w

Supercharge Your LLM Application Evaluations 🚀

Language models have achieved remarkable performance on a wide range of tasks that require natural language understanding. Nevertheless, state-of-the-art models have generally st

a Swedish language understanding benchmark that evaluates natural language processing (NLP) models on various tasks such as argumentation analysis, semantic similarity, and textual

TAT-DQA: A Document Visual Question Answering (VQA) Dataset, aiming to answer questions over visually-rich documents with a hybrid of Tabular and Textual Content in Finance

Breaking Common Sense: WHOOPS! A Vision-and-Language Benchmark of Synthetic and Compositional Images

Deriving Wisdom of the Crowd from LLM Benchmark Mixtures

2022-12

Recent AI-assistant agents, such as ChatGPT, predominantly rely on supervised fine-tuning (SFT) with human annotations and reinforcement learning from human feedback (RLHF) to al

How do large language models (LLMs) develop and evolve over the course of training? How do these patterns change as models scale? To answer these questions, we introduce \textit{

Flan 2022 Collection

Does Your Large Multimodal Model Achieve Human-like Mathematical Reasoning?

AlpacaEval Leaderboard

Beyond the Imitation Game collaborative benchmark for measuring and extrapolating the capabilities of language models

Benchmarking large language models' complex reasoning ability with chain-of-thought prompting

评测榜单旨在为大语言模型和多模态模型提供全面、客观且中立的得分与排名，同时提供多能力维度的评分参考，以便用户能够更全面地了解大模型的能力水平。

FELM: Benchmarking Factuality Evaluation of Large Language Models

🐢 Open-Source Evaluation & Testing library for LLM Agents

Holistic Evaluation of Language Models (HELM) is an open source Python framework created by the Center for Research on Foundation Models (CRFM) at Stanford for holistic, reproducib

Stanford

This repository contains code to quantitatively evaluate instruction-tuned models such as Alpaca and Flan-T5 on held-out tasks.

LawBench

Lighteval is your all-in-one toolkit for evaluating LLMs across multiple backends

LLMEval is a research series dedicated to building comprehensive, fair, and robust evaluation frameworks for large language models.

Leaderboard | M 3 CoT

a comprehensive benchmarking platform designed to evaluate large models' mathematical abilities across 20 fields and nearly 30,000 math problems.

Evaluation suite for LLMs

Evals is a framework for evaluating LLMs and LLM systems, and an open-source registry of benchmarks.

DreamBench++: A Human-Aligned Benchmark for Personalized Image Generation

OlympicArena: Benchmarking Multi-discipline Cognitive Reasoning for Superintelligent AI

Evaluating scientific problems

a benchmark platform designed for evaluating large language models (LLMs) on a range of tasks, particularly focusing on their performance in different aspects such as natural langu