OLMO-eval

Overview

OLMO-eval is a Python-based evaluation suite for large language models (LLMs). It provides standardized benchmarks and metrics to assess model performance across multiple tasks.

Best for

Best for
Researchers and developers evaluating OLMo or compatible LLMs with reproducible benchmarks

Use cases

• Running reproducible evaluations on LLMs using established benchmarks
• Comparing performance of different model versions or configurations
• Integrating evaluation pipelines into model training workflows

Notes

OLMO-eval is a Python-based evaluation suite for large language models (LLMs). It provides standardized benchmarks and metrics to assess model performance across multiple tasks.

379 stars on GitHub. Last updated 2025-07-11. Licensed Apache-2.0.

Use cases

• Running reproducible evaluations on LLMs using established benchmarks
• Comparing performance of different model versions or configurations
• Integrating evaluation pipelines into model training workflows

Pros

• Open-source and community-maintained under the Allen AI umbrella
• Simplifies running standard LLM evaluations with a single Python framework

Cons

• Small star count (379) indicates limited community adoption and support
• Primarily designed for OLMo models, may require adaptation for other architectures

Indexed from awesome-llm and enriched against its public facts.