Coder Eval

Overview

Coder Eval is an open-source testing framework for coding agents. It lets developers define sandboxed YAML test suites to verify that skills, MCP servers, and CLIs work correctly when used by an agent. It supports A/B experiments and CI gates for regression testing.

Best for

Best for
Teams building or maintaining coding agents and their tooling.

Use cases

• Validate MCP servers work correctly with coding agents
• Test CLI tools from an agent's perspective
• Gate CI on agent behavior to prevent regressions

Notes

Coder Eval is an open-source testing framework for coding agents. It lets developers define sandboxed YAML test suites to verify that skills, MCP servers, and CLIs work correctly when used by an agent. It supports A/B experiments and CI gates for regression testing.

145 stars on GitHub. Last updated 2026-10-01. Licensed Apache-2.0.

Use cases

• Validate MCP servers work correctly with coding agents
• Test CLI tools from an agent’s perspective
• Gate CI on agent behavior to prevent regressions

Pros

• Sandboxed tests isolate agent runs
• YAML-based suites are easy to version and review
• Integrates with CI for automated gates

Cons

• Community project with limited adoption (145 stars)
• Requires writing YAML test definitions
• May not cover all agent frameworks

Indexed from awesome-llmops and enriched against its public facts.