Coder Eval
by Community
Playwright for coding agents. Test that your skills, MCP servers, and CLIs actually work when an agent uses them — sandboxed YAML suites, A/B experiments, CI gates.
OSS
Coder Eval
Added 1 Oct 2026
Overview
Coder Eval is an open-source testing framework for coding agents. It lets developers define sandboxed YAML test suites to verify that skills, MCP servers, and CLIs work correctly when used by an agent. It supports A/B experiments and CI gates for regression testing.
Best for
Best for
Teams building or maintaining coding agents and their tooling.
Use cases
- Validate MCP servers work correctly with coding agents
- Test CLI tools from an agent's perspective
- Gate CI on agent behavior to prevent regressions
Notes
145 stars on GitHub. Last updated 2026-10-01. Licensed Apache-2.0.
Pros
- Sandboxed tests isolate agent runs
- YAML-based suites are easy to version and review
- Integrates with CI for automated gates
Cons
- Community project with limited adoption (145 stars)
- Requires writing YAML test definitions
- May not cover all agent frameworks
Pairs with
Other entries in the index that connect to this one. Click through to see the chain.
Crewai
Community
Framework for orchestrating role-playing, autonomous AI agents. By fostering collaborative intelligence, CrewAI empowers agents to work together seamlessly, tackling complex tasks.
OpenDevin
Community
🙌 OpenHands: AI-Driven Development
FastMCP
Various
🚀 The fast, Pythonic way to build MCP servers and clients.
Get the free Developer’s Field Guide
A 27-page field guide to the AI coding workflow with Claude. Claude Code, MCP servers, the prompt patterns that work, and what to delegate. Free.
Enter your work email. We send it straight over, plus a few short notes worth knowing. Unsubscribe any time.