iris-eval/mcp-server

Overview

iris-eval/mcp-server provides a standardized evaluation framework for agents using the Model Context Protocol (MCP). It scores output quality, detects safety failures, and enforces cost budgets to help developers assess and control agent behavior.

Best for

Best for
Developers building and evaluating agents that use the Model Context Protocol

Use cases

• Benchmark agent output quality against defined criteria
• Automatically catch safety violations during agent execution
• Enforce per-call or cumulative cost limits to prevent budget overruns

How to use

Install

npx @iris-eval/mcp-server --demo

Tools exposed

• IRIS_TRANSPORT
• IRIS_HOST
• IRIS_PORT
• IRIS_HOME
• IRIS_DB_PATH
• IRIS_LOG_LEVEL
• IRIS_DASHBOARD
• IRIS_DASHBOARD_PORT
• IRIS_DASHBOARD_HOST
• IRIS_API_KEY
• IRIS_ALLOW_UNAUTHENTICATED
• IRIS_ALLOWED_ORIGINS
• IRIS_NO_AUTO_LAUNCH
• IRIS_ANTHROPIC_API_KEY
• IRIS_OPENAI_API_KEY
• IRIS_LLM_JUDGE_MAX_COST_USD_PER_EVAL
• IRIS_CITATION_ALLOW_FETCH
• IRIS_CITATION_DOMAINS
• IRIS_OTEL_ENDPOINT
• IRIS_OTEL_SERVICE_NAME

Tested with

Claude Desktop, Claude Code, Cursor, Windsurf, Cline, Continue, VS Code, ChatGPT

Example client config

{\n  "mcpServers": {\n    "iris-eval": {\n      "command": "npx",\n      "args": ["@iris-eval/mcp-server"]\n    }\n  }\n}

Notes

iris-eval/mcp-server provides a standardized evaluation framework for agents using the Model Context Protocol (MCP). It scores output quality, detects safety failures, and enforces cost budgets to help developers assess and control agent behavior.

6 stars on GitHub. Last updated 2026-05-25. Licensed MIT.

Use cases

• Benchmark agent output quality against defined criteria
• Automatically catch safety violations during agent execution
• Enforce per-call or cumulative cost limits to prevent budget overruns

Pros

• Offers a formal evaluation standard for MCP-based agents
• Combines quality, safety, and cost checks in one tool
• Written in TypeScript for type-safe integration

Cons

• Very low GitHub star count (6) suggests limited community adoption
• Tightly coupled to the MCP ecosystem, not useful outside it
• Requires agent infrastructure already built on MCP

Indexed from awesome-mcp-servers-punkpeye and enriched against its public facts.