AI Pulse · Under the Radar

A new grading tool called mcpgrade just exposed a quiet problem: a third of widely used MCP servers fail basic usability checks, even though they technically follow the protocol. The issue isn’t whether they connect. It’s whether an AI agent can actually figure out what to do with them.

The tool audits schema and parameter documentation, not just compliance. When servers document their tools poorly, models accept tasks they shouldn’t handle about 50% of the time in live testing. Firecrawl, a popular scraping integration, saw tool-selection accuracy drop to 84% because of naming collisions. MongoDB, Notion, and Airtable’s official servers all scored poorly. These are not obscure hobby projects.

Why this matters to you

If you’re building agents that call external tools, or you’re thinking about it, this matters immediately. A model that picks the wrong tool half the time isn’t a minor bug. It’s a reliability problem that kills trust fast. You can’t scale something unreliable, and you can’t fix it if you don’t know it’s happening.

This is the kind of thing we build into an AI command centre: structured tool registries, schema validation, and usage logs that surface when a model is guessing instead of executing. If you’re running your own MCP servers, like we do with CodeGraph and RepoLens, an audit pass is worth doing now. The grading tool is public, and the discussion on Hacker News has practical detail.

The broader point: agent tooling is still immature. Even official integrations from big names can ship with documentation that confuses models. If you’re deploying agents in production, assume nothing works perfectly and test the failure modes yourself.