AI Pulse · Under the Radar

Most people assume the AI tool wraps around the model do not matter much. This test suggests they matter a lot.

Composio took one model, DeepSeek V4 Pro, and ran it through five different harnesses, the software scaffolding that handles how an AI agent takes actions, checks its work, and retries when something fails. Same model, same 30 tasks, only the harness changed. The result: DeepSeek’s own harness cost $0.028 per successful task. Claude Code cost $0.074 per successful task. That’s about 2.6 times more expensive to get the same job done, with the same underlying model doing the thinking.

Why does this matter if you’re running a business and not a research lab? Because when you’re pricing out AI agents for real work, whether that’s customer support, data cleanup, or research tasks, the model gets most of the attention but the harness is quietly doing a lot of the cost math. Two vendors can offer “the same AI” and hand you wildly different bills, not because the intelligence differs but because one wraps it in leaner, more efficient scaffolding.

A few caveats worth saying plainly. This is one vendor’s test, on their own benchmark, with no independent replication yet. The methodology is disclosed and the model was held constant, which is good practice, but you should treat the exact numbers as a signal to investigate, not gospel.

The practical takeaway is simple: when you’re comparing AI agent tools, ask about the harness, not just the model. This is exactly the kind of tradeoff we track when we build out an AI command centre, since cost-per-success across tools is the kind of thing you want visibility on before you scale usage, not after. You can see the original thread here: the original report.