AI Pulse · Under the Radar

A small benchmark result points to a cost problem most companies aren’t measuring yet. The same AI model, Kimi K3, was run through nine different agent harnesses on identical software engineering tasks. The model and checkpoint stayed the same. What changed was the system around it, how the agent planned work, used tools, handled context, checked results, and retried when things went wrong.

The cost per completed task ranged from $1.05 with Exo, at a 53.3% pass rate, to $18.34 with Claude Code at a similar pass rate. That is a 17.5x swing for broadly comparable outcomes. The cheapest option was not also the highest-quality option, which matters too. This is an early, community-run benchmark, not a final verdict on any tool. But it is a useful signal.

For an owner building AI into real operations, don’t treat model selection as the whole decision. If several providers can give you capable models, the bigger commercial question becomes the orchestration layer. Which workflow decides when to call the model, what context it sees, which tools it can use, how it verifies work, and when it stops? Those choices can determine whether an automated process costs a dollar or twenty.

This is especially relevant when you’re moving from a few AI experiments to repeatable internal agents. The model may be interchangeable. The harness around it is where reliability, unit cost, and operational control are won or lost. This is the kind of thing we build into an AI command centre, with the workflow and oversight treated as core infrastructure, not an afterthought.