Benchmarks miss much real work

A strong benchmark score can tell you how a model performs on the tasks included in that test. It may tell you much less about the work your staff actually do. Mercor estimates that 90% of American working hours go to tasks with little or no coverage in existing AI benchmarks. In its initial index of 29 benchmarks, only 208 of 923 occupations had even one task represented by at least five benchmark prompts.

That matters if you’re deciding whether AI is ready for a process in your business. A model’s score on a benchmark isn’t proof it can handle your team’s customer emails, finance checks or internal handoffs. You still need to test it on the real tasks, with the constraints and exceptions your staff deal with. Mercor’s map also shows that benchmark coverage varies across occupations and tasks, so broad scores may leave important gaps.

Mercor says its findings are provisional. They’re based on an initial list of benchmarks, not a complete or perfectly representative sample, and the company plans to update the map as it adds more. The item drew 581 Hacker News points and 525 comments, but the practical takeaway is straightforward: use benchmark results as one signal, then check performance against your own work. Mercor’s benchmark map offers a way to see where current tests cover work, and where they don’t.