For years, business leaders and engineering teams have been comparing AI coding tools using the same public leaderboards. SWE-bench, the most cited benchmark, tests models on open-source GitHub repositories. The problem is that almost no enterprise runs on open-source GitHub code.

A new benchmark published September 13, 2026 by Specific Labs changes that. Real-SWE evaluates frontier AI models on private, real-world production codebases licensed directly from actual companies. The results are striking, and not in the way the marketing suggests.

What Real-SWE Actually Tests

The benchmark includes eight model and harness configurations tested across ten task types, producing 640 scored rollouts. Each task comes from a private production codebase that was licensed from a real company, with the kinds of problems their engineers actually work on.

That sounds like a small methodological tweak, but it creates a fundamentally different challenge for AI systems. On private enterprise code, models must:

• Navigate proprietary systems whose architecture and logic aren’t on the public internet
• Work with sparse or outdated documentation
• Handle business-critical consequences, like getting billing right or calculating tax correctly across multiple services
• Resolve issues that cut across several interconnected systems

These are the conditions your internal engineering team works under every day. They are also conditions that public benchmarks have never measured.

Who Leads, And What It Costs

Anthropic’s Fable 5.1 came out on top with a 38.8% resolution rate, ahead of OpenAI’s GPT-6 Astra at 33.8% and Google’s Gemini 3.8 Flash at 31.2%.

But resolution rate is only part of the story. The more revealing number is cost per resolved task. Fable 5.1 costs roughly $17.94 per successfully resolved issue. Google’s Flash model resolves fewer tasks overall but costs around $8 per fix.

That trade-off matters more than it might seem. A team running 500 AI-assisted engineering tasks per month is looking at a $9,000 difference per month between the top two options if they optimise for accuracy alone. If they optimise for cost, they give up roughly 20% of their resolution rate.

Neither answer is obviously correct. The right choice depends on what the task is, what failure costs, and how much human review happens afterward.

Why Public Benchmarks Miss This

Models trained and evaluated primarily on public data have essentially seen the answer keys. SWE-bench repositories are crawled, indexed, and in many cases present in training data. When a model scores 80% on a public benchmark, that number may reflect pattern matching as much as genuine problem-solving capability.

Private enterprise codebases remove that advantage entirely. The model cannot have seen this code before. There are no prior solutions indexed anywhere. The failure modes that show up on Real-SWE, including misreading business context, breaking downstream integrations, and producing syntactically correct but functionally wrong patches, are the exact failures that cost real money in production.

The benchmark also surfaces which failure modes appear most often per model. Fable 5.1 tends to fail by over-engineering changes. GPT-6 Astra tends to fail by missing context that crosses service boundaries. Flash tends to fail on tasks with implicit domain knowledge requirements. These are not interchangeable failure modes, and knowing which one applies to your codebase changes which model you should reach for.

What This Means for Business

If you are choosing or budgeting AI coding tools based on public benchmarks, you are making a decision with incomplete information. The gap between benchmark performance and production performance on your actual codebase is real, and it varies by model.

The practical implications:

Evaluate on your own code. The most reliable way to know how an AI coding tool will perform in your environment is to test it in your environment, on real tasks, with real consequences. Most enterprise AI vendors will now offer some form of proof-of-concept engagement. Hold them to it.

Think cost per outcome, not cost per token. Token pricing alone does not tell you what a workflow actually costs. A model that resolves twice as many tasks at twice the per-task price is neither better nor worse in the abstract. It depends on your use case.

Resolution rate is a floor, not a ceiling. Even the best model on Real-SWE resolves less than 40% of tasks autonomously. That means a human reviews and handles the rest. The design of the human handoff process, not just the model, determines how much value you actually capture.

Complexity matters. Real-SWE shows that performance gaps between models widen on complex, multi-service tasks. If your AI coding initiative is focused on routine automation, the model ranking may not matter much. If you are tackling core infrastructure or multi-system integrations, model choice becomes much more consequential.

The data from Specific Labs is one data point, and enterprise codebases vary enormously. But Real-SWE represents exactly the kind of grounded, context-aware evaluation that the field has been missing. For any organisation making significant investments in AI-assisted development, this benchmark is worth understanding before the next procurement decision.

---

Enterprise DNA helps business leaders evaluate and deploy AI tools that deliver measurable outcomes. Talk to our advisory team about building an AI strategy grounded in real performance data.