What Engineers Expected Versus What Hit the Invoice

In late 2025, the consensus on r/LocalLLaMA and a few HN threads was that API costs would keep falling fast. The model release cadence had compressed to roughly six weeks. Open weights were catching up to frontier closed models. A lot of developers were quietly pricing new products around a working assumption of $0.20 to $0.50 per million input tokens by mid 2026.

That assumption is roughly half right. The sticker price on input tokens did drop. What most teams did not model well is the shape of the bill once you ship to real users, with real prompts, real tool calls, and real context windows.

A senior engineer at a 40-person fintech put it bluntly in a long r/MachineLearning thread. Their prototype ran on a popular frontier model at about $80 a month for two engineers. The production version, with a customer support copilot, jumped to $14,000 in March. Same code, same model, real traffic.

That gap between prototype and production is the single most consistent story across the community signal I read this year. Vendor pricing pages show you the catalog rate. The bill you actually pay is downstream of that.

It is not just startups. Enterprise AI budgets are breaking at the same rate, with agentic systems consuming 5 to 30 times more tokens than the chatbot-era workloads companies originally budgeted for. The prototype-to-production gap is wider at scale, not smaller.

Where Costs Actually Land in Real Workloads

The honest breakdown from a dozen practitioner write-ups and a few YouTube deep dives goes like this. For a typical RAG or chat workload, input tokens make up roughly 60 to 80 percent of total token volume, but output tokens drive 50 to 70 percent of the dollar cost. Output is priced 3x to 5x higher than input across every major provider, and that ratio has barely moved in 18 months.

Concrete ranges I’ve seen reported, in production, not on pricing pages:

• Long context summarization, 100k token windows, with a frontier model: $0.40 to $0.90 per call depending on output length
• Code generation with multi-turn refactoring: $0.05 to $0.25 per accepted completion
• Customer support copilot answering 200 to 400 token replies: $0.01 to $0.04 per resolved ticket
• Document extraction pipelines at scale, 50k pages a day: $1,200 to $4,500 a month, with one team on the Hacker News “Ask HN” thread reporting a $9k spike after a schema change

What surprised a lot of teams, and showed up repeatedly in r/ClaudeAI and r/OpenAI threads, was how much cost was hiding in tool calls and agent loops. A single multi-step agent task can rack up 20k to 80k tokens of intermediate reasoning. If your prompt pricing assumes one round trip per user message, the math will lie to you by a factor of 5 to 20.

Caching helps a lot, and most providers now offer 50 to 90 percent discounts on cached input tokens. The friction is that cache hit rates in production are wildly variable. One team in a practitioner blog post reported a 78 percent hit rate after restructuring their system prompt into static blocks. Another team got 12 percent because their context changed on nearly every call. The cache is not a discount you can count on until you instrument it.

The Gotchas That Blow Up the Forecast

Several patterns came up often enough that they deserve a name.

First, the “context creep” pattern. You start with a 4k context. Three months later, you are stuffing retrieved documents, chat history, tool outputs, and instructions into a 60k to 128k context. Latency is fine, quality is better, and the per-call cost quietly multiplied by 10x. This was a recurring theme in the Simon Willison blog and in a few YC founder threads. Nobody plans for it, and it usually shows up when a feature starts working well.

Second, the “streaming tax.” A surprising number of engineers reported that their cost projections assumed non-streaming responses. When they switched to streaming for UX reasons, the same model produced noticeably longer outputs in some cases, because users would wait longer and the model would elaborate. Output tokens per turn went up 30 to 60 percent. The HN thread on “Why is our LLM bill 2x what we modeled” had at least four teams saying exactly this.

Third, the “function calling explosion.” If you build an agent with a dozen tools, every tool description eats input tokens on every single call. One well-circulated post from a 12-engineer analytics startup calculated that tool definitions were costing them $1,800 a month just to be in the prompt. The fix was dynamic tool selection, which is its own engineering project.

Fourth, the “evaluation tax.” A few teams doing serious evals on every PR reported that their CI pipeline alone was running $400 to $1,500 a month in API costs. Smaller teams often skip this, then ship regressions they cannot catch.

Fifth, vendor lock-in around batch and committed-use discounts. A few practitioners warned that the 30 to 50 percent discount you get from a 6 or 12 month commitment is real, but it makes switching costs painful. Several teams reported being stuck on a model they had outgrown because the price delta was too high to justify a migration.

Which Workloads Stay Manageably Cheap

Costs are not uniformly bad. A few workload categories continue to be genuinely affordable in 2026, and the community signal agrees on which ones.

Classification, routing, and small structured extraction tasks are now effectively commodity priced. A 1k token prompt with a 50 token output through a small or distilled model runs $0.0001 to $0.0005 per call. Teams running millions of these a month report totals in the low thousands of dollars. A 3-person team in Lisbon wrote a public post-mortem on this. Their entire moderation pipeline cost $84 a month.

Embedding and retrieval workloads, especially with small embedding models, are similarly cheap. The story there is that the cost shifted to vector database hosting, not API spend. Not a single practitioner I read complained about embedding costs.

Batch processing of non-urgent workloads is the second genuine bright spot. Batch APIs across the major providers give 50 percent discounts in exchange for async turnaround, often 24 hours. Document summarization, log analysis, and report generation are obvious fits. Several teams in the r/MachineLearning weekly thread called batch their single biggest cost lever.

And finally, prompt caching, where it works, is excellent. The pattern from the practitioner write-ups is to design your system prompt as static blocks, prefix your variable content, and aim for cache hit rates above 60 percent. Teams that did this reported 40 to 70 percent reductions in total API spend with no quality change. The gap is in the engineering discipline to actually do it.

Stack Composition and What to Pair It With

The pattern I keep seeing in the most efficient production stacks is a tiered routing layer. One engineer in a widely-shared GitHub repo described their setup as: cheap small model for classification and routing, mid-tier model for standard generation, frontier model reserved for the hardest 10 to 20 percent of calls. Their reported bill was roughly 35 percent of what it would have been on a single-model architecture.

If you’re at the point of building or re-architecting a stack, this is also where custom AI application development becomes worth evaluating — the routing and caching infrastructure that makes AI economics work is engineering work that compounds, and getting it right the first time is meaningfully cheaper than refactoring it later.

Common pairings that came up:

• A vector database like pgvector, Qdrant, or Turbopuffer for retrieval, paired with a small reranker
• A lightweight observability layer. Helicone, Langfuse, Portkey, and a few homegrown solutions all appeared in practitioner posts. The teams without observability were the ones most often surprised by their bill
• A caching layer at the application level. Redis with semantic similarity thresholds, or a simple exact-match cache for repeated prompts
• A small open-weight model running locally for the cheapest calls, especially anything involving PII or sensitive data

What I did not see recommended much, despite the hype, was running a frontier open-weight model on your own hardware as a cost play. A few teams tried it. The amortized GPU cost, plus the engineering hours to operate it, made it more expensive than the API once you factored in utilization. The math works for very high volume, north of 50 million tokens a day, or for compliance reasons. For most teams, the API is still cheaper.

Who This Fits Best, and Where It Breaks

Based on the pattern across team sizes, the sweet spot is a 5 to 50 person team building a B2B product where LLM cost is a real but not existential line item. A typical monthly bill in this range, with proper observability and routing, lands between $1,500 and $12,000. The teams that struggled most were either very small, under 3 people, where every dollar of API spend was unbudgeted, or very large, above 200 engineers, where the lack of a centralized platform team led to runaway usage and shadow AI spend.

Solo developers and very small teams should treat AI API spend the way they treat cloud spend. Cap it, alert on it, and use the cheapest credible model for the job. Several YouTube creators with public build logs have been vocal about this. The default of “use the best model” is a luxury for prototypes.

Larger teams need a platform team or at least one engineer dedicated to cost. Without central observability and routing, a 200-engineer org will burn $80k to $200k a month in fragmented, untracked usage. The HN thread on enterprise LLM spend had three separate companies reporting exactly this pattern.

The practical next step is the free Working With Claude field guide. Thirty-two pages covering the ecosystem, Claude Code, and how to govern a rollout properly. Get your copy.