Most teams I talk to picked their LLM stack based on a benchmark that disappeared the moment real users hit the endpoint. The “200ms time to first token” headline from a model card tends to become 1.2 to 3.5 seconds under production load, and the people finding this out first are not reading vendor blogs. They are reading the HN thread titled “Why does my GPT-4o call feel slow?” and the r/LocalLLaMA posts that quietly admit vLLM is “fast on my 3090, brutal on anything with a real prompt.”

This piece is a reaction to what the practitioner community has actually said about LLM latency in production, and what those experiences mean for teams trying to pick a stack that survives real traffic.

What Practitioners Expected vs What They Got

The default expectation in late 2025 came from model release notes. Frontier labs were posting charts with TTFT under 300ms, sustained throughput above 100 tokens per second per user, and p99 latencies that fit comfortably under one second for short prompts. Engineers planning around those numbers built systems that felt reasonable on a laptop demo and started to crumble on a Friday afternoon traffic spike.

A consistent theme across the r/MachineLearning and r/LocalLLaMA threads is that the published numbers are real, but they describe a specific configuration. That configuration usually means short prompts, warm caches, single-stream requests, and ideally a colocated endpoint. Move any one of those variables, and the curve shifts.

On HN, the recurring complaint is less about raw model speed and more about variance. One engineer described a customer-facing assistant where p50 TTFT sat at 380ms, but p99 climbed past 4 seconds, which is the number that actually drives support tickets. Several comments noted that the p50 looked great in Datadog until they added a latency histogram, at which point the long tail became impossible to ignore.

The gap between “fast model” and “fast system” is the first thing most teams learn the hard way. It is also where the rest of this article spends its time.

Where Latency Actually Delivers in Production

There are places where the modern LLM stack genuinely performs, and the community has been specific about which ones.

For short, cached prompts against major hosted models, sub-500ms TTFT is realistic. Several practitioners on the Latency.space Discord and in GitHub issue threads have shared traces where a well-warmed Anthropic or OpenAI endpoint returned the first token in 280 to 450ms for prompts under 500 tokens. The cost per 1k tokens in that range lands around 0.0008 to 0.003 for input and 0.0024 to 0.006 for output on the smaller models, which makes interactive chat workloads feasible at modest spend.

Self-hosted inference on a single H100 with vLLM or TGI can hit 80 to 150 tokens per second for a 7B to 13B parameter model with batching, and that is fast enough for most internal copilots. The r/LocalLLaMA community has detailed build logs showing 70B-class models producing usable streams on a dual H100 node, with TTFT in the 100 to 250ms range once KV cache is warm. Quantized models (Q4, Q5) trade a small quality hit for roughly 1.4x to 2x throughput, which most engineers describe as an easy win for non-customer-facing workloads.

Streaming genuinely changes perceived performance. Multiple practitioners on YouTube and in the Datasette blog have pointed out that humans read at roughly 20 to 25 tokens per second, so any generation above that rate feels instant even if total completion time is high. The mental model shift is that you are not optimizing for “time to finish,” you are optimizing for “time to first token plus a smooth stream.” Once teams internalize this, the math around model choice changes.

Edge-deployed small models, the 1B to 3B parameter class running on device or in a colocated container, can deliver 50 to 200ms TTFT for classification, routing, and short extraction. Engineers building hybrid pipelines (small model for intent, larger model for response) consistently report that the small-model tier is the most underrated latency win in their stack.

Where It Falls Short

The honest list is longer than the vendor case studies suggest.

Cold starts on serverless GPU endpoints remain brutal. r/LocalLLaMA has a running thread on cold-start times for Runpod, Modal, and Lambda Labs, and the consensus range is 8 to 30 seconds for first request on a freshly booted container. Several practitioners have described workarounds like ping-every-N-seconds, dedicated always-on nodes, and routing the first user to a “warming” path. None of these are clean, and the cost of keeping a node warm 24/7 erases much of the per-token savings that made the serverless option attractive in the first place.

p99 is where most production systems die, and it is almost never the model’s fault. Network jitter, queuing, garbage collection pauses, and request bursts pile up at the tail. A recurring HN observation is that teams will optimize their prompt down by 200 tokens, celebrate a 50ms p50 win, and then ship a feature that triples concurrent users and discovers their p99 has quadrupled. Latency budgets need to be set against p95 or p99, and most public benchmarks do not give you that.

Cost surprises are the second most common thread. Teams plan against input and output token pricing, then get hit by retry storms. When a slow first response triggers a client-side timeout and a retry, you pay for the original request and the duplicate, and your effective p99 doubles on the books. Several practitioners in the r/devops subreddit have shared postmortems where 30 to 50 percent of their monthly LLM spend was retries, not new traffic. The fix is usually exponential backoff with jitter and a hard circuit breaker, but it is a fix teams only learn to build after the bill arrives.

Long prompts punish latency harder than people expect. A 32k context window sounds great until you realize that prefill time scales roughly linearly with prompt length, and TTFT on a 20k-token prompt against a frontier model can easily hit 3 to 6 seconds. Several engineers in the Latency.space community have shared traces where shaving a prompt from 15k to 4k tokens cut TTFT from 2.1s to 480ms. The lesson, repeated across dozens of posts, is that prompt engineering is latency engineering.

Reliability gaps around specific providers show up in outage postmortems. The r/LocalLLaMA and HN communities track provider incidents closely, and the pattern is that even a 99.9 percent SLA translates to roughly 43 minutes of downtime per month, almost all of which clusters around peak hours. Teams running customer-facing chat on a single provider have learned this the hard way, and several have moved to multi-provider routing as a result.

Onboarding friction is real for self-hosted stacks. vLLM and TGI are fast once configured, but the configuration is not trivial. Engineers describe weeks of tuning tensor parallelism, max model length, GPU memory utilization, and batching parameters. One well-liked comment on a vLLM GitHub issue summarized it as “the README is honest, the defaults are not.” Expect 2 to 6 weeks of engineer time to get a self-hosted deployment to a stable, monitored state for a team that has not done it before.

Who It Fits Best

Latency-sensitive customer-facing chat is the obvious fit, and the budget math works for teams of roughly 10 to 200 engineers. Below 10 people, hosted endpoints with a thin wrapper are almost always the right call, because the operational overhead of self-hosting exceeds the savings. Above 200, the volume justifies a hybrid where a self-hosted open model handles the long tail of simple requests and a frontier API handles the hard ones.

Internal copilots and tooling fit well into the self-hosted camp, especially for teams with strict data residency rules. A 4 to 6 person platform team can stand up a vLLM or TGI cluster on 2 to 4 H100s, serve a 70B quantized model, and cover several hundred internal users with p95 TTFT under 600ms. The community has been pretty unanimous that this is the sweet spot for open-weights inference.

High-volume batch jobs, classification, extraction, and embeddings rarely need the latency discussion at all. A 1 to 3 second TTFT is fine when you are processing 10 million documents overnight, and the cost-per-1k-tokens math favors smaller models aggressively. Engineers in the ML ops subreddit often point out that teams waste engineering time optimizing a workload that would be perfectly fine with a 1B parameter model and a longer wall clock.

Teams that should probably look elsewhere include those with sub-1ms latency requirements (the network round trip alone is a non-starter), those with strict offline operation and no GPU access, and any team that has not yet instrumented their current latency. Without traces, none of the tuning advice matters.

What Teams Pair It With or Replace It With

The most common pairing in the community is a small router model in front of a larger frontier model. The router handles intent, classification, and simple responses at 50 to 200ms, and only escalates to the expensive model when needed. Engineers report that 60 to 80 percent of traffic is handled by the small model, which drops effective cost per 1k tokens by 3 to 5x and tail latency by similar margins.

Caching sits next to routing as a default. Exact-match and embedding-based caches for common prompts are the single highest-ROI change most teams make, and several practitioners have shared numbers where a well-tuned cache cut their LLM bill in half and p99 by 2 to 3x. Redis with vector search, GPTCache, and custom in-memory layers are the typical picks.

Speculative decoding has matured enough that teams running self-hosted inference are starting to adopt it. A small draft model generates tokens that a larger model verifies in parallel, which can lift throughput by 1.5x to 2.5x with no quality loss. The setup overhead is real, and the r/LocalLLaMA community has been the loudest about which combinations actually work.

The most common replacement story is leaving a fully managed endpoint for self-hosted open weights once monthly spend crosses roughly 5 figures. Engineers describe a threshold around 30k to 80k per month where the per-token math starts to favor an H100 cluster, and the latency often improves because you control the routing and batching. The reverse replacement (self-hosted back to managed) happens when a team underestimates the operational load, which is a story that shows up regularly in r/sysadmin.

The pragmatic stack that has emerged across these threads is a router plus cache plus one or two model tiers, with traces flowing into OpenTelemetry from day one. Everything else is optimization on top of that foundation.

Closing Thoughts

The pattern across the community is that LLM latency is not a single number, it is a distribution, and the distribution is shaped more by your routing, caching, and prompt design than by which frontier model you pick. Teams that win on perceived speed are usually the ones who spent time on TTFT and p99, not on choosing between two model cards that differ by 100ms in a synthetic benchmark.

Want the practical version of this? The free Working With Claude field guide covers the full Claude ecosystem, Claude Code, and how to roll it out across a real business. Download it here.