The Setup Most Teams Start With

Almost every team I’ve talked to in the last eighteen months started the same way. They prototype against an API. OpenAI, Anthropic, or Google. They wire up a few prompts, ship a demo, and watch the cost line on the dashboard climb from a few dollars a day to a few hundred a week. Someone on the team posts a link to a Reddit thread about running Llama on a couple of H100s. The finance lead asks why we are paying a third party for something we could host. The CTO says they will look into it. That is the moment the self-hosting vs API decision actually starts.

What follows is rarely as clean as either camp makes it sound. Developers on r/LocalLLaMA tend to undersell the operational drag. Vendor documentation tends to undersell the gap between demo throughput and production throughput. The honest answer lives in the middle, and the middle is mostly about who is on your team and what your workload actually looks like.

What Practitioners Expected vs What They Got

The expected case for self-hosting is straightforward. Buy or rent a few GPUs, run an open weights model like Llama 3.1 70B or Qwen 2.5, and your marginal inference cost drops to a few cents per million tokens. The model is yours. The data never leaves your VPC. The unit economics are wildly better at scale.

The expected case for APIs is also straightforward. Zero ops, immediate access to the strongest models, and you only pay for what you use. Ship a product, validate demand, then decide whether to bring inference in-house.

The r/LocalLLaMA thread that gets cited the most lays out a very specific set of surprises. Practitioners who actually stood up self-hosted stacks consistently report four surprises. First, the GPU bill is not the GPU bill. Power, idle capacity, multi-AZ replication, and observability tooling all add up. Second, throughput in vLLM benchmarks does not survive contact with real traffic patterns. Long contexts, mixed prompt sizes, and bursty workloads crush the per-request numbers vendors publish. Third, model quality matters more than the benchmarks suggest. A 70B model that scores 88% on MMLU is not a substitute for a frontier API model on hard reasoning, and most teams discover this in the second week. Fourth, on-call rotation gets real. GPUs fail, drivers break, quantization choices cause silent regressions, and someone has to own that.

On the API side, the surprises are different but equally consistent. Practitioners in HN threads about cost optimization report that the real bill is not the chat model. It is the embedding model, the reranker, the eval calls, the retry traffic from rate limits, and the long-context completions nobody accounted for in the planning spreadsheet. One commenter on a thread from earlier this year described their bill going from $4,200 to $11,000 in a single month after they enabled tool use, because every tool call re-issued the full system prompt.

Where Self-Hosting Genuinely Delivers

The cases where self-hosting wins are real, and they are narrower than the open source community sometimes admits.

Latency is one of them. Practitioners running 7B to 13B models on a single A100 or H100 routinely report time to first token in the 40 to 90ms range, with sustained throughput above 200 tokens per second per request. That is faster than most public APIs on long-tail traffic. For interactive product surfaces where every 100ms matters, this gap is meaningful.

Predictable per-token cost is the second. Once your cluster is sized and the model is loaded, the marginal cost of one more request is close to zero. Teams running batch summarization, classification, or document extraction over millions of records describe this as the single biggest financial lever they have. One practitioner blog I read last quarter detailed a content moderation pipeline that cost roughly $0.18 per million tokens on a self-hosted 14B model against roughly $2.40 on the equivalent API tier. The math is not subtle.

Data residency is the third. If you are in healthcare, finance, defense, or any regulated vertical, self-hosting is not optional. Several practitioners on r/MachineLearning threads have said plainly that the question of self-hosting vs API was settled for them by their compliance team before engineering ever got a vote.

Where API Access Genuinely Delivers

The strongest models are not open weights. As of mid-2026, the frontier on hard reasoning, long-context retrieval, and code generation still belongs to API providers. Practitioners consistently report that a self-hosted 70B model is roughly two generations behind the best closed model on anything involving multi-step reasoning, tool use, or nuanced instruction following. If your product depends on that quality, the API is not a compromise, it is the only option.

Time to first request is the second API win. A team can be making API calls in twenty minutes. A self-hosted stack takes a minimum of three to six weeks to get into production shape, and that is for a team that has done it before. First-time teams regularly report two to four months. The HN thread from late last year where someone tallied the engineering hours for a “small” self-hosting project ended with an estimate of 1.4 FTE-months before they shipped anything stable. Multiply that by a fully loaded engineering salary and the “savings” question gets uncomfortable fast.

Capacity elasticity is the third. APIs absorb spiky traffic in a way a fixed GPU pool cannot. Practitioners running consumer-facing products consistently report that the worst day of the month for cost is the day they have a traffic spike, and on self-hosted infrastructure, the worst day of the month for cost is the day they had to provision a second cluster for a spike that did not materialize the next week.

The Cost Surprises Nobody Warns You About

The honest cost numbers from practitioners look roughly like this. A self-hosted 70B-class model on rented H100s runs about $1.50 to $2.80 per GPU hour depending on provider and commitment length. That translates to roughly $0.30 to $0.70 per million tokens for a well-tuned vLLM deployment at 60 to 70% utilization. A frontier API at the time of writing runs roughly $2.50 to $15 per million input tokens and $10 to $60 per million output tokens depending on the model tier.

On paper, the self-hosted number wins by a factor of 3 to 50x. In practice, practitioners report the following.

Utilization rarely stays above 60%. Off-peak traffic means you are paying for idle GPUs. Practitioners running internal tools consistently report utilization between 18 and 35%, which is where the API’s pay-per-token model starts to look better.

The hidden self-hosting line items include observability, a vector database, a model gateway, a model router, eval infrastructure, and at least one engineer with on-call rotation. Practitioner estimates I’ve seen put this overhead at $8,000 to $25,000 per month in fully loaded engineering cost, plus $1,500 to $4,000 in software and tooling. Below roughly 80 million tokens per month of production traffic, the API is usually cheaper when you account for all of this.

The hidden API line items include embedding costs, evaluation costs, retry traffic from rate limits, prompt caching overhead that does not always work as advertised, and the creeping realization that every new product feature needs a new API call. Several practitioners in cost-cutting threads have described their per-request token count doubling after a “small” refactor that added agentic tool use.

Reliability Gaps and Edge Cases

Self-hosting reliability gaps are operational. GPUs run out of memory on long contexts. Drivers drift after kernel updates. Quantization choices that worked at 4-bit break at 8-bit for certain prompts. A new model release means another week of benchmarking and integration. Practitioners running self-hosted inference in production report roughly 99.5 to 99.8% availability, which sounds high until you compare it to the 99.95% and above that major API providers publish. The 0.15 to 0.45 percentage point gap is the difference between one outage a quarter and one a year.

API reliability gaps are different. They are mostly around rate limits, regional outages, and silent model behavior changes. Practitioners consistently report that model behavior can shift between versions or even between quiet updates, which makes eval and regression testing an ongoing cost rather than a one-time project. The community signal on this is strong. Multiple YouTube comments under model review videos from the last six months describe products breaking after provider-side updates with no advance notice.

Who Each Option Actually Fits

Self-hosting fits teams with three characteristics. First, sustained production traffic above roughly 80 million tokens per month. Below that, the fixed cost overhead erases the unit economics advantage. Second, an existing platform or ML engineering team that can absorb GPU ops, even if reluctantly. Third, a workload that is latency-sensitive, data-residency-constrained, or both. Practitioner blogs and team retrospectives from companies in this profile are remarkably consistent about the value.

API access fits teams with three other characteristics. First, variable or unpredictable traffic. Second, product surfaces that depend on the strongest model available, particularly for code, reasoning, or agentic flows. Third, a team that would rather pay AWS than hire a second platform engineer. The HN wisdom on this is consistent. Use the API until the cost of the API exceeds the fully loaded cost of doing it yourself, and then have a very honest conversation about whether your team can actually do it.

The hybrid pattern is the most common one in mature teams, and it is the one practitioners recommend most often. A frontier API for the hardest 10 to 20% of traffic, a self-hosted open model for the remaining bulk, and a router that decides which call goes where based on difficulty scoring. This is the architecture that shows up in conference talks and engineering blog posts far more than the pure self-hosted or pure API stories. The tooling around it is still rough, but it is improving fast, with practitioners commonly pairing vLLM or TensorRT-LLM with OpenRouter, LiteLLM, or a custom router.

What Teams Pair With or Replace These With

On the self-hosted side, the practitioner consensus stack is vLLM or TensorRT-LLM for serving, with Kubernetes or a managed GPU cloud for orchestration, and LiteLLM or a custom gateway for routing. For smaller models, Ollama and LM Studio are common in dev environments but rarely survive to production. Practitioners replacing self-hosted stacks often cite TGI and Hugging Face’s inference endpoints as the comparison point.

On the API side, the consensus is to wrap calls behind an internal interface so swapping providers is a configuration change rather than a rewrite. Teams that have been burned by provider outages or behavior changes report this as the single most important piece of plumbing they put in place. A surprising number of practitioners also report moving significant workloads to the smaller, cheaper models in each provider’s lineup, including the open-weight models hosted on the same infrastructure as the frontier ones.

The honest summary from the community is that there is no winner. There is only the right tool for your traffic shape, your team, and your risk tolerance. The teams that have made the decision well are the ones that stopped reading vendor benchmarks and started running the numbers against their own production traces, with their own engineering cost loaded in, and on a realistic timeline.

The practical next step is the free Working With Claude field guide. Thirty-two pages covering the ecosystem, Claude Code, and how to govern a rollout properly. Get your copy.