Something unsexy but important just happened in AI infrastructure. Google Research published TurboQuant, a compression algorithm that cuts the memory required to run large language models by a factor of six — with no measurable accuracy loss.

The research was published March 25, 2026 and will be formally presented at ICLR 2026 in Rio de Janeiro on April 25. The algorithm is already generating open-source implementations, and its practical implications for anyone running AI at scale are worth understanding before the conference puts it front and centre.

What TurboQuant Actually Does

Large language models have a memory problem. When you ask a model to process a long document or maintain an extended conversation, it builds something called a KV cache — a store of intermediate computations that lets the model refer back to earlier parts of the context without recomputing from scratch. The KV cache grows linearly with context length and is a primary driver of GPU memory consumption for inference workloads.

TurboQuant compresses this KV cache to 3 bits per value, down from the 16 or 32 bits typically used. The result is a 6x reduction in memory usage. On an NVIDIA H100 GPU, this also delivers an 8x speedup for attention logit computation compared to unquantized 32-bit keys.

The breakthrough is that the compression is essentially lossless. Previous quantization methods introduced noticeable accuracy degradation at extreme compression ratios. TurboQuant sidesteps this using two techniques working together.

PolarQuant applies a random rotation matrix to each key and value vector before quantization. The rotation does not change the mathematical content of the vectors but redistributes variance uniformly across all dimensions, making extreme compression viable without outlier values introducing errors.

QJL (Quantized Johnson-Lindenstrauss) handles the optimisation process, providing theoretical guarantees on distortion rates. Together the two methods produce compression that is data-oblivious — it does not require calibration data and works on any model that uses a standard KV cache architecture.

This has been validated on Gemma, Mistral, and Llama-3.1-8B-Instruct. In practice, that means any standard model — open-source or API-based — can benefit without architectural changes.

Why the Memory Number Matters

Memory is the binding constraint for large-scale AI inference. When you are running thousands of concurrent requests against a model, GPU memory determines how many simultaneous sessions you can maintain, how long each context can be, and ultimately what you pay per query.

A 6x reduction in KV cache memory does not translate to a 6x reduction in total compute cost — there are other components at play. But it meaningfully shifts the economics of several scenarios.

Long-context workloads: Applications that process entire codebases, legal documents, financial reports, or extended conversation histories become cheaper to run. The same hardware can handle longer contexts at lower cost.

Concurrent capacity: A given infrastructure footprint can handle more simultaneous requests when each one requires less KV cache memory. For enterprise deployments running AI at scale, this directly affects unit economics and the cost-per-interaction that determines whether a workflow is commercially viable.

Local deployment: On-premise and edge AI deployments often run under tighter memory constraints than cloud infrastructure. Compression techniques like TurboQuant make it more viable to run capable models locally — relevant for regulated industries or businesses with data residency requirements.

The ICLR 2026 Moment

The ICLR conference is where applied research from Google Research, DeepMind, Meta, and academic labs gets formally stress-tested by the research community. Algorithms that hold up at ICLR tend to get integrated into mainstream inference frameworks within 6 to 18 months.

The GitHub repositories already appearing with PyTorch and llama.cpp implementations of TurboQuant suggest the open-source community is not waiting for official framework support. The algorithm is data-oblivious and architecture-agnostic, which makes it straightforward to implement independently of a specific model or runtime.

What This Means for Business

The AI infrastructure story of 2026 is not just about which models are getting smarter. It is equally about which models are getting cheaper to run.

Every efficiency breakthrough — quantization, smarter batching, improved hardware — drives down the cost per token. That has a compounding effect on what AI-powered workflows cost to operate at scale. The cost projections businesses built 6 months ago for AI deployments may already be conservative. Inference costs have declined faster than most enterprise models anticipated, and TurboQuant is part of that trend, not an isolated event.

For businesses deciding between cloud and local AI deployment, compression research like TurboQuant gradually shifts the calculus toward local being viable for more use cases — though total cost of ownership still depends on workload patterns, team expertise, and operational overhead.

The deeper point is structural. Organisations building data foundations and AI workflows now are doing so in an environment where the underlying infrastructure costs will keep falling. The value of those foundations compounds as the cost of running them decreases. The competitive gap between businesses that have built this infrastructure and those that have not will widen even if the cost of building it stays the same — and the cost of building it is also falling.

---

If you want the playbook other teams are using with Claude and Codex right now, grab the free Working With Claude field guide. Download it here.