Google Cloud Ships llm-d 1.0 With Cache-Aware Routing for TPU and GPU Fleets
Google team have released llm-d 1.0, the first stable version of its distributed inference stack. This open-source, Kubernetes-native layer turns vLLM or SGLang model servers into one coordinated pool. They released it at the PyTorch Conference North America. It targets self-hosted agent and RL workloads, where round-robin load balancing breaks down.
llm-d was founded in 2025 by Red Hat, IBM, Google Cloud, CoreWeave and NVIDIA. Red Hat, IBM and Google are the lead maintainers. The project counts more than 500 contributors from over 25 organizations. It is a CNCF Sandbox project, accepted on March 12, 2026, and is licensed under Apache 2.0.
Prefix-aware routing, KV cache offloading, prefill/decode disaggregation, wide expert parallelism and batch inference already shipped. Version 1.0 is about running them reliably in production, which IBM calls day 2 operations.
The llm-d router builds on the Kubernetes Gateway API Inference Extension. It tracks which replicas hold each cached prefix. It then weighs load and predicted latency before picking a replica. Agents call one endpoint, and each step lands where its context already lives.
When accelerator memory fills, KV cache moves to host memory, then SSD or Managed Lustre. It uses vLLM’s OffloadingConnector, Mooncake or SGLang’s HiCache for this. Prefill and decode run as separate pools, because prefill is compute-heavy and decode is memory-bound.
All figures below are self-reported by Google and llm-d adopters.
Adopters report similar patterns. Snap reports prefix cache hit rates of up to 75 to 80%. Zenteiq reports over 4x throughput, from 1.47 billion to 5.97 billion daily tokens per 1,000 TPU v6e chips. An IBM demo served 3,000 concurrent coding agents at 5 to 10 times below API pricing.
These are not strict rivals. CoreWeave runs llm-d alongside Dynamo, and NVIDIA contributes Dynamo components to llm-d.
On Google Cloud, GKE Inference Gateway runs the llm-d router as a managed service. It can also span clusters and regions with global failover. KEDA and HPA scale pools on queue depth and predicted latency. Kueue enforces per-model quotas. Verified Helm charts deploy TPU and GPU pools behind one endpoint.
Google team suggests a staged path. Start with prefix-aware routing in front of an existing vLLM deployment. Then add flow control, tiered KV offload and disaggregation. Published runs can be compared on Prism.
Check out the Google Cloud announcement, the IBM Research post, the GitHub repo and the GKE docs. All credit goes to the researcher of this project. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.
Asif Razzaq is the CEO of Marktechpost AI Media Inc.. As a visionary entrepreneur and engineer, Asif is committed to harnessing the potential of Artificial Intelligence for social good. His most recent endeavor is the launch of an Artificial Intelligence Media Platform, Marktechpost, which stands out for its in-depth coverage of machine learning and deep learning news that is both technically sound and easily understandable by a wide audience. The platform boasts of over 2 million monthly views, illustrating its popularity among audiences.
Datalab Releases OmniParseBench: An Open OCR Benchmark With 16,288 Pass/Fail Tests
The newsletter 70,000+ AI builders read. Model releases, tools and research, without the hype.
Practitioner-first AI/ML news and analysis, read by 1M+ developers and researchers every month.