The llm-d open-source project released version 1.0, a milestone aimed at making large-language-model inference more efficient to deploy and scale.
On Oct. 20, 2026, contributors announced llm-d 1.0 at the PyTorch Conference North America, hardening the open-source, Kubernetes-native framework for production-scale AI inference. Founded in 2025 by IBM Research, Red Hat, Google Cloud, CoreWeave and NVIDIA, the project now has more than 500 contributors across 25-plus organizations and joined the Cloud Native Computing Foundation as a sandbox project in March 2026. Version 1.0 adds priority- and fairness-aware multi-tenant traffic control, router hot standby and rolling upgrades, coordinated management of prefill and decode replicas, plus session-aware routing for agent conversations. It works with vLLM and SGLang, across GPUs and Google TPUs, and supports prefix-cache routing, KV-cache offloading and distributed model serving. IBM Research reports a roughly 98% prefix-cache hit rate and about $0.11 per million tokens in one self-hosted deployment. In another demonstration, llm-d served an approximately 753-billion-parameter GLM 5.2 model at more than 6.