We have some exciting news to share: NVIDIA Dynamo has officially hit v1.0, and we couldn’t be more thrilled. This is a huge milestone for the LLM inference ecosystem and for us at LMCache, it’s a moment worth celebrating.
What Is NVIDIA Dynamo, and Why Does It Matter?
If you haven’t been following Dynamo’s journey, here’s the short version: NVIDIA Dynamo is an open-source, high-throughput, low-latency inference serving framework purpose-built for deploying generative AI and reasoning models at scale. Unveiled at GTC 2025, NVIDIA Dynamo is designed from the ground up for the era of massive distributed GPU fleets, disaggregated serving, and reasoning models that generate tens of thousands of tokens per prompt. It does this through a set of elegant architectural choices: disaggregating prefill and decode phases onto separate GPUs, dynamically scheduling GPU resources in real time, intelligently routing requests to avoid redundant KV cache recomputation, and accelerating data movement with its low-latency NIXL transfer library.
What really makes Dynamo special, and what made us at LMCache immediately feel at home, is its philosophy of composability.
Why Composability Is the Right Bet
Here’s the thing about the LLM inference landscape: it’s moving fast. New inference engines, new hardware, new optimization techniques, the field looks different every six months. In that environment, locking yourself into a monolithic, vertically integrated stack is a recipe for falling behind on technical debt.
Dynamo took the opposite approach. It’s designed to be component-neutral: you pick the inference engine, you pick the routing strategy, you pick the cache management tool, and you pick the storage backend. Dynamo supports all major frameworks including vLLM, SGLang, NVIDIA TensorRT-LLM, and its modular architecture means developers can swap components in and out without rebuilding the world.
This is exactly the philosophy LMCache was built on. We believe the KV cache layer should be a first-class, pluggable component, not an afterthought baked into a specific inference engine. When we saw Dynamo’s design, we knew we wanted to be deeply integrated into it. And we are.
How LMCache Plugs Into Dynamo At Every Level
Our integration with Dynamo isn’t a thin wrapper or a bolt-on. It goes deep, across three distinct layers of the stack:
π§ Inference Level: vLLM, SGLang, and TensorRT-LLM (coming soon)
LMCache is integrated directly at the inference engine level. We currently support vLLM and SGLang, both of which are first-class citizens in the Dynamo ecosystem. TensorRT-LLM support is on the way and will complete the full backend picture. This means that regardless of which inference engine you’re running inside Dynamo, LMCache can serve as your offloaded KV cache layer, reusing computed key-value pairs across requests, users, and even nodes, without requiring you to recompute expensive prefills. You can also, if you need to, reuse the same KVCache across inference engines, which can enable seamless upgrade scenarios or complex mixed use of specific engines when needed.
π Routing Level: Cache-Locality-Aware Request Routing
One of Dynamo’s killer features is that its KV-aware router, which we had first demonstrated in vLLMβs production stack, routes inference traffic to minimize redundant recomputation. To make this work well, the router needs to know about cache locality – which nodes have which KV blocks already computed.
LMCache integrates at this level by implementing the message protocol that lets Dynamo’s router understand cache state across the cluster. This means routing decisions can be made with full awareness of where KV data lives, dramatically reducing wasted compute. It’s the difference between a smart routing layer and a dumb load balancer.
πΎ Storage Level: Full NIXL and Plugin Interface Support
At the storage layer, LMCache has full support for NIXL, NVIDIA’s low-latency data transfer library that supports NVIDIA NVLink, RDMA-capable NICs, and NVIDIA GPU Direct Storage. NIXL is what makes high-speed KV cache movement between GPUs, CPUs, and external storage actually practical at datacenter scale.
LMCache’s storage plugin interface is fully wired into Dynamo. This means LMCache can participate in Dynamo’s tiered memory strategy, offloading KV cache from GPU memory to CPU RAM, local SSDs, or networked storage while handling the reuse, eviction, and retrieval logic that makes caching actually effective (not just fast storage of data that never gets read again). Partners like VAST Data and WEKA have already validated this stack in production-scale tests, achieving transformative results on TTFT and throughput.
What This Means for You
If you’re deploying LLMs in production and you’re thinking about your inference stack, the LMCache + Dynamo combination gives you something genuinely new: a fully composable, cache-aware inference platform that doesn’t force you to choose between performance and flexibility.
We’ve been delighted to see so many users already deploying their critical models using LMCache and Dynamo together. Whether it’s long-context workloads where KV reuse avoids expensive prefill recomputes, multi-turn conversational AI where persistent caching slashes latency, or high-concurrency deployments where every GPU cycle counts, the combination delivers.
What’s Next
With Dynamo 1.0 out the door as a stable, production-ready foundation, we’re excited to keep pushing the integration further. TensorRT LLM support at the inference level is coming up next. We’ll also continue to deepen the cache-locality signals available to Dynamo’s router, and to expand the range of storage backends and hardware configurations supported.
The, soon to be released, refactor of LMCache, a.k.a. LMCache Operator, which separates KV cache management from the inference engine process (LMCache has its own container), furthers this composability and component independence principle to offer more flexibility and speed optimization in this critical field.
The inference ecosystem is converging around a set of shared, composable primitives β and we think that’s great news for everyone building on top of it. Dynamo 1.0 is a landmark moment in that convergence.
Congratulations to the NVIDIA Dynamo team on shipping v1.0. We’re proud to integrate with the stack. π
Resources:
Leave a Reply