
lmcache server, on the far side of the process boundary from the vLLM engines that use it. On Ascend NPUs, KV data crosses that boundary on one of two native paths.In November 2025, we announced that LMCache runs on Ascend NPUs (neural processing units). Back then, LMCache lived inside the inference engine process, coupled to the engine’s lifecycle. Today we’re taking the next step: LMCache multi-process (MP) mode is now supported on Ascend NPUs.
In MP mode, the KV cache belongs to a standalone lmcache server process, and vLLM attaches to it through the LMCacheMPConnector. We have validated this deployment on Ascend 910B and 910C systems (Atlas 800I A2 and A3) with mainstream models, including the Qwen series, DeepSeek-V4, and GLM-5.3.
NPU deployments default to lmcache_driven; just start the server and the engines.
Why a separate cache process?
If you read our June post, Understanding LMCache MP Mode Transfer Paths, you already know the basic design. When the KV cache lives inside the inference engine, it is tied to that engine’s lifetime: a worker restart or crash takes the warm cache with it. MP mode moves cache ownership into a dedicated daemon, which gives you three benefits:
- Cache that outlives the engine. vLLM workers can restart, reschedule, or crash without the accumulated KV cache being lost.
- One cache, many engines. Multiple vLLM instances share a single
lmcache server, so a prefix prefilled by one engine is a cache hit for all of them. - A lighter engine loop. Storing, retrieving, chunking, and eviction move into the server process, leaving only the connector’s necessary coordination on the engine side. Lookup and store run asynchronously wherever possible, keeping cache management, Python global interpreter lock (GIL) contention, CPU cost, and I/O overhead off the scheduling and decode path.
On the accelerator side, this all depends on one systems question: how do KV tensors physically cross the process boundary? That question is where the Ascend work focused.
Two transfer paths, both native
MP mode’s control plane is a lightweight remote procedure call (RPC) link, ZMQ by default, between each engine and the server. The data plane comes in two forms, and both are available on NPU:

lmcache_driven, the server holds an IPC handle to the worker’s NPU memory and moves the KV data itself; in engine_driven, the worker copies to and from a /dev/shm pool that both processes map, which is the server’s L1.lmcache_driven (default on NPU) | engine_driven | |
|---|---|---|
| Data path | The server pulls and pushes KV through NPU device IPC handles, moving data on the engine’s behalf | The vLLM workers gather and scatter the KV copies themselves, zero-copy from a named shared-memory pool |
| KV formats | Standard grouped-query and multi-head attention (GQA/MHA), plus multi-plane MLA/DSA layouts, including DeepSeek-V4’s packed four-plane (latent, rope, DSA indexer, scale) mixed-dtype format | Single-format GQA/MHA models (e.g. Qwen3); MLA/DSA and hybrid models use lmcache_driven |
In lmcache_driven mode, the worker exports its NPU memory as interprocess communication (IPC) handles. torch.npu implements the same interprocess event ABI that CUDA users know, including cross-process events that signal when a transfer is safe. The server then performs the gather and scatter with native multi-layer block-transfer kernels compiled for the NPU. The engineering payoff is visible in the format column: our June guide described extending DeepSeek-V4’s hybrid KV layout beyond the CUDA path as “an open opportunity.” On Ascend, that work is now done. DeepSeek-V4’s packed multi-head latent attention and DeepSeek sparse attention (MLA/DSA) planes move through the LMCache-driven path natively, with no host-side repacking or format downgrade.
In engine_driven mode, the server hands out chunks from a named /dev/shm pool that both processes map, so a store is a single copy straight into the shared L1 buffer. It is the right fit for Qwen3-class models where the workers can do their own gather/scatter efficiently.
A first outside the CUDA/HIP ecosystem. CUDA and HIP share one kernel stack. Ascend is the first accelerator outside that stack to run both MP transfer paths on a separately developed kernel stack whose sources are fully open — every Ascend C kernel LMCache ships for it is open source in LMCache-Ascend.
DeepSeek-V4 works with zero extra configuration. Opting into engine_driven takes a matching server flag (--supported-transfer-mode engine_driven) plus the engine-side lmcache.mp.mp_transfer_mode key; for the shared-memory (SHM) fast path, start the server with --shm-name <name> --no-l1-use-lazy as well.
Upstream first: how it’s built
A year ago, the LMCache-Ascend integration worked by patching LMCache at runtime. That approach was effective and fully API-compatible, but the Ascend logic lived entirely in the plugin. The MP-mode work moved the platform layer upstream while keeping hardware-specific kernels in the plugin. That split is the main structural change:
- The platform layer is upstream. NPU device backends (the device IPC wrapper, interprocess events, device ops, multi-plane KV format handling, and the automatic transfer-mode negotiation) are merged into the LMCache
devbranch (#5138, #5341, #5567). Ascend is fully supported in LMCache’s device abstraction rather than maintained as a downstream fork. - The plugin supplies what only hardware can. LMCache-Ascend contributes the Ascend C kernels (multi-layer block transfer for both plain K/V and packed multi-plane layouts).
For users, this means there is no fork to track and no waiting for a downstream port: MP-mode features built on the shared control plane and storage layers (new storage tiers, eviction policies, and API surface) are available on Ascend as soon as they land upstream. Hardware-specific data paths still ship in the plugin.
Three steps to get started
The minimal deployment is three steps. Inside your vLLM-Ascend container, install the two packages:
git clone https://github.com/LMCache/LMCache.gitcd LMCachepip install -v --no-build-isolation -e .cd ..git clone --recurse-submodules https://github.com/LMCache/LMCache-Ascend.gitcd LMCache-Ascendpip install -v --no-build-isolation -e .
Before you install LMCache-Ascend, check that its third_party/hcomm submodule matches your container’s CANN version. The lmcache_driven deployment guide linked below shows how.
Start the cache server:
lmcache server --chunk-size 128 --l1-size-gb 50 --eviction-policy LRU
Wait for LMCache zmq cache server is running on localhost:5555 before starting vLLM.
Then start vLLM with the connector:
vllm serve /path/to/Qwen3-32B \ --tensor-parallel-size 8 \ --no-enable-prefix-caching \ --port 8900 \ --kv-transfer-config '{ "kv_connector": "LMCacheMPConnector", "kv_role": "kv_both" }'
--no-enable-prefix-caching keeps the test hit attributable to LMCache rather than vLLM’s internal prefix cache; enable internal prefix caching independently if you want it. A line like Creating transfer context (device_type=npu, mode=lmcache_driven) per worker confirms the NPU data plane is up. Send the same long prompt twice: the server log shows Stored ... tokens then Retrieved ... tokens, and vLLM’s External prefix cache hit rate climbs above zero. The second request skips the prefill for every token that hit the cache.
DeepSeek-V4 runs with 4096-token chunks: the deployment guide starts the server with --chunk-size 4096, and the connector also negotiates the server chunk size upward to the model’s required alignment before sessions start. The full guides (environment preparation, container setup, per-model startup, and verification) are in the repository:
lmcache_drivendeployment guide (DeepSeek-V4 example): docs/mp_mode/lmcache_driven.mdengine_drivendeployment guide (Qwen3 example): docs/mp_mode/engine_driven.md
What’s verified and supported
| Component | Requirement |
|---|---|
| Hardware | Atlas 800I A2 and Atlas 800I A3 (Ascend 910B / 910C, supported); Atlas 300I Duo (not supported); Atlas 800I A5 (planned support) |
| Stack | vLLM ≥ v0.23.0 with vLLM-Ascend ≥ v0.23.0, CANN ≥ 8.5, Ascend Driver ≥ 25.5, PyTorch ≥ 2.8 and torch_npu |
| LMCache pairing | LMCache dev ↔ LMCache-Ascend main; for release builds, use the matched pair in the repository compatibility matrix |
| Models validated | Qwen series (e.g. Qwen3-32B, TP8, both modes), DeepSeek-V4 (e.g. V4-Flash, TP8/EP, lmcache_driven), GLM-5.3, and other mainstream models |
| Also supported | Qwen3.5 / Qwen3.6 / Qwen3.8 hybrid Mamba (lmcache_driven with --mamba-cache-mode align and --separate-object-groups) |
The main limit is scope. DeepSeek-V4’s compressed MLA/DSA layout and hybrid Mamba/GDN models (Qwen3.5 / Qwen3.6 / Qwen3.8) run through lmcache_driven; engine_driven is the validated path for single-format GQA/MHA models. Hybrid models use --mamba-cache-mode align together with --separate-object-groups (the all mode is outside the Ascend recipe), and CacheBlend and CacheGen are not supported in MP mode yet.
Looking ahead
MP mode on Ascend is the foundation we will keep building on: continued performance optimization across both transfer paths, extending engine_driven to compressed and hybrid KV layouts, multi-node and distributed caching, and support for upcoming Ascend hardware at launch. As with the first Ascend release, we’re building this together with the Ascend developer community. If you’re deploying on Atlas servers or Ascend Cloud and want cache that survives your engine restarts, now is a good time to try it. Issues and PRs are welcome in both repositories.
Acknowledgments
This achievement would not have been possible without the support of our partners and internal team. We extend our sincere gratitude to the Huawei Euler Team and other Ascend developers, including the following individuals, for their contributions:
Get involved
- LMCache-Ascend: https://github.com/LMCache/LMCache-Ascend
- LMCache: https://github.com/LMCache/LMCache
- Join the LMCache Slack community: https://join.slack.com/t/lmcacheworkspace/shared_invite/zt-36x1m765z-8FgDA_73vcXtlZ_4XvpE6Q
- LMCache documentation: https://docs.lmcache.ai/