By Dongjoo Seo, Jinyoung Moon @Samsung and the LMCache Team
TL;DR
LMCache can use byte-addressable memory exposed through Device-DAX as part of its KV cache hierarchy. The original /dev/dax backend has since expanded to cover optimized non-MP retrieval, MP DAX L2 with HTTP reconfiguration, and Device-DAX L1 with programmatic add and drain-remove operations.
This post covers the implementation, the differences between the L1 and L2 paths, the available performance data, and the current limitations.
1. Why Device-DAX Matters for KV Cache
Long contexts, multi-turn conversations, RAG, and agentic workloads all produce large KV caches. Reusing those caches saves prefill work. Once GPU memory fills, however, the serving stack needs another tier with more capacity and lower access overhead than conventional storage.
LMCache makes that KV state reusable across requests. In MP mode, a standalone LMCache server manages the cache while inference workers stay focused on model execution.
Device-DAX exposes byte-addressable memory through /dev/dax, allowing a process to mmap a device directly into userspace. The original LMCache Device-DAX RFC proposed using /dev/dax as a KV cache storage tier for persistent memory, CXL memory that the platform and kernel have already exposed as Device-DAX, and other byte-addressable memory devices.
From LMCache’s point of view, a DAX mapping is a large userspace memory arena. It is neither GPU memory nor a filesystem-backed disk tier, and it can sit between DRAM and slower storage.
2. Device-DAX Storage Backend
The first implementation added a Device-DAX backend for KV cache. This backend maps a /dev/dax device into userspace and uses the mapped region as an arena for KV cache chunks. The initial implementation introduced fixed-size slot allocation, LRU-style management, and documentation for configuring the DAX backend.
The design was straightforward:
- Treat the DAX device as a large memory-mapped arena.
- Store KV cache chunks into fixed-size slots.
- Read cached chunks back into CPU-backed memory objects.
- Let the rest of LMCache continue using its normal storage and retrieval flow.
The inference engine and vLLM connector did not need any DAX-specific protocol changes.
3. Optimizing the Restore Path
Retrieval was the first performance problem we had to address. The original path restaged chunks one at a time, which added noticeable overhead on long-context cache hits.
For in-process mode, we replaced that path with a staged batch pipeline. It groups the restore work, coalesces adjacent DAX spans, keeps workers alive between requests, and reuses a pinned staging slab. In our local Qwen/Qwen3-14B long-document QA test, mean query TTFT fell by about 61.6% and mean query-round time by about 33.3% versus the earlier backend. These results describe that PR setup, not a general performance guarantee.
We left the store path alone: KV data still passes through CPU memory before entering the DAX arena. Only retrieval changed.
The restore flow is:
- Reserve a batched set of readable DAX chunks.
- Allocate CPU restore buffers from LocalCPUBackend.
- Copy DAX data into a backend-owned pinned staging slab in coalesced regions.
- Copy from the staging slab into final CPU MemoryObj outputs.
- Upload those CPU outputs through the normal GPU connector path.
That change made the non-MP backend more useful for cache-hit-heavy workloads.
4. Device-DAX in Multiprocess (MP) Mode
In MP mode, the inference engine and LMCache server run in separate processes. This isolates resources and lets multiple workers or instances share cached KV data.
Supporting DAX here required an implementation of the MP L2 adapter contract.
Inside the MP server, Device-DAX is available at both L1 and L2, but the paths serve different roles. The L1 path extends the server’s directly managed memory capacity. The L2 path acts as a storage adapter with asynchronous store, lookup-and-lock, load, unlock, delete, usage, and status operations. Non-MP deployments use a separate Device-DAX backend.
Figure 1. Device-DAX is available through the non-MP storage backend and through both L1 and L2 paths in MP mode.
For example, a /dev/dax* endpoint can be CXL memory that the platform and kernel expose in Device-DAX mode.
No vLLM request or connector protocol changed; DAX stays behind LMCache’s storage abstraction.
Since the first MP adapter landed, the L2 load path has also been optimized. It now groups keys by device, releases the adapter-wide lock before copying, batches reads for each device, and runs bounded parallel sub-batches through a persistent worker pool. In the PR’s CXL Device-DAX test, mean reuse TTFT fell from 5.68 seconds to 3.31 seconds. As with the other benchmark numbers here, that result reflects the reported setup rather than a general guarantee.
The MP DAX adapter is volatile. Its key index lives in server memory and starts empty after a restart. Bytes may still be present on the device, but LMCache cannot reach them without the index.
For now, Device-DAX is a high-capacity runtime tier rather than a restart-recoverable persistent store.
5. Runtime Reconfiguration for MP DAX L2
Once the MP adapter worked, changing its capacity still required a server restart.
A restart also dropped the volatile in-memory key index. MP DAX L2 now supports live reconfiguration, so operators can change capacity without taking the server down.
For MP DAX L2, the generic HTTP reconfiguration API includes:
GET /reconfigure/{backend}/status
POST /reconfigure/{backend}/{operation}
For DAX, the supported operations are:
GET /reconfigure/dax/status
POST /reconfigure/dax/add
POST /reconfigure/dax/remove
POST /reconfigure/dax/resize
While the server is running, an operator can add a region, drain it, remove it with migrate or evict semantics, or resize it.
This avoids a restart when DAX capacity changes. The API is backend-generic, so other L2 adapters can implement the same interface later.
6. Hybrid L1: Device-DAX as L1 Capacity Expansion
Device-DAX can also back L1, not just L2.
In hybrid mode, LMCache allocates from DRAM first, then from Device-DAX, and finally from a lower tier. If –l1-devdax-path matches a device in a registered DAX adapter, LMCache consumes that device as L1 overflow capacity and removes it from the L2 adapter. Any other devices in the adapter remain available to L2. If no adapter device matches, the DAX device instead backs the entire L1 arena on its own (pure-DAX mode), where –l1-size-gb sizes the DAX mapping and no DRAM pool is created.
Supported layouts include:
30 GiB DRAM L1 only
30 GiB DRAM L1 + 30 GiB Device-DAX L1 overflow
30 GiB DRAM L1 + 30 GiB Device-DAX L2
30 GiB DRAM L1 + 30 GiB Device-DAX L1 overflow + raw-block L2
In L2 mode, DAX is a secondary storage adapter. In hybrid L1 mode, it is part of the L1 allocator.
The MP configuration reference now documents –l1-devdax-path as an optional /dev/dax* device or mmap-able file for the L1 backing arena. When used, pass –no-l1-use-lazy and –shm-name “” because L1 bytes live in the DAX mapping.
When CUDA is available, LMCache calls cudaHostRegister on the DAX mapping. If registration succeeds, GPU transfers use it as pinned host memory. Otherwise, LMCache stages data through pinned DRAM.
6.1 Runtime Reconfiguration for Device-DAX L1
Device-DAX L1 now supports runtime changes through DevDaxL1MemoryManager. add_device() adds an already-provisioned DAX device as an arena. remove_device() marks a non-primary arena as draining: new allocations stop, existing KV objects remain readable, and the mapping is released after the last allocation is freed.
get_arena_statuses() reports the current arena state. L1 reconfiguration works at whole-arena granularity, and removal is drain-only. It has no HTTP endpoint, resize operation, or live-object migrate/evict support. In pure-DAX mode, the primary arena is fixed. In hybrid mode, DRAM is primary, so the DAX arenas can be removed.
This landed in dev on July 29 and is available in that day’s nightly builds. It is not part of v0.5.2.
7. Benchmark Results
The hybrid L1 PR reported both microbenchmarks and end-to-end tests.
In the GPU-to-L1 host transfer microbenchmark on an NVIDIA H100 PCIe system, pinned DRAM reached about 55.5 GB/s for both store and load directions. Registered Device-DAX reached about 47.6 GB/s for store and about 50.1 GB/s for load. The staged fallback path was lower, around 22.8 GB/s store and 26.0 GB/s load.
End-to-end results depended on the model and working-set size. With Qwen/Qwen3-0.6B, hybrid L1 avoided some of the TTFT increase seen in L2 at larger working sets, but it did not beat DRAM everywhere. With Qwen/Qwen3-14B, hybrid L1 beat the 30 GiB DRAM-only control at 45 GiB and 75 GiB working sets. DAX L2 remained competitive when it preserved more second-pass hits.
Hybrid L1 is not a universal win. It is another placement choice. Whether DAX belongs in L1 or L2 depends on the working set, cache-hit pattern, model size, and GPU transfer path.
8. How to Try It
To use DAX as MP L2, start the server with a DAX adapter:
lmcache server \
–l1-size-gb 80 \
–eviction-policy LRU \
–l2-adapter ‘{
“type”: “dax”,
“device_path”: “/dev/dax1.0”,
“max_dax_size_gb”: 100,
“slot_bytes”: 268435456,
“num_store_workers”: 1,
“num_lookup_workers”: 1,
“num_load_workers”: 4
}’
For multiple startup devices with runtime hotplug enabled:
lmcache server \
–l1-size-gb 80 \
–eviction-policy LRU \
–l2-adapter ‘{
“type”: “dax”,
“devices”: [
{“device_path”: “/dev/daxX.X”, “max_dax_size_gb”: 100},
{“device_path”: “/dev/daxY.Y”, “max_dax_size_gb”: 100}
],
“slot_bytes”: 268435456,
“hotplug_enabled”: true,
“num_store_workers”: 1,
“num_lookup_workers”: 1,
“num_load_workers”: 4
}’
Use the HTTP interface on the default port, 8080:
curl http://127.0.0.1:8080/reconfigure/dax/status
curl -X POST http://127.0.0.1:8080/reconfigure/dax/add \
-H ‘Content-Type: application/json’ \
-d ‘{“device_path”: “/dev/daxZ.Z”, “size”: “100GiB”}’
curl -X POST http://127.0.0.1:8080/reconfigure/dax/remove \
-H ‘Content-Type: application/json’ \
-d ‘{“device_path”: “/dev/daxX.X”, “mode”: “migrate”}’
These APIs change LMCache mappings and metadata; they do not provision CXL/DAX devices in the kernel. The MP path also lacks per-TP partitions and restart metadata, accepts only single-buffer objects, and accounts for capacity and eviction in slots. Use destructive removal modes carefully.
To use DAX as L1 (pure-DAX mode), point –l1-devdax-path at a device that is not listed in any DAX adapter:
lmcache server \
–l1-size-gb 80 \
–eviction-policy LRU \
–l1-devdax-path /dev/dax1.0 \
–no-l1-use-lazy \
–shm-name “”
For hybrid L1 mode, list the same device in a DAX adapter:
lmcache server \
–l1-size-gb 80 \
–eviction-policy LRU \
–l1-devdax-path /dev/dax1.0 \
–no-l1-use-lazy \
–shm-name “” \
–l2-adapter ‘{
“type”: “dax”,
“devices”: [
{“device_path”: “/dev/dax1.0”, “max_dax_size_gb”: 100}],
“slot_bytes”: 268435456
}’
9. Current Limits and Future Work
The main limitations today are:
- MP DAX is volatile: restart recovery is not implemented.
- Capacity accounting is slot-based rather than exact payload-byte-based.
- Multi-tensor objects are not yet supported in the MP DAX path.
- Runtime reconfiguration manages LMCache mappings, not kernel-level device provisioning. MP DAX L2 exposes HTTP add/remove/resize operations; Device-DAX L1 is programmatic, changes whole arenas, and supports drain-only removal. The primary pure-DAX L1 arena cannot be removed.
- Some hardware-specific behavior depends on whether CUDA host registration succeeds.
Likely follow-up work includes on-device metadata, restart recovery, better multi-device policies, deeper CXL integration, and further transfer-path optimization.
10. Summary
We started with a narrow question: can LMCache use /dev/dax as a practical KV cache tier? The answer seems to be “yes”.
The current implementation covers non-MP storage, MP DAX L2, live L2 reconfiguration, and Device-DAX-backed L1. Each piece was added behind LMCache’s existing interfaces, so inference engines do not need DAX-specific behavior.
Recovery, device policy, and transfer efficiency still need work. Even so, Device-DAX already gives deployments another place to keep reusable KV data when DRAM capacity is tight.
