TL;DR
- Valkey for Context Memory is a collection of new features spanning the Valkey server, its official clients and SDKs, and inference framework integrations.
- LMCache’s built-in
`valkey`L2 adapter now has a remote direct memory access (RDMA) transport. Set`"transport": "rdma"`and the Valkey server moves KV cache chunks directly into and out of LMCache’s L1 arena over Elastic Fabric Adapter (EFA), with no copies on either end. - On the Valkey server, the open source Large Object module keeps those chunks in DRAM, or tiers them across DRAM and NVMe, as a cache that every inference worker in the fleet can share. In that way, it is like any other data object on the network
- Nothing above the transport changes. The adapter, the wire keys, and the eviction accounting all stay the same.
- Between two hosts with one 200 Gbit/s EFA each, four RDMA worker threads reach line rate with 4 MiB chunks. A single RDMA worker thread on 16 MiB chunks moves more data than TCP managed at any thread count we tested.
1. Why a shared KV tier, and why move bytes over RDMA
LMCache offloads KV cache from GPU memory into CPU memory and local disk storage. Those tiers remain tied to a specific inference worker, so useful state is lost in three common cases:
- The same document or stable prompt prefix is routed to a different inference worker.
- A session or agent turn returns after its state was evicted from local memory.
- Inference workers are scaled, drained, or replaced.
Each of those misses means repeated prefill. The GPU spends time recomputing work it has already done, and the user waits longer for the first token. A shared tier turns those misses into hits, which means more useful work from the same GPU fleet.
Valkey already provides a shared remote tier over TCP. Valkey is an open source fork of Redis, started by long-time Redis contributors under the Linux Foundation with support from Amazon, Google, Oracle, and many others. In an earlier Momento experiment with LMCache’s Valkey adapter (vLLM 0.28.0, LMCache 0.5.4, two NVIDIA L4 nodes on a 10 Gbit/s network), a second inference worker that had never run prefill reused all 30 cached documents. It reported a 5.97× speedup over its cold baseline.
What’s left is the cost of moving the bytes for large KV chunks. A 70B model running with tensor parallelism of 8 (TP=8) produces chunks of up to 16 MiB, and a single request can touch hundreds of them. The Valkey adapter already uses the synchronous GLIDE client (Valkey’s official client library) to avoid as many copies as it can, but every value still has to pass through client and command buffers and a TCP socket. At these sizes, that overhead dominates. In our measurements, a single TCP worker thread peaks at about 8 to 10 Gbit/s no matter how large the chunk.
So we took the payload off TCP entirely. Commands still travel over RESP (the Redis serialization protocol), but the bytes move over RDMA, straight between LMCache’s memory and the Valkey node.
2. Two channels, one shared cache

Each LMCache server talks to Valkey over two channels.
The control path is plain RESP over TCP. `BLOB.RDMA_HELLO` sets up an RDMA session between LMCache and the Valkey node. From then on, RESP carries the store, load, lookup, and delete commands, along with misses and completion replies. A store or load doesn’t carry the payload. Instead, it names a window of LMCache’s registered L1 arena as an (rkey, address, length) triple.
The data path is RDMA over EFA. The Valkey node’s fabric engine performs the transfer itself. On a store, the engine reads the chunk straight out of LMCache’s window. On a load, it writes the value straight into that window. When the Valkey node has several EFA devices, its fabric engine can pick the least-loaded one for each transfer.
The Valkey node initiates every RDMA transfer. LMCache only exposes its own L1 arena and never addresses memory on the Valkey node. Every inference worker shares the cache, so letting any one LMCache server write into the Valkey node’s memory would let a bug or compromise on one host reach every other worker. Having the Valkey node initiate every transfer keeps the trust boundary on the Valkey node.
3. Meet the pieces
| Component | Role |
|---|---|
| LMCache in multiprocess (MP) mode | Manages KV cache with an L1 arena in host memory and pluggable L2 adapters |
`valkey` L2 adapter with `transport: "rdma"` | Registers the L1 arena with the fabric and issues a store or load per chunk |
| Valkey GLIDE sync client | Manages connections and RDMA regions, and exposes `rdma_set` and `rdma_get` |
| Valkey Large Object module | Adds a custom Valkey data type for large objects and handles their placement, lifecycle, and reclamation across DRAM and NVMe |
| Fabric engine | Runs inside the Valkey Large Object module and performs server-initiated, one-sided RDMA over libfabric’s `efa-direct` provider |
LMCache stays the manager and scheduler. Together, the RDMA transport and the Valkey Large Object module make up Valkey for Context Memory, a shared cache tier that’s sized independently of the GPU fleet.
4. How it works inside LMCache
4.1 A transport option, not a new adapter
The Valkey adapter for LMCache already runs a pool of worker threads (`num_workers`), each with its own GLIDE client. GLIDE’s RDMA API matches that design in two ways. A registered RDMA region belongs to the client that registered it, and a region carries one transfer at a time. One registration per worker thread therefore gives `num_workers` concurrent transfers.
With `transport: "rdma"`, the adapter builds an RDMA worker pool instead of the TCP one, and that’s the only thing that changes. Threading, batching, partial-failure accounting, cluster mode, wire keys, and eviction all work as before. The one thing that doesn’t carry over is the stored data. The Valkey Large Object module uses its own data type, so RDMA and TCP caches need separate key prefixes or Valkey nodes (section 7).
4.2 Registering the L1 arena once
Each worker thread registers the whole L1 arena one time with the fabric at startup. The registration is a thin overlay over the arena pointer LMCache already has, so nothing is copied and nothing changes ownership.
For this to work, the arena must be fully populated before it’s registered, so run the LMCache MP server with `--no-l1-use-lazy`.
Registration is quick. On a c8gn.16xlarge, registering a populated 32 GiB arena takes about 50 ms with transparent huge pages and about 0.5 s with 4 KiB pages.
The constraint to plan around is capacity. An EFA device caps the total memory registered against it, and every thread’s registration counts. On a c8gn.16xlarge the cap is 384 GiB, so eight worker threads with a 32 GiB arena fit. On a g6.8xlarge it’s 96 GiB, so eight worker threads with a 16 GiB arena don’t fit. Size `num_workers × l1_size_gb` against your device. If you get it wrong, the adapter tells you at startup, and the error says whether to reduce `num_workers` or the L1 size.
4.3 From a memory object to an RDMA window
Every object LMCache stores or loads already lives somewhere in the L1 arena. The RDMA pool turns the object’s buffer into a window of the registered region:
offset = address(view) - l1_basewindow = region.window(offset, len(view))
Because the window comes from the buffer itself, the RDMA pool presents exactly the same submit API as the TCP pool, and the adapter doesn’t override anything per operation. A buffer outside the arena has no registration to transfer through, so it’s reported as a failure for that key. In MP mode every object comes from the arena, so that case points to a bug rather than something to quietly route around.
4.4 The transfers
| Operation | Command | What the Valkey node does |
|---|---|---|
| Session setup | `BLOB.RDMA_HELLO <client_rdma_address>` | Establishes the RDMA session with the client |
| Store | `BLOB.RDMA_SET key total_len [CRC crc32c] rkey addr len` | Reads the bytes out of the client’s window over RDMA and stores them |
| Load | `BLOB.RDMA_GET key rkey addr len` | Writes the value into the client’s window over RDMA and replies `[obj_len, crc32c]` |
| Lookup | `EXISTS key` | Native Valkey command |
| Delete | `DEL key` | The module releases the object’s buffers and any NVMe file |
| TTL (optional) | `EXPIRE key ttl` | Sent after the store command when `ttl_seconds` is set |
`BLOB.INFO key [LEN|CRC|TIER]` is also available for inspecting an object, including which tier currently holds it.
4.5 Safety and failure handling
Letting the Valkey node write straight into LMCache’s memory needs guardrails, and the design has several.
- Size check before any write:
`BLOB.RDMA_GET`carries the window’s length, and the Valkey node refuses a value that’s larger before it posts a single byte. That matters because every worker thread’s region spans the whole arena, so an unchecked write could run past the chunk into whatever comes next. A value that’s smaller than the window lands but is treated as a miss, because fixed-size chunks have to round-trip exactly. - Checksummed hits: The GLIDE client verifies the CRC32C in the load reply against the bytes that landed, so a reported hit has also been integrity-checked.
- Revoked regions: If a transfer fails without a reply from the Valkey node, for example because a connection drops, the GLIDE client revokes the region, since the Valkey node might still be using that memory. The worker thread re-registers the arena and retries the operation once. An error reply from the Valkey node leaves the region usable.
- Clean shutdown: A remote write can’t be cancelled once it’s posted, so RDMA transfers don’t time out. Closing a GLIDE client cancels its in-flight transfers, and the adapter closes clients before their regions.
- Fail fast on setup: If the installed GLIDE client was built without RDMA support, libfabric is missing on the LMCache host, or the Valkey node doesn’t have the Large Object module loaded, the adapter fails at construction with a message that names the missing piece.
5. DRAM and NVMe tiers inside the Valkey node
The Valkey Large Object module has two operating modes.
DRAM mode (`operating-mode Dram`) keeps every object in pre-allocated memory segments, registered with both the fabric and io_uring. It’s the fastest path, and it’s the mode we used for the results below.
Tiered mode (`operating-mode Tiered`) writes objects to NVMe files under `disk-dir`, using io_uring with O_DIRECT, and uses DRAM as a read cache. An object is promoted to DRAM after repeated access (`promote-min-hits`). When DRAM fills up, the coldest cached copies are dropped by LFU score, and the NVMe copy stays available.
In both modes, a stored object becomes visible only after its storage path has succeeded. Reads from NVMe flow through bounded staging buffers, so disk reads and RDMA writes overlap rather than running back to back.
6. RDMA vs TCP transfer throughput
We compared the two transports at the adapter level with `lmcache bench l2`, which drives the L2 adapter directly without an inference engine or MP server in the loop.
The test used two c8gn.16xlarge instances in one placement group, each with one 200 Gbit/s EFA. One instance ran `lmcache bench l2` and the other ran Valkey. For RDMA runs, Valkey had the Valkey Large Object module loaded in DRAM mode with `fabric-provider EfaDirect`. For the TCP baseline, it was a plain `valkey-server`, and the adapter used its default TCP transport.
Each request carried 32 keys, with as many requests in flight as there were worker threads. Every configuration ran one warmup round and three measured rounds. We measured stores and loads separately and flushed Valkey between configurations, and `transport` was the only setting that changed between runs.

Store throughput and latency
| Payload | Threads | RDMA Gbit/s | TCP Gbit/s | Speedup | Latency p50 ms | Latency p99 ms |
|---|---|---|---|---|---|---|
| 256 KiB | 1 | 14.9 | 9.3 | 1.6× | 4.5 | 4.6 |
| 256 KiB | 4 | 45.9 | 18.6 | 2.5× | 6.0 | 6.1 |
| 1 MiB | 1 | 44.9 | 9.6 | 4.7× | 6.0 | 6.0 |
| 1 MiB | 4 | 144.0 | 23.8 | 6.0× | 7.4 | 7.7 |
| 4 MiB | 1 | 92.3 | 9.5 | 9.7× | 11.7 | 11.7 |
| 4 MiB | 4 | 198.1 | 24.7 | 8.0× | 21.5 | 22.1 |
| 16 MiB | 1 | 130.5 | 9.5 | 13.8× | 32.9 | 33.0 |
| 16 MiB | 4 | 177.9 | 26.1 | 6.8× | 86.7 | 125.7 |
Load throughput and latency
| Payload | Threads | RDMA Gbit/s | TCP Gbit/s | Speedup | Latency p50 ms | Latency p99 ms |
|---|---|---|---|---|---|---|
| 256 KiB | 1 | 14.9 | 8.0 | 1.9× | 4.5 | 4.5 |
| 256 KiB | 4 | 43.6 | 27.1 | 1.6× | 6.3 | 6.6 |
| 1 MiB | 1 | 43.8 | 9.6 | 4.6× | 6.1 | 6.3 |
| 1 MiB | 4 | 136.5 | 36.6 | 3.7× | 7.8 | 8.2 |
| 4 MiB | 1 | 93.2 | 9.5 | 9.8× | 11.6 | 11.6 |
| 4 MiB | 4 | 193.5 | 36.9 | 5.2× | 22.2 | 22.4 |
| 16 MiB | 1 | 127.9 | 8.6 | 14.9× | 33.5 | 33.8 |
| 16 MiB | 4 | 197.8 | 32.8 | 6.0× | 86.8 | 87.0 |
We tested with 1, 4, 8, and 16 threads. The table depicts results for 1 and 4 threads. Each RDMA worker is significantly more efficient than a comparable TCP worker, and RDMA has a significantly higher total ceiling than TCP. At equivalent thread counts, RDMA is up to 8.0 times faster for stores and up to 6.0 times faster for loads.
Four RDMA worker threads drive the link at line rate, roughly 200 Gbit/s, for both stores and loads at 4 MiB and 16 MiB. TCP throughput peaks for stores at 26 Gbit/s with 4 threads, and for loads at 62.5 Gbit/s with 8 threads. Beyond that point, additional threads merely increase resource contention without improving throughput.
Sweeping chunk size demonstrates the bottleneck. A single TCP worker thread maxes out at 10 Gbit/s regardless of payload size, because per-byte copying and socket overhead dominate the cost. A single RDMA worker thread scales throughput with the payload, from 14.9 Gbit/s at 256 KiB to about 130 Gbit/s at 16 MiB. Large models produce multi-megabyte chunks, which is where the RDMA path produces the largest gain.
Latency tells the same story. With four worker threads, going from 4MiB loads to 16 MiB loads causes per-submit p50 latency to rise from 22.2 to 86.8 ms while throughput remains roughly the same at line rate. The extra latency comes from queuing larger payloads on a saturated link, rather than from slower transfers.
7. How to try it
Prerequisites
- EFA-capable instances for the LMCache servers and the Valkey node, in the same availability zone and VPC
- libfabric from the EFA installer
- LMCache
`0.5.8`and`valkey-glide-sync 2.6.0` - Valkey with the Valkey Large Object module loaded
pip install "lmcache==0.5.8" "valkey-glide-sync==2.6.0"
Start Valkey with the Valkey Large Object module
For DRAM mode:
valkey-server --port 6379 \ --loadmodule /path/to/libvalkey_large_object.so \ operating-mode Dram \ dram-maxmemory 64gb \ segment-size 1gb \ fabric-provider EfaDirect
For tiered mode, with NVMe behind DRAM:
valkey-server --port 6379 \ --loadmodule /path/to/libvalkey_large_object.so \ operating-mode Tiered \ disk-dir /mnt/nvme/valkey \ dram-maxmemory 64gb \ disk-maxmemory 1024gb \ disk-staging-size 1gb \ fabric-provider EfaDirect
Size `dram-maxmemory` and `disk-maxmemory` to your node.
Point LMCache at it
Add the RDMA transport to the Valkey L2 adapter, alongside your usual `lmcache server` flags:
lmcache server \ --l1-size-gb 32 \ --no-l1-use-lazy \ --l2-adapter '{ "type": "valkey", "transport": "rdma", "startup_nodes": "kv.internal:6379", "num_workers": 4, "rdma_provider": "efa-direct" }'
Every other `valkey` adapter field still works. On LMCache hosts with more than one EFA, set `rdma_interface` to choose which one the adapter uses.
Moving from the TCP transport
The Valkey Large Object module stores objects as its own data type, which plain `GET` and `SET` can’t read or replace. Give RDMA-backed caches their own `key_prefix`, or their own Valkey node, rather than mixing them with keys written over the TCP transport.
Sizing tips
- Check
`num_workers × l1_size_gb`against your EFA device’s registration cap (section 4.2). - On a single 200 Gbit/s EFA, four worker threads reach line rate at 4 MiB, and at 16 MiB for loads. Adding threads past that doesn’t add bandwidth on that link. It only uses more of the registration cap.
Check it’s working
`report_status()` on the adapter includes the RDMA settings. With `LMCACHE_LOG_LEVEL=DEBUG`, the LMCache server logs show L2 store tasks completing and prefetches finding keys in L2. To measure the transfer ceiling on your own hardware, run `lmcache bench l2` with 32 keys per submit and in-flight submits equal to `num_workers`, once with each transport. See the Valkey L2 adapter docs for the full configuration reference.
Developing without EFA
For local development and CI, start `valkey-server` with `fabric-provider Emulated` in the module arguments and set `"rdma_provider": "tcp"` in the LMCache adapter config. This uses libfabric’s TCP provider (libfabric 2.4 or later) and exercises the same code paths without EFA hardware.
8. Current limits
- EFA is the validated hardware path.
- Transfers go to and from registered host memory (the L1 arena), not directly to GPU memory.
- There is no TCP fallback for buffers outside the L1 arena.
- Wrapping the adapter with
`serde`still works, but each chunk then goes through an extra L1 buffer, so that path is no longer zero-copy. - Transfers always run against primaries, with no replica reads.
- Replication and failover for module objects are not inherited from base Valkey.
- In tiered mode, keeping objects on NVMe is not the same as restart durability.
- Setting
`ttl_seconds`adds one round trip per store.
9. Summary
LMCache can now move KV cache to and from a shared Valkey tier over RDMA, with no copies on either end. The Valkey node does the transfer itself, the Valkey Large Object module keeps the data in DRAM or across DRAM and NVMe, and the whole thing sits behind the existing Valkey adapter, so turning it on is a configuration change on the LMCache side. On a single 200 Gbit/s EFA, four RDMA worker threads reach line rate with 4 MiB chunks, and on chunks of 4 MiB or larger, a single RDMA worker thread is faster than TCP at any thread count we tested.
Acknowledgements
This integration was built by AWS and Momento, in collaboration with the LMCache team. Thanks to the LMCache maintainers for their reviews and for shaping the design of this integration.
Try it and tell us what you find
- Try the RDMA transport and open issues on LMCache, the Valkey Large Object module, or Valkey GLIDE.
- Join the conversation in LMCache GitHub Discussions and on the LMCache Slack.
- If you’d like help evaluating Valkey for Context Memory on your own workload, reach out to the Momento team.