lmcache

Valkey in LMCache: Building a Fast, Shared Memory Fabric

2026-11-19

TTony Valderrama (Momento)KKarsten Wade (LMCache)BBen Kugler (Momento)

TL;DR

  • Valkey for Context Memory is a collection of new features spanning the Valkey server, its official clients and SDKs, and inference framework integrations.
  • LMCache’s built-in `valkey` L2 adapter now has a remote direct memory access (RDMA) transport. Set `"transport": "rdma"` and the Valkey server moves KV cache chunks directly into and out of LMCache’s L1 arena over Elastic Fabric Adapter (EFA), with no copies on either end.
  • On the Valkey server, the open source Large Object module keeps those chunks in DRAM, or tiers them across DRAM and NVMe, as a cache that every inference worker in the fleet can share. In that way, it is like any other data object on the network
  • Nothing above the transport changes. The adapter, the wire keys, and the eviction accounting all stay the same.
  • Between two hosts with one 200 Gbit/s EFA each, four RDMA worker threads reach line rate with 4 MiB chunks. A single RDMA worker thread on 16 MiB chunks moves more data than TCP managed at any thread count we tested.

1. Why a shared KV tier, and why move bytes over RDMA

LMCache offloads KV cache from GPU memory into CPU memory and local disk storage. Those tiers remain tied to a specific inference worker, so useful state is lost in three common cases:

  • The same document or stable prompt prefix is routed to a different inference worker.
  • A session or agent turn returns after its state was evicted from local memory.
  • Inference workers are scaled, drained, or replaced.

Each of those misses means repeated prefill. The GPU spends time recomputing work it has already done, and the user waits longer for the first token. A shared tier turns those misses into hits, which means more useful work from the same GPU fleet.

Valkey already provides a shared remote tier over TCP. Valkey is an open source fork of Redis, started by long-time Redis contributors under the Linux Foundation with support from Amazon, Google, Oracle, and many others. In an earlier Momento experiment with LMCache’s Valkey adapter (vLLM 0.28.0, LMCache 0.5.4, two NVIDIA L4 nodes on a 10 Gbit/s network), a second inference worker that had never run prefill reused all 30 cached documents. It reported a 5.97× speedup over its cold baseline.

What’s left is the cost of moving the bytes for large KV chunks. A 70B model running with tensor parallelism of 8 (TP=8) produces chunks of up to 16 MiB, and a single request can touch hundreds of them. The Valkey adapter already uses the synchronous GLIDE client (Valkey’s official client library) to avoid as many copies as it can, but every value still has to pass through client and command buffers and a TCP socket. At these sizes, that overhead dominates. In our measurements, a single TCP worker thread peaks at about 8 to 10 Gbit/s no matter how large the chunk.

So we took the payload off TCP entirely. Commands still travel over RESP (the Redis serialization protocol), but the bytes move over RDMA, straight between LMCache’s memory and the Valkey node.

2. Two channels, one shared cache

Detail drawing of one transfer. On the left, an LMCache server holds its L1 arena, drawn as a strip of equal chunks. The last chunks, outlined heavily and shaded orange, are the window; a dimension line below marks its offset from the start of the arena and its length, len. A dashed line labeled RESP over TCP runs from the LMCache server to the Valkey node on the right and carries rkey, addr, and len, with no KV bytes. Two orange lines connect the window to the fabric engine inside the Valkey node. Both start at a dot on the fabric engine, which is labeled initiates every transfer. On store, the arrow points to the engine: the engine reads the window. On load, the arrow points to the window: the engine writes the window. A dashed vertical line just left of the Valkey node is labeled trust boundary; no line starts on the LMCache side and reaches into the node's memory.
A store or load names a window of LMCache’s L1 arena; it never carries the bytes. The Valkey node’s fabric engine starts every transfer, reading the window on a store and writing it on a load, so LMCache never addresses the node’s memory.

Each LMCache server talks to Valkey over two channels.

The control path is plain RESP over TCP. `BLOB.RDMA_HELLO` sets up an RDMA session between LMCache and the Valkey node. From then on, RESP carries the store, load, lookup, and delete commands, along with misses and completion replies. A store or load doesn’t carry the payload. Instead, it names a window of LMCache’s registered L1 arena as an (rkey, address, length) triple.

The data path is RDMA over EFA. The Valkey node’s fabric engine performs the transfer itself. On a store, the engine reads the chunk straight out of LMCache’s window. On a load, it writes the value straight into that window. When the Valkey node has several EFA devices, its fabric engine can pick the least-loaded one for each transfer.

The Valkey node initiates every RDMA transfer. LMCache only exposes its own L1 arena and never addresses memory on the Valkey node. Every inference worker shares the cache, so letting any one LMCache server write into the Valkey node’s memory would let a bug or compromise on one host reach every other worker. Having the Valkey node initiate every transfer keeps the trust boundary on the Valkey node.

3. Meet the pieces

ComponentRole
LMCache in multiprocess (MP) modeManages KV cache with an L1 arena in host memory and pluggable L2 adapters
`valkey` L2 adapter with `transport: "rdma"`Registers the L1 arena with the fabric and issues a store or load per chunk
Valkey GLIDE sync clientManages connections and RDMA regions, and exposes `rdma_set` and `rdma_get`
Valkey Large Object moduleAdds a custom Valkey data type for large objects and handles their placement, lifecycle, and reclamation across DRAM and NVMe
Fabric engineRuns inside the Valkey Large Object module and performs server-initiated, one-sided RDMA over libfabric’s `efa-direct` provider

LMCache stays the manager and scheduler. Together, the RDMA transport and the Valkey Large Object module make up Valkey for Context Memory, a shared cache tier that’s sized independently of the GPU fleet.

4. How it works inside LMCache

4.1 A transport option, not a new adapter

The Valkey adapter for LMCache already runs a pool of worker threads (`num_workers`), each with its own GLIDE client. GLIDE’s RDMA API matches that design in two ways. A registered RDMA region belongs to the client that registered it, and a region carries one transfer at a time. One registration per worker thread therefore gives `num_workers` concurrent transfers.

With `transport: "rdma"`, the adapter builds an RDMA worker pool instead of the TCP one, and that’s the only thing that changes. Threading, batching, partial-failure accounting, cluster mode, wire keys, and eviction all work as before. The one thing that doesn’t carry over is the stored data. The Valkey Large Object module uses its own data type, so RDMA and TCP caches need separate key prefixes or Valkey nodes (section 7).

4.2 Registering the L1 arena once

Each worker thread registers the whole L1 arena one time with the fabric at startup. The registration is a thin overlay over the arena pointer LMCache already has, so nothing is copied and nothing changes ownership.

For this to work, the arena must be fully populated before it’s registered, so run the LMCache MP server with `--no-l1-use-lazy`.

Registration is quick. On a c8gn.16xlarge, registering a populated 32 GiB arena takes about 50 ms with transparent huge pages and about 0.5 s with 4 KiB pages.

The constraint to plan around is capacity. An EFA device caps the total memory registered against it, and every thread’s registration counts. On a c8gn.16xlarge the cap is 384 GiB, so eight worker threads with a 32 GiB arena fit. On a g6.8xlarge it’s 96 GiB, so eight worker threads with a 16 GiB arena don’t fit. Size `num_workers × l1_size_gb` against your device. If you get it wrong, the adapter tells you at startup, and the error says whether to reduce `num_workers` or the L1 size.

4.3 From a memory object to an RDMA window

Every object LMCache stores or loads already lives somewhere in the L1 arena. The RDMA pool turns the object’s buffer into a window of the registered region:

offset = address(view) - l1_base
window = region.window(offset, len(view))

Because the window comes from the buffer itself, the RDMA pool presents exactly the same submit API as the TCP pool, and the adapter doesn’t override anything per operation. A buffer outside the arena has no registration to transfer through, so it’s reported as a failure for that key. In MP mode every object comes from the arena, so that case points to a bug rather than something to quietly route around.

4.4 The transfers

OperationCommandWhat the Valkey node does
Session setup`BLOB.RDMA_HELLO <client_rdma_address>`Establishes the RDMA session with the client
Store`BLOB.RDMA_SET key total_len [CRC crc32c] rkey addr len`Reads the bytes out of the client’s window over RDMA and stores them
Load`BLOB.RDMA_GET key rkey addr len`Writes the value into the client’s window over RDMA and replies `[obj_len, crc32c]`
Lookup`EXISTS key`Native Valkey command
Delete`DEL key`The module releases the object’s buffers and any NVMe file
TTL (optional)`EXPIRE key ttl`Sent after the store command when `ttl_seconds` is set

`BLOB.INFO key [LEN|CRC|TIER]` is also available for inspecting an object, including which tier currently holds it.

4.5 Safety and failure handling

Letting the Valkey node write straight into LMCache’s memory needs guardrails, and the design has several.

  • Size check before any write: `BLOB.RDMA_GET` carries the window’s length, and the Valkey node refuses a value that’s larger before it posts a single byte. That matters because every worker thread’s region spans the whole arena, so an unchecked write could run past the chunk into whatever comes next. A value that’s smaller than the window lands but is treated as a miss, because fixed-size chunks have to round-trip exactly.
  • Checksummed hits: The GLIDE client verifies the CRC32C in the load reply against the bytes that landed, so a reported hit has also been integrity-checked.
  • Revoked regions: If a transfer fails without a reply from the Valkey node, for example because a connection drops, the GLIDE client revokes the region, since the Valkey node might still be using that memory. The worker thread re-registers the arena and retries the operation once. An error reply from the Valkey node leaves the region usable.
  • Clean shutdown: A remote write can’t be cancelled once it’s posted, so RDMA transfers don’t time out. Closing a GLIDE client cancels its in-flight transfers, and the adapter closes clients before their regions.
  • Fail fast on setup: If the installed GLIDE client was built without RDMA support, libfabric is missing on the LMCache host, or the Valkey node doesn’t have the Large Object module loaded, the adapter fails at construction with a message that names the missing piece.

5. DRAM and NVMe tiers inside the Valkey node

The Valkey Large Object module has two operating modes.

DRAM mode (`operating-mode Dram`) keeps every object in pre-allocated memory segments, registered with both the fabric and io_uring. It’s the fastest path, and it’s the mode we used for the results below.

Tiered mode (`operating-mode Tiered`) writes objects to NVMe files under `disk-dir`, using io_uring with O_DIRECT, and uses DRAM as a read cache. An object is promoted to DRAM after repeated access (`promote-min-hits`). When DRAM fills up, the coldest cached copies are dropped by LFU score, and the NVMe copy stays available.

In both modes, a stored object becomes visible only after its storage path has succeeded. Reads from NVMe flow through bounded staging buffers, so disk reads and RDMA writes overlap rather than running back to back.

6. RDMA vs TCP transfer throughput

We compared the two transports at the adapter level with `lmcache bench l2`, which drives the L2 adapter directly without an inference engine or MP server in the loop.

The test used two c8gn.16xlarge instances in one placement group, each with one 200 Gbit/s EFA. One instance ran `lmcache bench l2` and the other ran Valkey. For RDMA runs, Valkey had the Valkey Large Object module loaded in DRAM mode with `fabric-provider EfaDirect`. For the TCP baseline, it was a plain `valkey-server`, and the adapter used its default TCP transport.

Each request carried 32 keys, with as many requests in flight as there were worker threads. Every configuration ran one warmup round and three measured rounds. We measured stores and loads separately and flushed Valkey between configurations, and `transport` was the only setting that changed between runs.

Two line charts side by side, store on the left and load on the right, plotting throughput in Gbit/s against payload size per key at 256 KiB, 1 MiB, 4 MiB, and 16 MiB. Each panel has four lines: RDMA in orange and TCP in dark blue, each at 4 threads (heavy line, filled markers) and 1 thread (thin line, open markers). A dashed line marks the 200 Gbit/s line rate. Store, RDMA 4 threads: 45.9, 144.0, 198.1, and 177.9. RDMA 1 thread: 14.9, 44.9, 92.3, and 130.5. TCP 4 threads: 18.6, 23.8, 24.7, and 26.1. TCP 1 thread: 9.3, 9.6, 9.5, and 9.5. Load, RDMA 4 threads: 43.6, 136.5, 193.5, and 197.8. RDMA 1 thread: 14.9, 43.8, 93.2, and 127.9. TCP 4 threads: 27.1, 36.6, 36.9, and 32.8. TCP 1 thread: 8.0, 9.6, 9.5, and 8.6. Both RDMA lines climb with payload size and the 4-thread line meets the line rate at 4 MiB; both TCP lines stay flat near the bottom. A dotted vertical line between 4 MiB and 16 MiB marks a roughly 10 MiB chunk from a 70B model at TP=8, a size that was not measured.
Four RDMA worker threads reach the 200 Gbit/s line rate at 4 MiB, and a single RDMA thread climbs with payload size while TCP stays flat. Two c8gn.16xlarge hosts with one 200 Gbit/s EFA each, lmcache bench l2, 32 keys per submit, module in DRAM mode. The dotted line marks a ~10 MiB chunk from a 70B model at TP=8, a size not measured here.

Store throughput and latency

PayloadThreadsRDMA Gbit/sTCP Gbit/sSpeedupLatency p50 msLatency p99 ms
256 KiB114.99.31.6×4.54.6
256 KiB445.918.62.5×6.06.1
1 MiB144.99.64.7×6.06.0
1 MiB4144.023.86.0×7.47.7
4 MiB192.39.59.7×11.711.7
4 MiB4198.124.78.0×21.522.1
16 MiB1130.59.513.8×32.933.0
16 MiB4177.926.16.8×86.7125.7

Load throughput and latency

PayloadThreadsRDMA Gbit/sTCP Gbit/sSpeedupLatency p50 msLatency p99 ms
256 KiB114.98.01.9×4.54.5
256 KiB443.627.11.6×6.36.6
1 MiB143.89.64.6×6.16.3
1 MiB4136.536.63.7×7.88.2
4 MiB193.29.59.8×11.611.6
4 MiB4193.536.95.2×22.222.4
16 MiB1127.98.614.9×33.533.8
16 MiB4197.832.86.0×86.887.0

We tested with 1, 4, 8, and 16 threads. The table depicts results for 1 and 4 threads. Each RDMA worker is significantly more efficient than a comparable TCP worker, and RDMA has a significantly higher total ceiling than TCP. At equivalent thread counts, RDMA is up to 8.0 times faster for stores and up to 6.0 times faster for loads.

Four RDMA worker threads drive the link at line rate, roughly 200 Gbit/s, for both stores and loads at 4 MiB and 16 MiB. TCP throughput peaks for stores at 26 Gbit/s with 4 threads, and for loads at 62.5 Gbit/s with 8 threads. Beyond that point, additional threads merely increase resource contention without improving throughput.

Sweeping chunk size demonstrates the bottleneck. A single TCP worker thread maxes out at 10 Gbit/s regardless of payload size, because per-byte copying and socket overhead dominate the cost. A single RDMA worker thread scales throughput with the payload, from 14.9 Gbit/s at 256 KiB to about 130 Gbit/s at 16 MiB. Large models produce multi-megabyte chunks, which is where the RDMA path produces the largest gain.

Latency tells the same story. With four worker threads, going from 4MiB loads to 16 MiB loads causes per-submit p50 latency to rise from 22.2 to 86.8 ms while throughput remains roughly the same at line rate. The extra latency comes from queuing larger payloads on a saturated link, rather than from slower transfers.

7. How to try it

Prerequisites

  • EFA-capable instances for the LMCache servers and the Valkey node, in the same availability zone and VPC
  • libfabric from the EFA installer
  • LMCache `0.5.8` and `valkey-glide-sync 2.6.0`
  • Valkey with the Valkey Large Object module loaded
pip install "lmcache==0.5.8" "valkey-glide-sync==2.6.0"

Start Valkey with the Valkey Large Object module

For DRAM mode:

valkey-server --port 6379 \
--loadmodule /path/to/libvalkey_large_object.so \
operating-mode Dram \
dram-maxmemory 64gb \
segment-size 1gb \
fabric-provider EfaDirect

For tiered mode, with NVMe behind DRAM:

valkey-server --port 6379 \
--loadmodule /path/to/libvalkey_large_object.so \
operating-mode Tiered \
disk-dir /mnt/nvme/valkey \
dram-maxmemory 64gb \
disk-maxmemory 1024gb \
disk-staging-size 1gb \
fabric-provider EfaDirect

Size `dram-maxmemory` and `disk-maxmemory` to your node.

Point LMCache at it

Add the RDMA transport to the Valkey L2 adapter, alongside your usual `lmcache server` flags:

lmcache server \
--l1-size-gb 32 \
--no-l1-use-lazy \
--l2-adapter '{
"type": "valkey",
"transport": "rdma",
"startup_nodes": "kv.internal:6379",
"num_workers": 4,
"rdma_provider": "efa-direct"
}'

Every other `valkey` adapter field still works. On LMCache hosts with more than one EFA, set `rdma_interface` to choose which one the adapter uses.

Moving from the TCP transport

The Valkey Large Object module stores objects as its own data type, which plain `GET` and `SET` can’t read or replace. Give RDMA-backed caches their own `key_prefix`, or their own Valkey node, rather than mixing them with keys written over the TCP transport.

Sizing tips

  • Check `num_workers × l1_size_gb` against your EFA device’s registration cap (section 4.2).
  • On a single 200 Gbit/s EFA, four worker threads reach line rate at 4 MiB, and at 16 MiB for loads. Adding threads past that doesn’t add bandwidth on that link. It only uses more of the registration cap.

Check it’s working

`report_status()` on the adapter includes the RDMA settings. With `LMCACHE_LOG_LEVEL=DEBUG`, the LMCache server logs show L2 store tasks completing and prefetches finding keys in L2. To measure the transfer ceiling on your own hardware, run `lmcache bench l2` with 32 keys per submit and in-flight submits equal to `num_workers`, once with each transport. See the Valkey L2 adapter docs for the full configuration reference.

Developing without EFA

For local development and CI, start `valkey-server` with `fabric-provider Emulated` in the module arguments and set `"rdma_provider": "tcp"` in the LMCache adapter config. This uses libfabric’s TCP provider (libfabric 2.4 or later) and exercises the same code paths without EFA hardware.

8. Current limits

  • EFA is the validated hardware path.
  • Transfers go to and from registered host memory (the L1 arena), not directly to GPU memory.
  • There is no TCP fallback for buffers outside the L1 arena.
  • Wrapping the adapter with `serde` still works, but each chunk then goes through an extra L1 buffer, so that path is no longer zero-copy.
  • Transfers always run against primaries, with no replica reads.
  • Replication and failover for module objects are not inherited from base Valkey.
  • In tiered mode, keeping objects on NVMe is not the same as restart durability.
  • Setting `ttl_seconds` adds one round trip per store.

9. Summary

LMCache can now move KV cache to and from a shared Valkey tier over RDMA, with no copies on either end. The Valkey node does the transfer itself, the Valkey Large Object module keeps the data in DRAM or across DRAM and NVMe, and the whole thing sits behind the existing Valkey adapter, so turning it on is a configuration change on the LMCache side. On a single 200 Gbit/s EFA, four RDMA worker threads reach line rate with 4 MiB chunks, and on chunks of 4 MiB or larger, a single RDMA worker thread is faster than TCP at any thread count we tested.

Acknowledgements

This integration was built by AWS and Momento, in collaboration with the LMCache team. Thanks to the LMCache maintainers for their reviews and for shaping the design of this integration.

Try it and tell us what you find

Table of Contents

Share via:

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

More from the blog