Encrypting KV cache at rest using LMCache

TL;DR

LMCache can now encrypt KV cache at rest in the L2 tier (S3, fs, RESP, …) in vLLM or SGLang deployments with AES-GCM, keyed per cache_salt so each tenant’s ciphertext is distinct. It is implemented as a serde — the same pluggable transform layer used by the fp8 and turboquant quantizers — so it works with any L2 adapter, with no adapter or controller changes.

This post covers the threat model, why the serde layer is the right seam, the wire format and key model, and how to turn it on raw and through the operator.

1. Why Encrypt the KV Cache?

Multi-turn conversations, RAG, and agentic workloads make KV cache reuse the biggest lever in reducing serving cost. Whether you serve with vLLM or SGLang, LMCache’s whole job is to keep that KV state around: in GPU memory (L0), in host RAM (L1), and durably in a shared L2 backend such as an S3 bucket, a filesystem, or a RESP store.

That last tier changes the security picture. KV cache is a lossy-but-real encoding of the tokens that produced it — system prompts, user documents, and conversation history. In a multi-tenant deployment, many tenants’ KV lands in one shared L2 backend, and anyone who can read that storage can read all of it. “Anyone” includes a misconfigured bucket policy, a snapshot, or a storage admin outside the serving team.

The scope of this encryption feature is deliberately narrow: this protects L2 bytes against a reader of the remote storage. L1 (host RAM) and L0 (GPU) still hold plaintext, so a party with access to a running MP server process is not defended against — that is a different threat model. This feature is at-rest confidentiality for the durable tier rather than end-to-end encryption.

2. Why a Serde Is Exactly the Right Shape

LMCache’s MP-mode L2 path already has a transform seam: the serde. Serde is “a framework for serializing and deserializing Rust data structures efficiently and generically.”

A serde runs on both legs of the L1↔L2 round trip — serialize on store (L1→L2) and deserialize on load (L2→L1) — and plugs into any L2 adapter through the existing SerdeL2AdapterWrapper. The quantizer serdes (fp8, turboquant) already live there.

Encryption wants precisely this shape:

• It must be reversible — that is, a symmetric cipher that is decryptable with the key. The two-leg serde contract enforces that by construction.
• It should apply to every L2 backend uniformly rather than be re-implemented per adapter. As a serde, aesgcm composes with S3, fs, RESP, or anything else behind the wrapper.
• It runs off the inference hot path with the L1↔L2 leg as the durable-tier transfer rather than the L0↔L1 CUDA-IPC transfer. This way encryption cost hides behind storage latency instead of adding to TTFT.

3. The Cipher: AES-128-GCM, and Why

The serde (#4235) defaults to AES-128-GCM — an AEAD cipher, so one pass buys both confidentiality and integrity. The alternatives, on server hardware with AES-NI:

AES-128 is already computationally unbreakable, and KV cache is short-lived and regenerable, so the harvest-now-decrypt-later argument for 256-bit keys barely applies — 256 is a configurable variable, not the default.

Two non-alternatives worth naming: compression is not encryption (it makes nothing secret, and KV is high-entropy so it barely shrinks anyway), and if you want smaller L2 objects, quantization is the size lever — compose fp8/turboquant before encryption.

Encrypted Format

Each KVCache chunk is stored as:

[1B version][12B IV][ciphertext || 16B GCM tag]

• version — a format byte, so the scheme can evolve without breaking stored blobs.
• IV — a fresh random 96-bit nonce per chunk, stored in the clear. An IV is not secret; it must only never repeat for the same key.
• ciphertext ‖ tag — the plaintext length exactly (no padding) plus the 16-byte integrity tag.

Fixed overhead is 29 bytes per chunk, so the serde’s size estimate is exact rather than an upper bound. And the integrity half of AEAD does real work here: a tag mismatch — tampered bytes, or the wrong key — raises InvalidTag, which the wrapper turns into a load miss. The engine re-fetches or recomputes; corrupted ciphertext can never be silently deserialized into a model’s attention states.

4. Encryption Keys: cache_salt Selects, a KeyProvider Supplies

LMCache already has a tenant selector: cache_salt, which scopes cache entries and appears (in cleartext) in the L2 object name. The key model keeps it in that role — cache_salt picks which key; it is never the key itself. Secret material comes from a swappable KeyProvider:

• HkdfKeyProvider (default, shipped) derives a per-tenant key as HKDF-SHA256(master_key, info = "lmcache-l2-aesgcm-v1" + cache_salt) from one master key read from a file (master_key_path — in Kubernetes, a mounted Secret). Derived keys are cached per salt behind a lock; the empty salt (anonymous traffic) is just another valid tenant bucket.
• KeyringKeyProvider (future) would provision true per-tenant keys via KMS or per-tenant mounts.

The trust model deserves honesty: with HKDF, every tenant gets distinct ciphertext, but any holder of the master key can derive every tenant’s key. That is “fleet vs. outside” protection — it defends L2 bytes from anyone outside the serving fleet, not tenants from each other. Real cross-tenant isolation needs per-tenant keys plus tenant-to-node placement (otherwise a shared DaemonSet puts every key on every node anyway), which is why it is deferred rather than half-shipped.

5. Turning It On

Raw config — the serde is a sub-dict on whatever L2 adapter you already use:

{"type": "fs", "base_path": "/data/lmcache/l2",
 "serde": {"type": "aesgcm", "key_provider": "hkdf",
           "master_key_path": "/etc/lmcache/keys/master", "aes_bits": 128}}

6. What It Costs

Almost nothing where it matters. Encryption runs in the serde thread pool on the L1↔L2 leg — the transfer whose latency is already dominated by the storage backend — not on the L0↔L1 CUDA-IPC path that feeds inference. At ~4–8 GB/s/core for AES-128-GCM on AES-NI hardware, a single worker keeps pace with most L2 backends; max_workers scales it if yours is faster. Space overhead is a flat 29 bytes per chunk against chunks that are typically hundreds of kilobytes to megabytes.

7. Limitations and What’s Next

Stated plainly, because at-rest encryption is easy to over-claim:

• L1 and L0 are plaintext. This is L2-at-rest confidentiality only.
• Metadata is not hidden. The cache_salt (tenant identity) and a content-derived chunk_hash remain in the L2 object name, so a bucket observer can see which tenant stored what, and detect cross-tenant content overlap, without decrypting anything. Closing this — salting the chunk hash, pseudonymizing the salt at ingress — is a separate metadata-hardening step.
• HKDF is fleet-level trust. Per-tenant key isolation (KeyringKeyProvider + tenant-to-node placement) is future work; the factory currently rejects key_provider: keyring rather than pretending.
• Key rotation is manual today, requiring a new master key, invalidated cache, andre-fill.

The full design discussion lives in RFC #4127, and the shipped design doc in docs/design/v1/distributed/serde/aesgcm.md.

The aesgcm serde landed across three PRs: #4203 (threading ObjectKey through the serde interface), #4235 (the AES-GCM serde and HkdfKeyProvider), and #4274 (the operator’s spec.l2Backend.serde field). Try it, and tell us what your threat model needs next.