The LMCache Coordinator: A Control Plane for Fleet-Wide KV Cache

LMCache’s multi-process (MP) mode gives every node its own KV cache server, but each server sees only itself. The coordinator is the control plane that makes them act as one cache: it builds a shared, fleet-wide view from cache events and lets pluggable policies act on it. Our goal is one KV cache for the whole fleet, for every inference workload: online serving, agents, and reinforcement learning (RL) rollouts. Policies adapt the cache to each workload, and the control plane scales from one cluster to many.

The coordinator ships in LMCache today. It is opt-in and stays out of the request path: if it goes down, inference keeps running on every node.

Motivation: from per-node KV cache to a fleet-wide control plane

Background: KV cache, offloading, and MP mode

Every LLM inference request starts with prefill: the model reads the whole prompt and builds up the state that every later token depends on. In full-attention layers that state is the keys and values of each token; in hybrid models, linear-attention and Mamba layers keep a fixed-size recurrent state instead. LMCache calls all of it the KV cache. Prefill is the expensive part, so inference engines like vLLM and SGLang keep this cache in GPU memory and reuse it whenever a new request shares a prefix, skipping prefill for that prefix and cutting time to first token (TTFT).

That reuse only works while the cache is still on the GPU, and contexts keep getting longer. Multi-turn chats, agents and agentic RL rollouts with long tool traces, and retrieval-augmented generation (RAG) prompts full of documents, all produce more KV than GPU memory can hold for long, so it is evicted before it can be reused. KV cache offloading keeps it alive by moving it to cheaper tiers, such as CPU DRAM, disk, or remote storage, and loading it back on reuse, which costs far less than recomputing it.

Offloading alone leaves one weakness: when the cache lives inside the engine process, a restart or crash of the engine can wipe it out. LMCache’s MP mode removes that tie by running the cache as a standalone server on each node, decoupled from the engine. Each MP server has two tiers: L1 in CPU DRAM, shared by every engine on the node, and L2 on local disk or S3 behind it. The cache survives engine restarts, cannot crash the engine, and keeps its CPU work off the inference path. For a gentle introduction to MP mode and how KV moves between the engine and the cache server, see Understanding LMCache MP Mode Transfer Paths.

But each server is still an island

MP mode solves caching within a node. A real deployment, though, is a fleet: tens of nodes, each with its own MP server, whether they serve users or generate rollouts for RL training. Each server is independent by design and knows only about itself.

RL training shows the cost first. Suppose an RL job samples sixteen responses from each prompt, spread over four nodes, each offloading only to CPU memory (L1). Node A prefills a long prompt and keeps its KV in L1. Moments later, nodes B, C, and D get samples of the same prompt, and each prefills it again from scratch, because none of them knows node A already holds it. The KV they need sits in a peer’s memory, one network hop away.

The same gap appears across the fleet. A few examples:

1. Lookup. Where in the fleet does a prefix live? It needs a fleet-wide index, not a per-node one.
2. Online scheduling and policies. While requests run, something has to decide where work goes and how its KV moves: which node still has room in L1, whether to load KV from a peer or recompute it, where to store new KV, and what to evict. Today each server makes these calls alone, seeing only itself.
3. Offline, proactive fleet-wide cache control. Outside the request path, an operator or an RL scheduler wants to promote a prompt’s KV into node C’s memory before a traffic shift or the next rollout batch, demote cold KV to cheaper storage, move KV from a busy node to an idle one, pin a hot system prompt so it is never evicted, or delete one tenant’s data. Today that means talking to every server one by one with manual scheduling.

Why a coordinator

Networking solved this kind of problem long ago by splitting the data plane from the control plane. Routers move packets fast; a separate control plane learns the whole topology and tells them where to send things. KV cache offloading has so far been almost all data plane: moving bytes between GPU, CPU, disk, and network as fast as possible. What is missing is the part that sees the whole fleet and decides.

That is the coordinator: one standalone process that builds a fleet-wide view of every MP server’s cache and uses it to enforce quotas, evict, pin, prefetch, and look things up across machines.

A process that sees the whole fleet could easily become its bottleneck, so we currently keep the coordinator out of the request path. A request that hits an MP server’s local cache is served without ever talking to it. The coordinator only advises: its view of the fleet can lag slightly behind, so a server checks with the server that holds the KV before using it. If the coordinator goes down, inference keeps running on every node, and only the fleet-wide features wait for it to come back.

The vision

Every node’s GPU memory, CPU memory, and storage should behave like a single pool, so KV computed anywhere can be found and reused everywhere, whether the fleet is serving users or generating rollouts for RL. The MP servers stay the data plane that moves the bytes; the coordinator becomes the programmable control plane that decides what to keep, where to put it, and when to move it. Those decisions should be policies anyone can write and swap, so new workloads, new hardware, and new ideas can change how the cache behaves without touching the engines.

Architecture: one stream in, plain HTTP out

The coordinator is currently a FastAPI process with a simple shape. Facts about the fleet’s caches flow in as one event stream. Decisions flow back out as ordinary HTTP calls to endpoints the MP servers already have. Everything in between is either a view (what is true) or a controller (what to do about it).

Every cache fact enters through the event gate, but fleet membership arrives separately: servers register when they start and send heartbeats while they run (POST /instances and heartbeat in the table below). Every action leaves as an HTTP call to a server’s existing endpoint. The controllers are the pluggable part.

Follow one KV chunk through the system:

1. A server reports it. When an MP server stores, reads, or deletes a chunk, it records a cache event and sends a batch of them to the coordinator about once a second.
2. The events travel directly over HTTP by default, or through Kafka in production.
3. The event gate admits them. It is the single entry point for every fact about the fleet’s caches.
4. The broadcaster fans them out to every component that consumes events.
5. Views turn events into a picture of the fleet: the registry (who is alive), the key directory (where each chunk lives), and usage (bytes per tenant and server). Views are shared by all controllers; state specific to one policy, like a least recently used (LRU) order, stays private to its controller.
6. Controllers act on that picture. Built-in ones handle eviction and prefetch; controllers are pluggable, so you can add or replace them without changing LMCache.
7. Decisions go out as plain HTTP to endpoints the MP servers already expose, such as DELETE /cache/objects or POST /cache/prefetches.

The split between views and controllers matters later: views are the shared picture every controller reads, and controllers hold the policy, which is the part we made pluggable.

Current features: what ships today

All of the features below are in LMCache v0.5.6 and later. They are opt-in: an MP server started without a coordinator URL behaves exactly as before.

Starting one takes a single command, and each MP server joins with two flags, one to register and one to stream its cache events:

lmcache coordinator --port 9300 \
  --checkpoint-path /data/coord.ckpt \
  --metadata-path /data/coord-meta.json
lmcache server ... --coordinator-url http://coordinator:9300 \
  --coordinator-event-reporting

Most of these features are small, because the views do most of the work. Quota enforcement keeps no fleet state of its own beyond the quotas: the eviction controller compares each tenant’s budget with the bytes the usage view already counts, and picks victims from an LRU order built from the same access events. It currently assumes a shared L2, such as one bucket mounted by every server, so a delete sent to any one server frees the bytes for everyone. A deployment with local-disk L2 would swap in a controller that sends each delete to the server that owns the copy, which is what the next section is about.

Pluggable design: controllers as plugins

The built-in eviction policy is a reasonable default, not the right answer for every deployment. One team wants an L1 budget as well as an L2 one. Another wants deletes sent to the exact server that holds a copy, because its L2 is local disk rather than a shared bucket. A third wants a placement policy nobody upstream has thought of yet. So we made the policy layer, the controllers, something you can add and replace without forking LMCache.

A controller is one class

You subclass the Controller marker, and the coordinator finds it by scanning a package. There is no list to edit in create_app and no field to add to a config class. Beyond that, a controller opts into extra roles just by having the right method:

Controllers can live outside the LMCache tree

Put your controllers in an ordinary Python package and name it at startup. Here a package called acme_controllers adds a controller that serves /acme/reaper, with its own setting:

lmcache coordinator --extra-config '{
  "controller_packages": ["acme_controllers"],
  "acme.reaper_interval": 10
}'

A controller can replace a built-in

Loading a new eviction controller is not enough, because the built-in one would still run and the two would fight over the same state and the same endpoints. So you switch the built-in off by name:

lmcache coordinator --extra-config '{
  "controller_packages": ["acme_controllers"],
  "disabled_controllers": ["FleetEvictionController"]
}'

Three small rules make swaps safe:

• Endpoints belong to their controller. /quota and /cache/pins come from the eviction controller’s own get_routers(). Disable it and they disappear, so a replacement can serve the same paths and existing clients keep working.
• State belongs to the component, not the class. Durable sections are named by what they hold (pins, quotas), so a replacement inherits the pins and quotas the old controller saved.
• Typos fail loudly. A package that does not import, or a disabled name that matches nothing, stops the coordinator at boot. An operator who thinks they disabled something is in a worse position than one who sees an error.

Controllers may only depend on views

A controller asks for the views it needs (registry, key directory, usage) and gets the same shared instances everyone else reads. It cannot reach another controller. Shared state is what views are for; if two controllers need each other’s policy, they were really one controller. This keeps an out-of-tree controller from breaking when a peer ships somewhere else. For the same reason, views are not pluggable: which views exist is the coordinator’s contract with every controller.

Fault tolerance: being wrong safely

A control plane fails in two ways. It can go down, or it can stay up and quietly believe something false. The first is easy to survive here, because the coordinator is off the request path: servers keep serving, and fleet features wait. The second is the dangerous one, so most of the design is about keeping the picture correct through restarts, crashes, and lost messages.

When a server restarts or dies

An MP server restart wipes its DRAM, so every L1 entry it reported is now a lie. The coordinator catches this with the incarnation number. When the first batch from a new incarnation arrives, the gate fences the old one: every consumer drops the L1 entries that server reported. L2 entries are deliberately kept, because those bytes sit on disk or in S3 and survive the restart. They leave only when a DELETE event says so.

A server that crashes and never comes back is caught by heartbeats instead. If it is silent for 30 seconds (checked every 10 seconds), the health loop removes it and fences its L1 entries the same way.

When events are duplicated or lost

Every server’s stream is ordered by a per-server sequence number, seq, and that per-server order is the only ordering the design needs. Each server is the only writer of its own facts, so there is no global clock and no cross-server tie-breaking. A batch with an old seq is a retry or a redelivery, and the gate drops it. Applying an event is also idempotent: storing the same chunk twice just overwrites it, and deleting something absent does nothing.

A skipped seq means events were lost. The gate flags that server’s slice as possibly stale and keeps going. With Kafka, loss is rare: the server’s producer buffers up to 64 MB and retries in order for up to five minutes, so a broker restart delays events instead of dropping them.

When the coordinator restarts

The coordinator writes two files, because its state comes in two kinds:

Derived state is large and changes constantly, so it is written periodically; operator intent is small and irreplaceable, so it is saved on every change. The checkpoint also stores the Kafka offset that produced its views, and that offset is the only read cursor, so after a restart the restored state and the resume point always agree. Taking a checkpoint briefly pauses ingest, so the saved views are consistent with each other.

Every persistence failure is survivable. A missing or corrupt checkpoint means a cold start, not a crash, and a failed write is retried on the next tick. Likewise, if one view fails on a batch, only that view misses it.

High availability today

Today, high availability means a fast, correct restart of a single instance, not a hot standby. A restart loses no events when the deployment has:

1. Kafka retention longer than any expected outage.
2. --checkpoint-path and --metadata-path on a persistent volume.
3. A liveness probe on GET /healthz.
4. In Kubernetes, run the coordinator with replicas: 1 and the Recreate update strategy. The default RollingUpdate briefly runs two coordinators, and two Kafka consumers in the same consumer group would each receive only part of the fleet’s events.

Observability

Like any service, the coordinator tracks request rates and latencies, for both the calls it serves and the calls it makes to MP servers. But those numbers alone miss its real risk. Everything it knows comes from the servers’ events, so if events are lost, late, or half-applied, it keeps answering and quietly answers wrong. Its observability therefore starts from the event stream and the work built on top of it, and asks four questions:

1. Is the view complete? Did any batches go missing, did a server admit to dropping events, and did any view fail to apply a batch? A failed apply means two views now disagree, and that is serious enough to alert an on-call engineer.
2. How old is it? How long does a stored chunk take to show up in the directory, and has a server stopped sending events while still heartbeating?
3. Are the loops alive? When did the health, checkpoint, and eviction loops last finish a pass?
4. What is each controller doing? Did it start, how much work is it doing, and what is it failing at?

A few rules keep this cheap and honest. Each component keeps one stats() snapshot, and both metrics and the status page read it, so they never disagree. Metric labels come only from fixed lists; per-server values are gauges keyed by instance_id that disappear when the server leaves, and cache keys or their hashes are never used as labels. And a scrape must never walk a whole index under a lock, because a slow scrape would stall ingest. Counts are kept up to date as events are applied instead.

Two things are deliberately not done. /healthz never fails just because the coordinator is degraded, because Kubernetes would restart it and throw away its state. And there is no readiness check that waits for warm-up, because it would block the very events needed to warm up.

Every controller reports on itself

Each controller has an up/down gauge, so one that fails to start is visible at once, and a controller with its own loop reports when that loop last finished a pass. Each role then adds its own metrics. Eviction counts sweeps, keys selected, keys skipped, and why (pinned, or no server can delete them), and delete requests sent or failed. Quota reports each tenant’s bytes used against its limit, pins report how many keys are protected, and prefetch tracks requests submitted, in flight, time to complete, and how many requested keys were actually found.

These metrics are named by role, not by class: whichever controller enforces eviction emits the eviction.*, quota.*, and pins.* metrics. A replacement may add labels and metrics but never rename them, so swapping in your own eviction controller keeps every dashboard and alert working.

Everything is exported as OpenTelemetry metrics under the lmcache_coordinator. prefix, scraped by Prometheus from GET /metrics or pushed to a collector with --otlp-endpoint. For people, GET /status shows the same snapshots on one page: per-server stream state, Kafka position, controller and loop health, and the last checkpoint.

Future directions: from knowing to deciding, from one coordinator to many

Today the coordinator mostly knows: where every chunk lives, how much each tenant uses, how full each server is. Its decisions are still simple, such as LRU eviction under a quota, or a prefetch someone explicitly asked for. A full control plane needs six capabilities: state, lookup, policy, action, scale, and reliability. The diagram above and the table below show where the coordinator stands on each today and where it goes next, and the rest of this section walks through the biggest shifts.

Smarter policy

Inference workloads reuse KV in different ways, and we expect the biggest gains ahead to come from policies built around them. Multi-turn chat and coding agents keep returning to long, growing contexts. Multi-agent workflows hand work between roles in patterns we can learn. RL rollouts sample the same prompt many times across many nodes, and RAG requests share documents at arbitrary offsets. Each pattern deserves its own placement, prefetch, and eviction strategy, and each can arrive as a controller reading the shared views, with no new data path. The plugin model and record-and-replay make this practical: a new policy ships as an out-of-tree controller, and it can be compared, tuned, or even learned against a real fleet’s replayed traffic before it touches production.

Taken one step further, people need not write these policies by hand. A self-evolving agent could own the policy layer: watch the fleet through the shared views and metrics, propose a new controller, evaluate it against replayed traffic, and promote it only when it beats the current one, then repeat as workloads shift. The coordinator already has the pieces such a loop needs: controllers as the unit of change, replay as a safe test bench, and observability as the score.

Richer actions

Smarter policies need more ways to act than delete, pin, and prefetch. Promoting KV into a node’s memory, demoting it to cheaper storage, and moving or duplicating it across nodes would let a policy reshape where the cache lives, not only trim it.

Cheaper lookups

As policies depend more on the coordinator, lookups move closer to the request path and event volume grows with the fleet, so the cost of each call starts to matter. Today the coordinator speaks plain HTTP with JSON, which is easy to read and debug. Alternatives such as gRPC offer typed contracts, streaming for events, and cheaper calls, and the MP servers already offer gRPC to engines, so the right protocol is worth revisiting as the fleet grows.

Growing with the fleet, and beyond it

As fleets reach thousands of nodes and inference spans both serving and RL training, one process will not hold every view or make every decision. The coordinator itself may also become distributed, or even hierarchical: a coordinator per rack or cluster making fast local decisions, a parent above them setting fleet-wide policy, and warm standbys taking over without a cold start. The same structure stretches across datacenters and clouds, where cooperating coordinators place and move KV with bandwidth and price as inputs. The event-driven design makes this possible: every view is derived from ordered per-server event streams, so state can be split, rebuilt, or moved by replaying streams instead of copying it.

Every step keeps the same shape as today: events in, shared views, pluggable policy, plain actions out. That shape is what lets the coordinator grow from a fleet-wide directory into the control plane for one KV cache across the whole fleet.

To try it, start with the user guide, Running the coordinator. To discuss the design or propose a controller, join the control-plane RFC, #4226.

References

Papers

• Yihua Cheng et al. LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference. arXiv:2510.09665, 2025.
• Rui Zhang, Chaeeun Kim, Liting Hu. A Policy-Driven Runtime Layer for Agentic LLM Serving. arXiv:2605.27744, 2026.

Blog posts

• Jiayue Chen, Tony Lin and the LMCache Team. Understanding LMCache MP Mode Transfer Paths: A Beginner’s Guide. LMCache Blog, June 2026.

Design docs (in the LMCache repository, v0.5.6 tag)

• Coordinator overview: REST surface, layout, lifecycle
• Cache-event ingest: event gate, sources, Kafka cursor
• Cache-event emission: the MP-server side and its sinks
• Key directory
• Usage tracking and eviction, warm prefetch, fleet memory usage
• Controllers and plugins
• Durable state
• Observability
• User guide: running the coordinator

Issues and pull requests (LMCache/LMCache)

• Control-plane RFC, #4226
• Backbone and membership: #3488, #3516
• Quota, usage, and eviction: #3630, #4310
• Warm prefetch: #3827
• Key directory from cache events: #4275, #4292
• Ingest layer and controllers: #4526
• Durable state: #4653, #4678
• Pluggable controllers: #4723, #4760, #4895
• Kafka transport: #5285, #5286, #5354, #5464
• Record and replay: #5312, #5345
• Observability: #4389, #4800, #5533