Introducing the MP Coordinator

TL;DR

MP servers store and move KV caches. The coordinator helps the fleet know which KV caches already exist.

---

In a distributed inference fleet, a local KV cache miss does not always mean the KV cache does not exist. The exact KV cache a request needs may already have been computed on a different process, GPU worker, or node. Recomputing it wastes GPU cycles. Reusing it first requires an answer to one question: does this KV cache already exist somewhere in the fleet, and if so, where?

LMCache’s MP Coordinator answers that question. LMCache’s MP (multiprocess) mode already moves KV cache management out of individual inference processes and into a standalone server. The coordinator extends that idea across the fleet: it keeps a lightweight view of which LMCache servers are alive and what cache state they report. KV data never flows through the coordinator, so it does not add another storage tier. It holds only the control-plane metadata needed to discover and manage reusable KV cache across servers.

This post explains why that split between metadata and data matters, how the coordinator builds its view of the fleet, and how you can try it yourself.

From process-local KV to MP mode

Traditionally, an inference engine manages its KV cache inside the engine process itself. This is simple, but it also ties the lifetime and visibility of cached KV to that process.

That becomes increasingly limiting as serving systems scale out.

Consider a node running several inference workers. Worker A may have already computed a long system prompt, retrieved document, or conversation history. If the next request containing the same context reaches Worker B, Worker B cannot necessarily reuse A’s state. The computation exists, but the cache is effectively siloed.

LMCache’s MP mode changes this architecture.

Instead of embedding all cache management inside each inference process, LMCache can run as a standalone MP server. Multiple vLLM processes can connect to the same server, allowing them to share a common cache service while isolating cache management from the inference engine itself. This gives LMCache an independent memory-management layer and lets cache capacity, storage tiers, and serving processes evolve separately.

Conceptually, the architecture moves from:

to:

That removes one important boundary: the process boundary.

But a production fleet contains many nodes, and therefore many MP servers:

Now the same problem appears one level higher.

MP Server A may contain exactly the KV cache needed by a request arriving at Node B. How does the system know? That is the job of the MP Coordinator.

---

The coordinator is a directory, not a cache

The easiest way to think about the MP Coordinator is as a directory for the LMCache fleet.

Each MP server still manages its own cache and serves KV data. The coordinator maintains metadata describing the fleet: which servers are alive, what cache entries have been reported, and where those entries are located.

A simplified view looks like this:

The coordinator does not store or serve the KV caches themselves.

The coordinator is part of the control plane. Cache reads, storage operations, and peer-to-peer KV transfers remain in the MP server and storage layers.

That separation matters. Putting a centralized service directly in the KV data path would turn it into both a bandwidth bottleneck and a potential latency bottleneck. Instead, LMCache centralizes the comparatively small amount of information required to reason about cached state while keeping the much larger KV payloads on the data path where they belong.

Building a fleet-wide view

The coordinator builds this view in four steps.

1. Servers register and stay alive through heartbeats

An MP server configured with a coordinator URL registers itself with the coordinator and periodically sends heartbeats. The current default heartbeat interval is five seconds.

Registration is deliberately best-effort: an MP server keeps serving requests even if the coordinator is temporarily unavailable, so losing the control plane does not stop the data plane.

This gives the coordinator a continuously updated view of which cache servers make up the fleet.

2. Cache events describe how state changes

Knowing which servers exist is useful, but it does not tell us what they cache.

For that, MP servers can enable coordinator event reporting. Servers then report cache lifecycle events such as:

• STORE: a cache object was added,
• ACCESS: an existing cache object was used,
• DELETE: a cache object was removed.

These events can be delivered directly over HTTP or through Kafka. The coordinator consumes them and incrementally updates its view instead of repeatedly scanning every cache server.

This distinction is important when deploying the coordinator: registration alone gives the coordinator fleet membership; event reporting gives it cache placement information.

3. The Event Gate makes the event stream usable

Distributed systems inevitably produce retries, restarts, delayed messages, and duplicated events.

Before cache events modify the coordinator’s state, they pass through an Event Gate. The Event Gate tracks the event sequence and incarnation associated with each server, allowing the coordinator to reject duplicate events and events belonging to stale server incarnations.

This prevents, for example, an event from an old server process from resurrecting cache state after that process has already restarted.

4. The Key Directory maps cache objects to placements

Events admitted by the Event Gate update the coordinator’s Key Directory.

At a high level, the directory maintains a mapping like:

KV cache
    ├── MP Server A / L1
    └── MP Server C / L2

A placement contains information such as the owning LMCache instance, storage tier, backend, and object size. The directory also keeps indexes that associate cached chunks with their token information.

The result is a fleet-wide answer to a question that would otherwise require probing every cache independently:

Where have we already seen this KV cache?

---

Exact reuse—and reuse beyond prefixes

For normal KV lookup, LMCache chunks a token sequence and derives cache keys from those chunks. The coordinator’s directory lookup can take the token sequence, derive the corresponding keys for complete chunks, and return the placements currently known for those keys.

Prefix caching works well when the cached content starts at exactly the same position in a new request. But useful context is not always reused as a prefix. A retrieved document, for example, may appear at one position in one prompt and somewhere else in another.

LMCache’s CacheBlend mechanism targets this form of non-prefix reuse. When the coordinator is started with Blend lookup enabled, it maintains an additional index over reported chunk contents. A query can then discover cached chunks that occur anywhere inside the new token sequence, together with their original and new positions.

The broader idea is useful: once reusable KV spans an entire fleet, finding reuse can become richer than asking whether a single local prefix happens to match.

---

Why the directory is intentionally “soft state”

A natural question is whether the coordinator needs a strongly consistent database of every KV cache in the fleet.

LMCache deliberately does not build one.

LMCache’s design documentation describes the Key Directory as eventually consistent soft state. Its answers are placement hints that must ultimately be validated by the server owning the cache object.

That tradeoff follows directly from the semantics of a cache.

If metadata is briefly stale and a supposedly cached object has already disappeared, the system can fall back to another copy or recompute it. A stale directory entry can hurt cache-hit rate or add lookup overhead, but it must not change the generated result.

This also changes how failures are handled.

When an MP server disappears or restarts, the coordinator can efficiently fence the L1 placements belonging to that server. The implementation maintains a reverse index specifically so that removing one server’s L1 state does not require scanning the entire fleet-wide directory. L2 state can persist independently and is removed through its own cache lifecycle events.

The coordinator gives up transactional consistency in exchange for metadata that is accurate enough to capture reuse, without putting consensus on the critical path of every cache operation.

---

How LMCache’s P2P path uses the MP Coordinator

LMCache’s P2P path also uses the coordinator, but the coordinator is not currently the lookup service for every P2P cache miss.

For P2P, the coordinator supplies fleet membership. MP servers discover peers through it, establish connections with those peers, and perform the actual lookup and transfer directly between servers. Through the P2P transfer engine (NIXL by default), the receiving server reads the KV data directly from the peer’s memory, over RDMA where the network supports it, and the tensors never pass through the coordinator.

So there are two related functions:

Coordinator
    ├── Who is in the fleet?
    ├── What cache placements have been reported?
    └── What reusable chunks can be found fleet-wide?

MP servers
    ├── Own the actual cache
    ├── Validate cache availability
    └── Move KV data between storage tiers / peers

Keep this boundary in mind when you deploy: the coordinator helps LMCache find and manage reusable state, and every byte of KV data stays with the MP servers.

---

Performance: increasing the reuse domain matters

The coordinator is primarily a metadata service, so its benefit should not be measured as “the coordinator makes a lookup N× faster.” Its value comes from enabling the serving system to capture KV reuse that would otherwise be hidden behind process or machine boundaries.

Existing LMCache results illustrate why expanding that reuse domain matters.

In LMCache’s MP-mode evaluation with Qwen3-235B-A22B-Instruct-2507-FP8 on eight H100 GPUs, replacing vLLM’s in-process KV offloading (50 GB of host memory per rank, 400 GB in total) with one shared LMCache MP server of the same 400 GB reduced mean TTFT from 3.98 seconds to 0.29 seconds, roughly 13.7× lower. P99 TTFT fell from 13.55 seconds to 1.30 seconds, while mean decoding speed increased from 9.81 to 37.47 tokens/s.

Metric	vLLM in-process offload	MP mode	Improvement
Mean TTFT	3.98 s	0.29 s	~13.7×
P99 TTFT	13.55 s	1.30 s	~10.4×
Mean decoding speed	9.81 tok/s	37.47 tok/s	~3.8×

An earlier LMCache P2P experiment showed the same effect between inference instances. Two vLLM instances ran on separate GPUs of one host, and the second reused contexts the first had already cached. For the query round, mean TTFT was 0.490 seconds on the second instance against 2.028 seconds on the first, and round time was 7.964 seconds against 38.323 seconds, roughly 4× and 5× better. The authors called that workload contrived, with full reuse, and it predates the MP Coordinator architecture. Read these numbers as the value of cross-instance reuse, not as a coordinator benchmark.

The larger the useful reuse domain, the less often expensive prefill work needs to be repeated.

MP mode expands that domain across processes. Fleet-wide coordination makes it possible to reason about reuse across servers.

---

How to try it out

The quickest way to see the coordinator in action is to start an MP server with event reporting enabled and inspect the directory that the coordinator builds.

Use a current LMCache checkout and install it locally:

pip install -e .

Start the coordinator:

lmcache coordinator \
    --host 0.0.0.0 \
    --port 9300

Then start an MP server and connect it to the coordinator:

lmcache server \
    --host 127.0.0.1 \
    --port 5555 \
    --http-port 8080 \
    --l1-size-gb 20 \
    --eviction-policy LRU \
    --instance-id node-a \
    --coordinator-url http://127.0.0.1:9300 \
    --coordinator-event-reporting

The last flag is important. Without --coordinator-event-reporting, the server can register with the coordinator, but the coordinator does not receive the STORE, ACCESS, and DELETE stream required to build its fleet-wide key directory.

Now connect vLLM to the MP server using LMCacheMPConnector:

vllm serve <model> \
    --port 8000 \
    --kv-transfer-config '{
        "kv_connector":"LMCacheMPConnector",
        "kv_role":"kv_both",
        "kv_load_failure_policy":"recompute",
        "kv_connector_extra_config":{
            "lmcache.mp.port":5555
        }
    }'

After sending requests through vLLM, you can inspect the coordinator’s view of the fleet:

curl -s http://127.0.0.1:9300/instances \
    | python3 -m json.tool

and inspect reported cache entries:

curl -s \
  'http://127.0.0.1:9300/directory/keys?tier=l1&limit=20' \
  | python3 -m json.tool

To turn this into a real fleet experiment, start another MP server and vLLM instance with a different instance ID and point it at the same coordinator. Cache events from both servers will contribute to the same directory.

If you want to experiment with fleet-wide CacheBlend discovery, start the coordinator with:

lmcache coordinator \
    --host 0.0.0.0 \
    --port 9300 \
    --enable-blend-lookup

and keep event reporting enabled on the MP servers. For cross-node P2P transfer, the MP servers additionally need their P2P configuration and peer-accessible addresses; the coordinator handles peer membership while the actual lookup and KV transfer remain server-to-server.