LMCache on Amazon SageMaker HyperPod: Disaggregated Prefill and Decode

Authors: Xuan Lu (AWS), Yihua Cheng (Tensormesh), Vinay Arora (AWS), Swapnil Palod (AWS)

Overview

LLM inference has two phases with opposite resource profiles: prefill processes the entire prompt in parallel and is compute-bound, while decode emits one token at a time and is memory-bandwidth-bound. When run on the same GPU, they contend — a single long prompt stalls token generation for every concurrent request, and chunked prefill reduces that interference without removing it. Customers running long-context, high-concurrency workloads see it as per-token latency spikes that no amount of serving-layer tuning fixes.

Disaggregated Prefill and Decode (DPD) splits the two phases onto separate GPU pools, eliminating contention entirely. It only pays off if the key-value (KV) cache can move between pools faster than recomputing it — otherwise you have traded interference for a transfer stall. LMCache supplies that transfer layer: its PD (prefill-decode) backend orchestrates a GPU-to-GPU handoff over NIXL, which on AWS leverages Elastic Fabric Adapter (EFA) RDMA. On Llama 3.3 70B Instruct, this delivers up to 66% lower per-token latency and 64% higher output throughput — the rest of this post shows how.

HyperPod Inference is the serving stack for Amazon SageMaker HyperPod — a Kubernetes operator that turns a custom resource into a complete deployment: model-server pods, intelligent router, load balancer, and observability. The worker images ship with vLLM, LMCache, NIXL, and the EFA libfabric provider pre-integrated, making DPD deployable without assembling the stack yourself.

In this blog we show how LMCache’s PD backend is wired on HyperPod Inference. On Llama 3.3 70B Instruct, DPD holds per-token latency flat as concurrency rises — up to 66% lower than a colocated baseline — while improving output throughput by up to 64%.

LMCache PD Backend over NIXL and EFA

LMCache runs inside the vLLM engine as a KV connector (LMCacheConnectorV1), in one of two PD roles:

• Sender (prefiller) — computes KV cache, stages it in a pinned GPU buffer, and pushes it to the target decoder over NIXL.
• Receiver (decoder) — pre-registers a GPU buffer, accepts the incoming KV cache, and begins generation from the loaded state without executing prefill.

The transfer path is four layers, each with a distinct job:

1. LMCache PD backend — the orchestration layer. Issues the put on the prefiller and the retrieve on the decoder, splits the KV cache into chunks, and signals completion so the decoder knows when it can start generating.
2. NIXL — transport abstraction. Registers GPU buffers so they can be addressed remotely and selects the appropriate RDMA operation for the peer. NIXL supports several backends; on HyperPod, it targets libfabric for EFA.
3. libfabric — user-space networking API. Drives the EFA hardware directly — the application talks to the NIC’s queues without traversing the kernel network stack, eliminating syscalls and buffer copies.
4. EFA — the hardware. It moves the bytes as GPU-Direct RDMA, reading and writing GPU memory directly rather than staging through host memory.

With 3,200 Gbps of EFA bandwidth on ml.p5.48xlarge, transfers complete in single-digit milliseconds — small enough to disappear next to prefill compute. 

An intelligent router sits in front of both pools, applying a token threshold (default 4,096) to decide whether a request takes the disaggregated path or runs end-to-end on a decoder. One endpoint therefore serves mixed long and short traffic with no client-side logic.

Benchmark Setup

We ran two hardware configurations, both 1 prefiller + 1 decoder across two nodes, against a colocated baseline on a single node.

Benchmarks were run with genai-bench v0.0.4:

genai-bench benchmark \

    --api-backend openai \

    --api-base http://<router>:8081 \

    --api-model-name /opt/ml/model \

    --model-tokenizer <tokenizer-path> \

    --task text-to-text \

    --traffic-scenario "D(4096,256)" \

    --max-requests-per-run 50 \

    --num-concurrency 8 --num-concurrency 16 --num-concurrency 32 --num-concurrency 64

The baseline uses the same command pointed directly at the colocated worker.

Benchmarking Performance

To measure the impact of disaggregated prefill-decode, we benchmarked a 1P:1D DPD deployment against a colocated baseline serving Llama 3.3 70B Instruct. The charts below show the percentage improvement DPD delivers over the colocated baseline on two instance families. Higher bars indicate larger DPD advantage.

Across both hardware configurations, DPD delivers consistent gains on per-token output latency (TPOT), end-to-end latency, and throughput as concurrency grows:

• Per-token latency stays flat under load.
  • DPD isolates decode from prefill interference, keeping TPOT constant regardless of concurrent long-context requests. Improvement ranges from 22% (C=8) to 66% (C=64) on H100, and 28% (C=8) to 48% (C=32) on H200.
• Throughput scales with concurrency.
  • The dedicated decoder runs at full CUDA graph efficiency without prefill interruption. Output throughput improves up to 35% on H100 and up to 64% on H200 at higher concurrency levels.
• End-to-end latency improves at P50.
  • The cumulative TPOT savings across output tokens outweigh the KV transfer cost. E2E P50 improves 14-32% on H100 and 29-41% on H200.

DPD does introduce a modest increase in time-to-first-token (TTFT) — on the order of single-digit milliseconds — due to the KV cache transfer over EFA RDMA. For streaming workloads where consistent per-token delivery matters more than initial response time, this tradeoff is favorable. The conditional routing threshold (default 4,096 tokens) ensures short requests bypass disaggregation entirely, avoiding transfer overhead where it would not pay off.

Transfer Flow

When a request above the routing threshold arrives:

1. The router tokenizes the prompt, decides to disaggregate, and selects a decoder.
2. It sends the prompt to a prefiller together with that decoder’s NIXL endpoint metadata.
3. The prefiller computes the KV cache. LMCache copies it into the PD buffer and pushes it to the decoder over NIXL and EFA, overlapping compute with transfer.
4. The router forwards the request to the same decoder.
5. The decoder’s LMCache retrieves the KV cache from its buffer into vLLM’s paged KV cache and begins generation, executing no prefill.
6. Below the threshold, steps 2 and 3 are skipped: the router sends the request straight to a decoder for local chunked prefill, avoiding transfer cost where it would not pay off.

When DPD Helps Most

DPD provides the greatest benefit when:

• Prompts are long: inputs regularly above roughly 4K tokens, where prefill runs long enough to disturb in-flight decode.
• Concurrency is high: many simultaneous requests make interference continuous rather than occasional. Our benchmarks showed the strongest benefit at concurrency 32.
• Traffic is mixed: conditional routing serves long and short prompts from one endpoint with no client-side logic.
• The model is large: tecomputation cost scales with model size, so transferring KV cache beats recomputing it by a wider margin.

A colocated deployment remains the better choice for batch or offline workloads optimizing for TTFT, for low-concurrency deployments, and for short-prompt-only traffic.

Summary

DPD delivers decode isolation: per-token latency stays flat as concurrency rises, while a colocated deployment degrades. On Llama 3.3 70B Instruct, that translates to up to 66% lower per-token latency and up to 64% higher output throughput — with the margin widening exactly where colocated serving struggles most: many concurrent long-context requests. The tradeoff is a modest TTFT increase for disaggregated requests, which conditional routing eliminates for short prompts. If your production traffic includes long contexts at high concurrency, DPD on SageMaker HyperPod is ready to deploy.

For deploying DPD on HyperPod, see Disaggregated Prefill and Decode for HyperPod inference and the AWS blog walkthrough. For more about SageMaker HyperPod, visit the Amazon SageMaker HyperPod documentation.