Sending a KV cache across the Internet can now be faster and cheaper than computing it again. For DeepSeek-V4-Pro with about 100,000 tokens of context, the compressed KV cache is about 1 GB. Moving it over a fully utilized 10 Gbps link takes about 0.8 seconds and costs roughly $0.09 in AWS egress. Recomputing it on an AWS 8×B200 cluster takes about 15 seconds and costs $0.47.
In “An Internet for the KV Cache: Rethinking Classical Infrastructure Boundaries in the LLM Inference Age,” a paper posted to arXiv on August 2, 2026, Siddhant Ray, Nick Feamster, and Junchen Jiang of the University of Chicago propose an Internet for the KV cache: infrastructure that treats cached inference state as content to distribute, placing and moving it across storage tiers, regions, and cloud providers. Today that state mostly stays inside one serving system, datacenter, or cloud. Agents, retrieval, tool calls, and multi-turn conversations revisit overlapping context again and again, so reusing its KV cache saves prefill work wherever the next request runs.
Whether to store or recompute has no single answer. The model architecture, the infrastructure’s prices, and how long the application waits before reusing the context decide it together. This post, adapted from the paper, explains that tradeoff and what still needs to be built.
Why revisit the infrastructure now?
Three directions are changing the economics of inference:
- Model optimizations, such as mixture-of-experts execution and sparse attention, reduce computation or the historical state that inference must access.
- In KV cache optimizations, blending and editing expand reuse, compression shrinks cached state, and memory compaction reduces the context that needs to be retained.
- System optimizations, such as prefill–decode (P/D) disaggregation, scheduling, offloading, and faster interconnects, improve utilization and movement across GPU, CPU, storage, and network tiers. LMCache provides building blocks for this movement.
For 100,000 tokens, the KV cache shrinks from 9 GB with DeepSeek-V3.2 to 1 GB with DeepSeek-V4-Pro, approximately a 9× reduction, using its compressed cache format. On the compute side, NVIDIA Rubin delivers 50 PFLOPS of NVFP4 inference compute, compared with 15 PFLOPS for Blackwell Ultra, a 3.3× increase.
These advances change both sides of the decision: how expensive a KV cache is to compute, and how expensive it is to store and move. Infrastructure needs to make those decisions together.
When transferring context beats recomputing it
The example in the opening uses DeepSeek-V4-Pro (1.6T-A49B: about 1.6 trillion total parameters, 49 billion active), whose full-precision, compressed KV cache for about 100,000 tokens occupies about 1 GB. The transfer figures assume a fully utilized 10 Gbps Internet link and the AWS Internet-egress rate; the recompute figures assume an AWS 8×B200 cluster. Under these assumptions, transfer is approximately 19× faster and 5.2× cheaper.
This comparison uses transfer time and egress cost versus prefill time and compute cost; storage costs are part of the broader decision below. Transfer pricing and GPU pricing determine the economics. The AWS prices in this post come from the paper’s analysis and are examples for illustration only. They are not a recommendation of any provider or configuration, and they do not describe a supported LMCache deployment.
For an agent that revisits context between tool calls, the time until reuse creates an opportunity to fetch state before it is needed.
The KV cache becomes a shared inference resource
A KV cache represents context that a model can use during generation. Keeping that representation available lets serving systems reuse work across requests and workflows.
This perspective connects to the LMCache post “Stop Calling It KV Cache: It’s Something Much Bigger”. The KV cache becomes a storage, communication, and scheduling primitive, with systems such as LMCache, TensorRT-LLM, llm-d, and NVIDIA Dynamo exposing parts of that capability.
Store or recompute? Several answers can be right
The decision compares CacheCost, the cost of storing and transferring reusable state, with PrefillCost, the cost of recomputing it.
At break-even, two ratios are equal. The model-side ratio is prefill computation divided by KV cache size. The infrastructure-side ratio is storage and transfer costs relative to GPU compute cost, and it includes the application’s reuse timing. Storing is cheaper when the infrastructure-side ratio is below the model-side ratio, so changing the model, provider, or time until reuse can change the decision.
| # | Model | Infrastructure | Reuse interval (K) | Infrastructure-side ratio | Model-side ratio | Decision |
|---|---|---|---|---|---|---|
| 1 | DeepSeek-V4-Pro | AWS B200 + S3 bucket | 1 day | 0.436 | 11.410 | Storing cheaper |
| 2 | DeepSeek-V4-Pro | AWS B200 + S3 bucket | 1 month | 13.083 | 11.410 | Recomputing cheaper |
| 3 | DeepSeek-V4-Pro | GCP B200 + GCS | 1 day | 0.670 | 11.410 | Storing cheaper |
| 4 | DeepSeek-V4-Pro | Azure B200 + Blob Hot LRS | 1 month | 5.512 | 11.410 | Storing cheaper |
| 5 | DeepSeek-V4-Pro | GMI B200 + high storage | 1 day | 20.250 | 11.410 | Recomputing cheaper |
| 6 | DeepSeek-V4-Flash | AWS B200 + S3 bucket | 1 day | 0.436 | 4.325 | Storing cheaper |
| 7 | DeepSeek-V4-Flash | AWS B200 + S3 bucket | 1 month | 13.083 | 4.325 | Recomputing cheaper |
Table 1: Storage versus recomputation across models, infrastructure, and reuse settings. Source: Ray, Feamster, and Jiang, arXiv:2608.01526v1, Table 1.
For DeepSeek-V4-Pro on AWS, moving from one day to one month reverses the result. At one month, switching to the Azure configuration reverses it again. Partial recomputation and blending add further choices between these two extremes.
Make expensive misses less frequent
Prefix caching captures exact shared prefixes, but changes to the input can prevent reuse of later cached state. A high hit rate can still hide expensive long-context misses.
Reducing both the frequency and the cost of misses matters. Reusing non-prefix chunks through techniques such as CacheBlend and Cache-Craft broadens the reusable context, including when retrieved chunks appear in different orders.
Build for the next generation of workloads
Longer-running, interconnected agents will need context across workflows and environments. Internet video delivery offers a useful parallel: advances in distribution infrastructure can enable applications that later depend on it at scale.
An Internet for the KV cache
Today’s cloud boundaries shape placement
Cloud architecture and pricing encourage keeping storage near compute. Region and availability-zone boundaries affect latency and transfer charges, as shown in the AWS transfer-cost overview and Google Cloud network pricing.
Systems such as SkyPilot and SkyServe make compute placement more flexible. KV cache distribution also needs fine-grained state discovery, prefetching, and recomputation fallback.
Store anywhere, use everywhere
The proposed architecture would place and move KV caches across storage tiers, regions, and providers according to expected demand.
Agent workflows create useful flexibility: a context can be reused after seconds, minutes, or hours. State needed soon can stay near the GPU. State needed later can move to cheaper, more distant storage and be prefetched before reuse.
A control plane would discover available state, estimate bandwidth and delivery deadlines, and choose whether to transfer, replicate, partially fetch, or recompute that state. If delivery becomes too slow or costly, the system can recompute the state and update its placement policy.
The network becomes an input to inference decisions: bandwidth, latency, congestion, and price help determine which action is best.
What still needs to be built?
Realizing this architecture requires progress in three areas:
- Network-aware movement needs transfer costs and delays to be visible, plus transport and routing mechanisms that deliver KV cache predictably across infrastructure boundaries.
- Adaptive infrastructure and control planes need to combine discovery, placement, compression, prefetching, scheduling, and recomputation. Delayed transfers or failures should trigger graceful fallback and revised placement.
- Coordination across providers, regulators, and users needs to define ownership, pricing, privacy, correctness, and failure responsibilities. Standardization efforts similar to the work of the Internet Engineering Task Force (IETF) would support interoperable transfer and security protocols, including user-controlled storage.
Where this leads
Internet-scale KV cache management needs new protocols, abstractions, and control planes. LMCache and other serving systems supply useful building blocks. Realizing the broader vision requires coordinating compute, storage, and networking around reuse.
The central proposal: place reusable inference state wherever storage, delivery, and recomputation together best meet the workload’s cost and latency needs.
Further reading from industry: Anthropic prompt caching, reducing KV cache bottlenecks with NVIDIA Dynamo, Google Cloud external KV cache on Managed Lustre, and AWS GPUDirect with FSx for Lustre and TurboQuant.
Adapted from Ray, Feamster, and Jiang, “An Internet for the KV Cache: Rethinking Classical Infrastructure Boundaries in the LLM Inference Age,” arXiv:2608.01526v1, licensed under CC BY 4.0. Siddhant Ray is a graduate student at the University of Chicago.
