What the KV cache is, and where it comes from
When you type a prompt into a chatbot, a large language model (LLM) turns your text into numbers, runs those numbers through many layers of matrix multiplications, and uses the result to predict which token should come next. It appends that token to the sequence, and runs the process again to predict the token after that, and keeps repeating until the numbers have become the response you read on the screen.
The model has to work with numbers because its core operation is multiplication—specifically, multiplying large matrices of numbers together. Your prompt is first divided into tokens, which may be whole words or pieces of words. Each token is then represented as a vector, a list of numbers that the model can operate on. Those vectors are multiplied by the model’s own learned numbers, or weights, across many layers of computation. The final result is a set of probabilities for the next token.
This process is called inference. The interface makes it seem like it’s just words flowing in and words flowing out, but the hardware underneath is moving and transforming large arrays of numbers. Those arrays occupy memory (e.g., your GPU), and the intermediate results produced along the way have to live somewhere until the next operation can use them.
Some of that data only gets used once, then it’s thrown away. But a lot of that intermediate data is needed again and again: the model needs it for every later token it generates. Rather than recompute that data from scratch each time, the model computes it once and stores it for reuse. That store is called the KV cache, and whether a system keeps it around or throws it away is the difference between inference that’s fast and affordable and inference that isn’t.
To understand why the cache exists (and why managing it becomes a problem of its own) we will follow the same path as the model: how your prompt turns into numbers the model can work with, why the model has to keep looking backward over what it’s already read, what specifically gets computed and cached as a result, and what it costs when that cache isn’t there to lean on.
From words to vectors
A model cannot operate directly on words. The first step is tokenization: splitting text into units drawn from the model’s vocabulary. A token might be a whole word, part of a word, punctuation, whitespace, or another recurring piece of text. Typically, a token is about ¾ of a word (roughly 4 characters); common words are a single token, rarer ones split into pieces.
Each token is represented by an integer ID. The model uses that ID to look up an embedding, a vector containing a fixed number of learned values. A vector is simply a list of numbers, but it gives the model a form it can transform with matrix multiplications. The token-to-embedding mapping is designed such that related words land near each other. “Cat” and “dog” end up close together in that space; “cat” and “car” don’t, even though they look similar as text.
The initial embedding is only the starting point. As the vector passes through the model’s layers, its representation changes based on the surrounding context. The token “bank,” for example, should contribute differently in “river bank” than in “bank account.” The mechanism that lets one token incorporate information from other tokens is called attention.
Why the model looks backward: attention
To predict what comes next, the model needs more than the latest token in isolation. It needs the context that came before it.
Consider the sentence “The cat sat down because it was tired.” When the model processes “it,” information associated with “cat” may be especially relevant. Attention gives the model a way to compare the current token with earlier tokens and combine the information that matters for the current step.
This happens throughout the model, not only for obvious references such as pronouns. At every layer, attention allows each token position to draw information from the context available to it. Different attention heads can learn to emphasize different relationships, such as nearby syntax, long-range references, formatting patterns, or other useful signals.
Query, Key, and Value
Attention works with three vectors derived from each token’s current representation, and together they work like a lookup: each token asks a question, offers a label to be matched against, and holds the content it hands over if it’s picked.
- Query — what this token is looking for
- Key — what this token has to offer, for other tokens comparing against it
- Value — the information this token passes along if it gets attended to
Take the sentence again: to decide how much “it” should attend to “cat,” the model compares “it”‘s Query against “cat”‘s Key. That’s the same kind of similarity measurement used to place tokens in vector space to begin with. Repeated against every earlier token, that comparison produces a set of attention weights: how relevant is each earlier token to me, right now? The token’s output is then built by blending the earlier tokens’ Values together, weighted by those scores. Keys decide how much to attend; Values are the content that actually gets used.
For the current token, the model compares its Query with the Keys of the tokens it is allowed to see. Those comparisons produce attention scores. After the scores are normalized into weights, the model uses them to blend the corresponding Values. Keys help determine where to attend; Values provide the information that gets combined.
These are learned numerical operations, not literal questions, labels, or stored definitions. The lookup heuristic is useful to conceptually understand the process: compare the Query against Keys, then retrieve a weighted combination of Values.
| TECHNICAL DETAIL — TRACING THE DIMENSIONS FROM TOKEN TO KV CACHE Every token starts as a single vector of length dmodel, the model’s hidden size (often a few thousand; Llama 3 8B uses 4096). Three learned weight matrices, WQ, WK, WV, each project that vector into a new one of length dk: q = xWQ k = xWK v = xWV Do this for every token in a T-token sequence, and each of Q, K, V becomes a matrix of shape (T, dk), with one row per token. In practice dk is split across multiple attention heads so each head works with a smaller slice, but the total amount stored per token doesn’t change. Now compound across layers: the model runs this whole process independently at every layer, producing one Key matrix and one Value matrix per layer, per sequence. The full KV cache is all of those matrices together, across every layer. |
| TECHNICAL DETAIL — FIXED WEIGHTS AND REQUEST-DEPENDENT TENSORS The main text treats Query, Key, and Value as individual vectors. In an implementation, a layer processes many token positions and attention heads together, so these values are stored as matrices or higher-dimensional tensors. Omitting the batch dimension, let:
Usually, dmodel = HQdhead. The input to one attention layer is: X ∈ ℝT × dmodel X is request-dependent data: its first dimension grows with the number of tokens being processed, and its values change from one prompt and layer to another.The layer also contains learned projection matrices: WQ ∈ ℝdmodel × (HQ·dhead) WK, WV ∈ ℝdmodel × (HKV·dhead) WO ∈ ℝ(HQ·dhead) × dmodel These are model weights. They were learned during training and remain fixed during inference. Their dimensions do not grow with the prompt. Some software stores the matrices transposed, but the mathematical operation is the same. Multiplying the request-dependent representations by the fixed weights produces the Query, Key, and Value activations: Q = XWQ, K = XWK, V = XWV After reshaping into heads: Q ∈ ℝT × HQ × dhead K, V ∈ ℝT × HKV × dhead In standard multi-head attention, HQ = HKV. In grouped-query attention, several Query heads share one Key/Value head, so HKV < HQ. This reduces the amount of K/V data that must be stored and read. |
| TECHNICAL DETAIL — THE ATTENTION FORMULA For one attention head, scaled dot-product attention is commonly written as: Attention(Q, K, V) = softmax( QKT ⁄ √(dk) ) · V QKT computes similarity scores between Queries and Keys. Dividing by √dk keeps those scores at a manageable scale; without it, softmax’s output would collapse almost entirely onto whichever token scored highest. Softmax converts the scores into weights that sum to 1, and multiplication by V produces a weighted combination of the Value vectors. The vectors come from learned projections of the representation x at that layer: q = xWQ k = xWK v = xWV Real models usually perform this operation with multiple attention heads at every transformer layer. |
Two phases: prefill and decode
Inference engines usually describe generation as two phases: prefill and decode.
During prefill, the model processes the prompt. Because all prompt tokens are already known up front, their computations can be performed in parallel within each layer: one matrix multiplication produces every prompt token’s Query, Key, and Value at once, and another produces every position’s attention scores against every other position, all in a single pass. (Each position is still restricted to seeing itself and earlier positions — enforced by something called a causal mask, which we’ll come back to in a moment) As the prompt moves through the model, the engine produces and stores a Key and Value for every prompt token at every layer. This creates the initial KV cache — the model’s understanding of everything in the prompt, saved for reuse.
During decode, the model generates one new token at a time. For each new token, every layer computes just that token’s Query, Key, and Value — a single small computation, not the batched pass prefill used. The Query attends to the Keys already stored for that layer, uses the corresponding Values, and continues through the rest of the model. Once the token has been processed, its new Key and Value are appended to the cache.
The selected token is then fed back into the model for the next step. This loop — generate a token, append it, and use the longer sequence to generate another — is called autoregressive generation. Every step of that loop reads the entire cache built so far, but only ever computes and appends one new Key and Value; nothing earlier is redone.
| TECHNICAL DETAIL — HOW ATTENTION CHANGES FROM PREFILL TO DECODE For one attention head, scaled dot-product attention is: Attention(Q, K, V) = softmax( QKT ⁄ √(dhead) ) · V Let TQ be the number of Query positions being processed now and TK the number of Key/Value positions available as context. Across all Query heads, the attention-score tensor has shape: HQ × TQ × TK During prefill, a layer may process a prompt of T tokens at once. Then TQ = TK = T, so the score tensor has the logical shape: HQ × T × T The causal mask blocks entries that would let a token use a future position. Optimized attention kernels may compute the scores in tiles instead of materializing this entire tensor in GPU memory. During one-token decode, the layer processes one new Query position (TQ = 1) while attending over the cached prefix (i.e., TK includes the entire cached prefix plus the new position). At a context length of T, the score tensor is approximately: HQ × 1 × T An important distinction is that KV caching avoids recomputing the earlier token representations, but the new Query still has to read and attend over a context that grows with the sequence. |
Why earlier Keys and Values can be reused
A model that generates text from left to right is causal: a token may use itself and earlier tokens, but it cannot use tokens that have not been generated yet. During attention, a causal mask blocks those future positions, which is what lets prefill compute every position’s scores in one parallel pass without letting any position peek ahead.
This one-way flow means that once the model has processed a particular prefix, extending that same prefix with another token does not change the Keys and Values already computed for the earlier positions. The new token needs to attend to them, but the model does not need to create them again.
That reusable collection of Keys and Values (across all cached token positions and transformer layers) is the KV cache. Queries are not retained in the same way because a Query is used to perform the lookup for a token at the moment that token is processed. The earlier Keys and Values are what later tokens continue to need.
Think of it like taking notes during a meeting. When someone speaks, you add a new line to the notes; you do not rewrite every earlier line before recording the new one. The KV cache lets generation extend the existing numerical record instead of reconstructing it at every step.
Reusing a cache beyond one generation
Within a single generation, reusing the KV cache is the normal decode path. Reusing it across separate requests is a different problem.
A chat interface may display one continuous conversation, but the backend often receives each turn as a new request containing the conversation so far (e.g., the original prompt, the model’s reply, and now your new message appended on the end). If the serving system still has the matching prefix cached—and can recognize and access it—it can skip prefill for those cached tokens and process only the new suffix. If the cache was discarded, evicted, or left behind on another engine, the same conversation history must be processed again.
The same pattern appears in many production workloads:
- Chat requests repeatedly include the same system prompt and conversation history.
- Agents make a series of calls that carry forward instructions, tool results, and intermediate state.
- RAG applications may reuse the same retrieved documents or other large context across related requests.
- Multiple users or requests may begin with an identical shared prefix.
In each case, the application may send many tokens that the model—or another model instance—has already processed. A reusable KV cache can turn that repeated prefill computation into a cache lookup and transfer instead.
| TECHNICAL DETAIL — WHAT DETERMINES KV-CACHE SIZE For a conventional transformer, the approximate KV-cache size is: 2 × L × T × HKV × Dhead × B where 2 accounts for Keys and Values, L is the number of transformer layers, T is the number of cached tokens, HKV is the number of Key/Value heads, Dhead is the size of each head, and B is the number of bytes used for each stored value. |
| TECHNICAL DETAIL — WORKED EXAMPLE — KV-CACHE SIZE FOR LLAMA 3 8B Llama 3 8B has a model width of 4096, 32 transformer layers, 32 Query heads, and 8 Key/Value heads. Each head therefore has width: dhead = 4096 ⁄ 32 = 128 For one layer, the projection matrices have the mathematical shapes:
Suppose prefill processes four tokens. Ignoring the batch dimension:
On the next decode step, the layer receives one new token representation:
The Query is used for that step and can be discarded. The new Key and Value are appended because future tokens will need them. Across all layers, an approximate KV-cache size for one sequence is: 2 × L × T × HKV × dhead × B where B is the number of bytes per stored value and the leading 2 accounts for both Keys and Values. With BF16 values, Llama 3 8B requires: 2 × 32 × 8 × 128 × 2 = 131,072 bytes per cached token, or 128 KiB per token per sequence. That is about 12.5 MiB for 100 tokens and 1 GiB for 8,192 tokens, before allocator overhead and subject to how the cache is sharded across devices. The important contrast is that the projection weights remain fixed regardless of context length, while the retained K/V tensors grow with every cached token. |
Why bother — the cost of not caching
Without a cache, producing the 100th token means recomputing Keys and Values for all 99 tokens that came before it. That’s extremely expensive to do at each step, and gets even more expensive as the conversation grows. That cost also compounds: the 10th token redoes 10 tokens’ worth of work, the 100th redoes 100 tokens’ worth, and the total work across a whole reply grows roughly with the square of its length.
With a KV cache, each earlier token’s Keys and Values are computed once. Decode can process only the newly appended token through the model and reuse the stored state for everything before it. This removes a large amount of redundant computation and is one of the basic optimizations that makes autoregressive generation practical.
Note that the cache does not make every decode step cost the same amount. As the sequence grows, the new Query still has more cached Keys and Values to read and attend over. A later token therefore has a larger attention context than an earlier one. What caching avoids is rerunning the model for all the old token positions simply to recreate a state the engine has already computed.
| TECHNICAL DETAIL — WHY THE COST GROWS WITH THE SQUARE OF THE LENGTH Without a cache, generating a reply of length T means that at step t you redo t−1 tokens’ worth of Key/Value work. Summed across the whole reply: Σ (t = 1 … T) (t − 1) ≈ T2 ⁄ 2 quadratic growth. A 100-token reply costs on the order of 5,000 of these recomputations. Stretch the reply to 1,000 tokens and the cost jumps to roughly 500,000: a 10x longer reply costing about 100x the work. With the cache, each step contributes exactly one new Key/Value computation, so total cost grows linearly with T instead of with its square. |
The memory tradeoff
The KV cache saves computation by holding on to intermediate results, but holding on to them consumes memory. You can think of the KV cache as the model’s working memory of the entire exchange; it only grows as the session continues. More specifically, the KV cache’s footprint grows with the number of cached tokens, transformer layers, KV heads, head dimension, and bytes per stored value. It also grows separately for every active sequence whose state cannot be shared.
The fastest place to keep this data is usually GPU memory. But the GPU must also hold model weights, temporary activations, communication buffers, and other runtime data. Under concurrent or long-context workloads, KV caches can consume enough memory to limit how many requests the engine can serve at once.
Inference engines therefore manage the cache under a fixed memory budget. They may allocate it in blocks, reclaim blocks from finished requests, or evict cached state when space is needed elsewhere. Once a useful cache is no longer available (and no copy exists outside the engine) the next matching request has to perform prefill again, paying the full computation cost.
That leaves a basic tradeoff:
- Keep more KV data in scarce GPU memory so it remains fast to reuse.
- Discard it to make room, then pay to recompute it when the same context returns.
Why LMCache exists
LMCache adds another option: treat the KV cache as reusable data that can live beyond the immediate GPU-memory allocation and, when useful, beyond a single request or inference-engine process.
At a high level, LMCache can move KV data through a storage hierarchy that includes GPU memory, CPU memory, local storage, and remote backends. When a later request reuses cached context, the serving system can retrieve the corresponding KV state rather than rebuilding all of it through prefill. This can reduce repeated computation, improve time to first token, and leave GPU memory available for more active work.
Retrieving a cache is not free, and it is not always faster than recomputing one. The result depends on factors such as how much context is reused, where the cache is stored, transfer bandwidth, and whether the lookup produces a hit. The systems-problem is therefore not simply whether to cache, but what to keep, where to keep it, when to move it, and how to make it available to the engine that needs it.
Those are the questions LMCache is designed to answer. The next post will move from the motivation to the architecture: how LMCache stores, transfers, coordinates, and serves KV data outside the lifecycle of a single inference request.
