Context length and the KV cache
Why a model can load successfully, then run out of memory during a conversation.
Reviewed 2026-09-08 · Beginner
The conversation has a memory cost
During autoregressive generation, an attention model can keep previously computed keys and values instead of recomputing them for every new token. This is the KV cache. It is not a database of facts and does not permanently train the model.
The context budget includes instructions, conversation history, retrieved text, tool output, and the generated answer. Reserve room for output rather than filling the entire supported context with the prompt.
A useful formula, with boundaries
For ordinary full-attention layers, an approximate KV payload in bytes is 2 × layers × KV heads × head dimension × tokens × bytes per cache value × independent sequences.
This formula is not universal. Sliding-window layers may stop growing at their window limit. MLA can use a different representation, and recurrent or state-space layers have different state. Runtime allocation, block padding, and workspaces add overhead.
Reduce memory deliberately
Shorten context or reduce concurrent sequences before assuming weights are the problem. A supported quantized cache may save memory but can affect quality or latency. Offloading can save accelerator memory while adding transfer cost.
Some engines preallocate a cache budget; others grow it as generation proceeds. A low idle allocation and a successful short prompt do not prove that the maximum workload will fit. Test a representative long prompt.