Attention and hybrid architectures
Separate architectural memory behavior from the kernel that executes it.
Reviewed 2026-09-08 · Advanced
Read the layer topology
Full attention, sliding-window attention, and recurrent state do not retain history in the same way. Hybrid models combine layer types, so applying a full-attention formula to every layer can misrepresent memory.
Grouped-query attention shares key/value heads across query heads. The number of KV heads, not just the number of query heads, matters for a standard cache estimate. MLA introduces a representation choice whose real allocation depends on runtime support.
An attention backend is not a new architecture
FlashAttention and PyTorch SDPA describe implementations of attention computation. Optimized kernels reduce intermediate memory traffic; they do not magically remove model weights or all retained conversation state.
Support depends on model structure, precision, hardware, and installed packages. A configuration flag is not evidence that the desired kernel is active. Inspect startup logs and measure the target workload.
Treat a lower bound as incomplete evidence
sizeof.ai exposes published topology and separates safe ordinary estimates from runtime-specific lower bounds. Do not convert an unknown recurrent-state allocation into zero.
For unfamiliar architectures, use the exact runtime release’s support documentation, then measure peak allocation during prefill and generation. Record the cache mode with the result so another person can reproduce it.