Quantization without the guesswork
Understand bit widths, file formats, and why two “4-bit” models can behave differently.
Reviewed 2026-09-08 · Beginner
What fewer bits actually change
Quantization stores model values at lower precision to reduce memory use. It can change accuracy and runtime speed; neither quality loss nor speedup is a fixed percentage.
As a first approximation, weight bytes are parameter count × bits per weight ÷ 8. For a hypothetical 8-billion-parameter model at exactly 4 bits, that is 4 billion bytes, or about 3.73 GiB. Real artifacts include scales, metadata, mixed-precision tensors, and other overhead.
Bits are not a compatibility format
GGUF, MLX quantizations, AWQ, and GPTQ are not interchangeable simply because they use a similar bit width. The runtime needs to support the architecture, file format, quantization method, and target hardware together.
A filename such as Q4_K_M describes a particular quantization scheme, not a guarantee that every tensor uses exactly four bits. Use the actual full-model artifact size when possible rather than inferring it from the label.
Choose by measured quality
Start with a supported moderate quantization, then test your real prompts. If quality is insufficient, try a higher-precision variant or a different model. If memory is tight, first check whether you have unnecessarily large context or concurrency.
Weight quantization and KV-cache quantization are separate controls. Changing one does not automatically change the other. Extreme compression deserves especially careful evaluation.