MoE: active is not total
Understand expert routing without underestimating the weights that must be stored.
Reviewed 2026-09-08 · Advanced
A subset of experts works on each token
A mixture-of-experts model routes each token through selected expert networks. Active parameters describe the parameters involved in that token’s computation; total parameters include all experts and shared components.
Different tokens may select different experts. A model advertised with a small active count can therefore require far more weight storage than a dense model of that active size.
Budget total resident weights
Use total parameters or a verified full-model artifact for the weight budget. Do not calculate resident weights from active parameters alone.
Expert offload and expert parallelism can change where weights live and how much communication occurs. These are runtime deployment strategies, not a universal reduction factor. System RAM, accelerator memory, interconnect, and disk traffic must be considered separately.
Evaluate the actual deployment
Two MoE models with similar active counts can differ in total experts, shared layers, attention, and communication cost. Compare memory and speed using the exact quantization, runtime, and hardware topology.
A low active parameter count is not a tokens-per-second prediction. Measure prompt processing and generation separately, including concurrent load if you intend to serve multiple users.