Local model field guide

Practical guides for choosing, sizing, running and measuring local models.

• Your first local model

  Go from a computer and a use case to a small, working local conversation.

• Choose a model, not just a size

  Read model names, licenses, formats, and evidence before committing to a download.

• Quantization without the guesswork

  Understand bit widths, file formats, and why two “4-bit” models can behave differently.

• Context length and the KV cache

  Why a model can load successfully, then run out of memory during a conversation.

• Attention and hybrid architectures

  Separate architectural memory behavior from the kernel that executes it.

• MoE: active is not total

  Understand expert routing without underestimating the weights that must be stored.

• Plan RAM, VRAM, and disk together

  Keep separate budgets for storage, system memory, accelerator memory, and headroom.

• Download the right model files

  Distinguish full models, shards, adapters, projectors, and calibration artifacts.

• Run a GGUF model with llama.cpp

  A local-first starting point for CPU, GPU, and hybrid inference.

• Run a model on Apple silicon with MLX

  A clean Python environment and a small MLX-compatible model on your Mac.

• Serve a model with vLLM

  Bring up a loopback-only API before tuning concurrency or opening network access.

• Before you expose a model server

  A working local endpoint is not a secured public service.

• Find the cause, not a silent fallback

  Use the first failing stage to distinguish download, loading, memory, and generation problems.

• Benchmark a workload you actually use

  Measure latency, throughput, peak memory, and quality without mixing incomparable runs.

• Use the sizeof.ai estimate API

  Reproduce a memory estimate and handle uncertainty explicitly in your own tools.

• Read a memory estimate honestly

  Know what is published, calculated, runtime-dependent, or still unknown.