Find the cause, not a silent fallback
Use the first failing stage to distinguish download, loading, memory, and generation problems.
Reviewed 2026-09-08 · Beginner
Download or access failure
For a 401 or 403, check token scope and whether the repository requires accepting access terms. For a missing repository or artifact, verify the exact owner, name, revision, and filename. Do not retry indefinitely or switch models without saying so.
A partial shard set, insufficient disk space, or interrupted download must be resolved before trusting the resulting model. Keep the first useful error and check free disk space.
Out of memory: identify when it happens
If loading fails, the resident weights and startup allocations may exceed available memory. If a short prompt works but a long one fails, context, prefill workspaces, or concurrency are likely contributors.
Reduce one variable at a time: context, concurrent requests, or weight precision. Record actual free memory and other running processes. Do not call an offloaded or smaller-model retry equivalent to the original configuration.
Slow or incorrect output
For unexpectedly slow output, confirm the accelerator backend is active and check swapping, offload, thermal constraints, and concurrent workloads. Separate slow model download, slow first-token time, and slow generation.
For repeated tokens or nonsense, check the model’s chat template, tokenizer, supported architecture, quantization, and generation settings. Test a simple known prompt before adding retrieval or tools. Capture the model revision, runtime version, exact command, first error, and a non-sensitive reproducer.