Run a GGUF model with llama.cpp

Install for your hardware

Use the official llama.cpp installation or build guide for your operating system and accelerator. A binary without your intended backend may run on CPU even when a GPU is present.

Confirm the installed version and available commands before downloading a model. The current upstream README uses the unified llama CLI; older releases commonly expose llama-cli and llama-server instead. Use the help for your installed release, not a mixture of examples from different versions.

llama --help
llama cli --help

Run the official small-model example

The upstream quickstart provides this Hugging Face GGUF example. It downloads model files, so check free disk space and any network charges first. Review the repository and license before use.

The model is an installation smoke test, not a recommendation for every task. Confirm sensible output and the active hardware backend in the logs before moving to a larger model.

llama cli -hf ggml-org/Qwen3.5-0.8B-GGUF

Scale one setting at a time

Choose a compatible GGUF quantization, then configure context and GPU offload using your release’s documented options. Record the selected file rather than only the repository.

For an API server, follow the upstream server guide and bind it to loopback for the first test. Never assume an example server is safe to expose publicly. Use the serving-security checklist before sharing it.