Serve a model with vLLM

Match the installation to the accelerator

Follow the official installation path for your GPU, operating system, and driver. CUDA, ROCm, and other supported backends have different requirements. Do not assume a generic package install selects the correct combination.

Start in an isolated environment. This guide assumes vLLM is installed and the example architecture is supported. It does not provision a cloud machine or enable paid inference services.

Start on loopback

The official quickstart uses Qwen2.5-1.5B-Instruct. Binding explicitly to 127.0.0.1 keeps this initial HTTP listener local to the machine. This is a setup example, not a production security configuration.

The first launch downloads weights and may compile or warm up components. Read startup errors directly; do not hide a failed load by automatically selecting a different model.

vllm serve Qwen/Qwen2.5-1.5B-Instruct --host 127.0.0.1 --port 8000
# In a second terminal, after startup completes:
curl --fail-with-body http://127.0.0.1:8000/v1/models

Test chat, then concurrency

A model-list response proves the API is reachable, not that generation is correct. Send a short chat request and verify the model id, output, errors, and memory.

Then test realistic prompt lengths and simultaneous requests. Cache preallocation and batching change resource use. A single-user fit estimate must not be treated as a concurrency guarantee.

curl --fail-with-body http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"Qwen/Qwen2.5-1.5B-Instruct","messages":[{"role":"user","content":"Explain VRAM briefly."}],"max_tokens":64}'