Benchmark a workload you actually use

Measure more than tokens per second

Time to first token captures how long someone waits before the answer begins. Prompt processing measures prefill. Decode speed describes generated tokens after prefill. End-to-end latency also includes queuing and transport.

Report peak memory and error rate alongside speed. A high throughput number with failed requests or poor answers is not a successful deployment. No estimated memory figure on sizeof.ai is a measured performance benchmark.

Use a small repeatable protocol

Fix the model revision, artifact, runtime version, hardware, context, prompt, output limit, and concurrency. Warm up separately and label cold-start measurements. Repeat the run rather than selecting the fastest sample.

• Test a short prompt, a representative prompt, and a long prompt within supported limits.
• Separate cached-prefix and uncached runs; they measure different work.
• Use the same output-length policy and note early end-of-sequence behavior.
• Measure single-user latency and your intended concurrent load separately.
• Retain raw observations and report the median and tail latency when sample size is meaningful.

Use runtime tooling, then the real application

llama.cpp includes dedicated benchmark tooling; MLX LM documents generation and benchmark options; vLLM provides serving-oriented tools. Confirm flags with the installed version’s help.

A synthetic benchmark isolates compute but does not cover your full application. Follow it with real prompts and explicit correctness checks. Record environmental differences before comparing results from different people.