llama.cpp
5 min read
llama.cpp is a C/C++ inference engine for running large language models locally. Started by Georgi Gerganov as a plain-C port of Meta's LLaMA, it has grown into one of the most widely used backends for on-device and self-hosted LLM inference — Ollama, LM Studio, and many others wrap it.
Its design goal is dependency-free, portable inference: no Python, no CUDA required, runs on CPU with optional GPU acceleration, and squeezes large models onto consumer hardware through aggressive quantization. It is built on the author's own tensor library, ggml.
Why it matters
• CPU-first — runs usefully on a laptop with no GPU, using SIMD (AVX2/AVX512, NEON) and integer quantization. GPU is an optional accelerator, not a requirement.
• Quantization — 2-, 3-, 4-, 5-, 6- and 8-bit weight formats let a 70B model fit in tens of GB instead of hundreds. 4-bit (Q4_K_M) is the common quality/size sweet spot.
• Portable — single self-contained binaries, no runtime. Builds on Linux, macOS, Windows, Android, even in the browser via WebAssembly.
• Broad model support — LLaMA/Llama 2/3, Mistral, Mixtral (MoE), Qwen, Gemma, Phi, DeepSeek, and many more, plus multimodal vision models.
GGUF
GGUF (GGML Universal Format) is llama.cpp's model file format — the successor to the older GGML/GGJT formats. A single .gguf file bundles:
• the quantized tensor weights,
• the full model architecture / hyperparameters (layer count, head count, context length, rope settings…),
• the tokenizer (vocabulary + merges),
• and arbitrary metadata as key/value pairs (chat template, license, etc.).
Because everything is self-describing, a GGUF file is portable and version-tolerant — the loader reads the metadata rather than needing matching code. Most models on Hugging Face are distributed pre-converted to GGUF; otherwise convert_hf_to_gguf.py converts a Hugging Face checkpoint, and llama-quantize re-quantizes a GGUF to a smaller type.
Quantization types
Naming follows Q<bits>_<variant>:
• Q4_0, Q8_0 — legacy "type-0" round-to-nearest blocks.
• Q4_K_M, Q5_K_M, Q6_K — "K-quants", mixed precision per block, better quality at the same size. _S/_M/_L = small/medium/large variants.
• IQ2_XXS … IQ4_NL — "i-quants", importance-matrix–based, best quality at very low bitrates (needs an imatrix computed from sample data).
• F16 / BF16 — full half-precision, no quantization.
Rule of thumb: Q4_K_M for general use, Q5_K_M/Q6_K when you have headroom, Q8_0 for near-lossless, sub-4-bit only when memory-starved.
Key binaries
The build produces several tools (historically prefixed main, server…, now llama-*):
| Binary | Purpose |
|---|---|
llama-cli | Interactive / one-shot text generation from the terminal |
llama-server | HTTP server with an OpenAI-compatible API and a built-in web UI |
llama-quantize | Quantize a GGUF model to a smaller type |
llama-bench | Benchmark prompt-processing and token-generation speed |
llama-perplexity | Measure perplexity to evaluate quantization quality |
llama-imatrix | Compute an importance matrix for i-quants |
llama-embedding | Produce embedding vectors from text |
llama-tokenize | Inspect tokenization |
llama-gguf-split | Split/merge large GGUF files into shards |
Building
The build uses CMake. The interesting part is selecting a hardware backend at configure time.
shell
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
# CPU-only
cmake -B build
cmake --build build --config Release -j
# NVIDIA (CUDA)
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j
Backend flags:
• -DGGML_CUDA=ON — NVIDIA GPUs
• -DGGML_HIP=ON — AMD GPUs (ROCm)
• -DGGML_METAL=ON — Apple Silicon (on by default on macOS)
• -DGGML_VULKAN=ON — cross-vendor GPU via Vulkan
• -DGGML_SYCL=ON — Intel GPUs
• -DGGML_BLAS=ON — faster prompt processing via a BLAS library
Prebuilt binaries are published on the GitHub releases page, and it is packaged in many distros (llama-cpp / llama.cpp) and Homebrew.
Running models
llama-cli
shell
# One-shot prompt
llama-cli -m model.gguf -p "Explain quantization in one sentence."
# Interactive chat using the model's built-in chat template
llama-cli -m model.gguf -cnv
# Pull straight from Hugging Face
llama-cli -hf ggml-org/gemma-3-1b-it-GGUF
Common flags:
• -m <file> — model path (-hf <repo> downloads from Hugging Face).
• -p <text> — prompt; -f <file> reads it from a file.
• -n <N> — number of tokens to generate (-1 = until EOS).
• -c <N> — context size in tokens (0 = use the model's trained max).
• -ngl <N> — number of transformer layers to offload to the GPU. Set high (e.g. 99) to put the whole model on the GPU; lower to split between GPU and CPU RAM.
• -t <N> — CPU threads.
• --temp, --top-k, --top-p, --repeat-penalty — sampling parameters.
llama-server
llama-server exposes the model over HTTP and ships a web chat UI.
shell
llama-server -m model.gguf -c 4096 -ngl 99 --host 0.0.0.0 --port 8080
It provides:
• a built-in web UI at http://localhost:8080,
• an OpenAI-compatible API (/v1/chat/completions, /v1/completions, /v1/embeddings, /v1/models) so existing OpenAI SDK clients and front-ends like Open WebUI work unchanged,
• and native endpoints (/completion, /embedding, /tokenize, /health, /props, /slots) with extra features such as multiple parallel request slots and grammar-constrained output.
http
POST /v1/chat/completions
Content-Type: application/json
{
"model": "local",
"messages": [
{ "role": "user", "content": "Write a haiku about CPUs." }
],
"stream": true
}
Notable features
• Grammar-constrained / structured output — GBNF grammars (or a JSON Schema) force generation to match a syntax, guaranteeing valid JSON, enums, etc.
• Multimodal — vision models (LLaVA, Qwen-VL, Gemma 3, etc.) via an mmproj projector file, so images can be sent alongside text.
• Speculative decoding — a small draft model proposes tokens that the large model verifies, increasing throughput.
• KV-cache quantization — quantize the attention cache to fit longer contexts in less memory.
• Slots / parallel decoding — llama-server can serve several requests concurrently, batching them through one model instance.
• LoRA adapters — apply fine-tuned adapters on top of a base GGUF at load time.
Relation to other tools
• Ollama wraps llama.cpp with model management and a friendlier CLI/daemon, hiding GGUF and quantization details.
• LiteLLM can route OpenAI-compatible app traffic to llama-server, alongside hosted providers and other local/self-hosted AI backends.
• The underlying ggml tensor library is also reused by whisper.cpp (speech-to-text) and other projects from the same author.