Knowledge Base
whisper.cpp
5 min read

whisper.cpp is a lightweight C/C++ inference engine for OpenAI Whisper-style automatic speech recognition (ASR). It is part of the same ggml ecosystem as llama.cpp, with the same philosophy: portable, dependency-light, quantized local inference that can run on CPUs, Apple Silicon, GPUs, mobile devices, WebAssembly, and small edge machines.

Use it when you want local/offline speech-to-text, transcription sidecars, voice assistants, meeting transcription, command recognition, or an OpenAI-like local /audio/transcriptions backend.

Why it matters

• Local speech-to-text — runs fully offline; useful for privacy-sensitive audio and voice interfaces.
• Portable — supports macOS, Linux, Windows, iOS, Android, Java, WebAssembly, Raspberry Pi, and Docker.
• Hardware acceleration — CPU-only works; also supports Apple Metal/Core ML, CUDA, ROCm/HIP, Vulkan, OpenVINO, Ascend NPU, and more.
• Quantization — smaller ggml model files reduce memory/disk use.
• Streaming-ish examples — includes microphone/real-time examples such as whisper-stream and voice-command examples.
• HTTP server — includes whisper-server, an HTTP transcription server with OpenAI-like API routes.
• VAD support — can use voice activity detection to process only speech segments.

Model sizes

Common converted ggml model names:

ModelTypical use
tiny, tiny.enFastest, lowest accuracy; wake-word-ish or low-resource experiments.
base, base.enGood local default for quick transcription.
small, small.enBetter accuracy while still practical on laptops.
medium, medium.enHigher accuracy, more memory/latency.
large-v3, large-v3-turboBest quality; use GPU/Metal/Core ML when possible.

The .en models are English-only and usually faster/more accurate for English. Multilingual models support transcription and translation across languages.

Build and quick start

shell
git clone https://github.com/ggml-org/whisper.cpp.git
cd whisper.cpp

# Download a converted model
sh ./models/download-ggml-model.sh base.en

# Build
cmake -B build
cmake --build build -j --config Release

# Transcribe sample audio
./build/bin/whisper-cli -m models/ggml-base.en.bin -f samples/jfk.wav

whisper-cli expects 16-bit WAV input in many common examples. Convert arbitrary audio with FFmpeg if needed:

shell
ffmpeg -i input.mp3 -ar 16000 -ac 1 -c:a pcm_s16le output.wav

Useful commands

shell
# Transcribe
./build/bin/whisper-cli -m models/ggml-base.en.bin -f audio.wav

# Translate to English instead of transcribing original language
./build/bin/whisper-cli -m models/ggml-small.bin -f audio.wav --translate

# Print confidence colors
./build/bin/whisper-cli -m models/ggml-base.en.bin -f audio.wav --print-colors

# Word-ish timestamps by limiting segment length
./build/bin/whisper-cli -m models/ggml-base.en.bin -f audio.wav -ml 1

# Experimental tinydiarize speaker turns
./models/download-ggml-model.sh small.en-tdrz
./build/bin/whisper-cli -m models/ggml-small.en-tdrz.bin -f audio.wav -tdrz

Hardware acceleration

Build flags follow the newer ggml naming style:

shell
# Apple Silicon / Metal
cmake -B build -DGGML_METAL=1
cmake --build build -j --config Release

# NVIDIA CUDA
cmake -B build -DGGML_CUDA=1
cmake --build build -j --config Release

# AMD ROCm/HIP
cmake -B build -DGGML_HIP=1
cmake --build build -j --config Release

# Vulkan
cmake -B build -DGGML_VULKAN=1
cmake --build build -j --config Release

# OpenBLAS CPU acceleration
cmake -B build -DGGML_BLAS=1
cmake --build build -j --config Release

On Apple Silicon, Core ML can accelerate the encoder through the Apple Neural Engine, but it requires generating a Core ML encoder model and building with -DWHISPER_COREML=1.

Voice activity detection

whisper.cpp supports VAD to skip non-speech regions. This can improve latency/cost for long recordings with silence.

shell
./models/download-vad-model.sh silero-v6.2.0

./build/bin/whisper-cli \
-m models/ggml-base.en.bin \
-f audio.wav \
--vad \
-vm models/ggml-silero-v6.2.0.bin

Common knobs:

• --vad-threshold
• --vad-min-speech-duration-ms
• --vad-min-silence-duration-ms
• --vad-max-speech-duration-s
• --vad-speech-pad-ms

HTTP server

whisper-server exposes transcription over HTTP and is the easiest way to wire whisper.cpp into an app or AI gateway.

shell
./build/bin/whisper-server \
--host 127.0.0.1 \
--port 8080 \
-m models/ggml-base.en.bin

The server is useful as a sidecar for systems like LiteLLM, voice assistants, and internal media pipelines. Treat it as a speech-to-text service; LiteLLM can centralize OpenAI-compatible AI calls, but an adapter may be needed if your app expects exact OpenAI /v1/audio/transcriptions semantics.

Docker

The upstream project publishes Docker images such as:

• ghcr.io/ggml-org/whisper.cpp:main
• ghcr.io/ggml-org/whisper.cpp:main-cuda
• ghcr.io/ggml-org/whisper.cpp:main-vulkan

Example:

shell
# Download model into a persistent host directory
docker run -it --rm \
-v /path/to/models:/models \
ghcr.io/ggml-org/whisper.cpp:main \
"./models/download-ggml-model.sh base /models"

# Transcribe local audio
docker run -it --rm \
-v /path/to/models:/models \
-v /path/to/audio:/audio \
ghcr.io/ggml-org/whisper.cpp:main \
"whisper-cli -m /models/ggml-base.bin -f /audio/input.wav"

Wiring patterns

Local voice assistant

flowchart TD
    Mic[Microphone]
    VAD[VAD]
    STT[whisper.cpp]
    LLM[LiteLLM chat model]
    TTS[Piper or other TTS]
    Speaker[Speaker]

    Mic --> VAD --> STT --> LLM --> TTS --> Speaker

Meeting/audio transcription

flowchart TD
    Audio[Audio or video file]
    FFmpeg[FFmpeg normalize to 16 kHz mono WAV]
    Whisper[whisper.cpp transcription]
    Summarizer[LLM summarizer]
    Notes[Markdown notes]

    Audio --> FFmpeg --> Whisper --> Summarizer --> Notes

Alternatives

• faster-whisper — Python/CTranslate2 implementation; often faster for GPU/server transcription and easier in Python stacks.
• OpenAI Whisper API — hosted transcription; less ops, but sends audio to OpenAI and costs per use.
• WhisperX — adds alignment and diarization workflows around Whisper.
• SpeechBrain / NVIDIA NeMo — broader speech research/toolkit ecosystems.
• Handy — not an alternative engine but a ready-made desktop app: push-to-talk dictation into any text field using Whisper or Parakeet models locally.

Caveats

• Whisper is ASR, not a full voice assistant; you still need VAD, LLM/tool orchestration, TTS, and state.
• Long audio should be chunked and normalized before transcription.
• Speaker diarization support is limited/experimental in whisper.cpp; use specialized diarization tooling for serious multi-speaker work.
• The model license and data policy depend on the specific model files you use, not just the whisper.cpp code license.