Chatterbox

1 September 20267 min read

Chatterbox is a family of open-source text-to-speech (TTS) models by Resemble AI. It is a strong first choice for advanced self-hosted TTS when you want natural speech, zero-shot voice cloning, multilingual generation, and voice-agent-friendly latency without using a hosted TTS API.

Compared with lightweight local TTS such as Piper, Chatterbox is heavier and more GPU-oriented, but much more expressive and flexible.

Why it matters

• Advanced self-hosted TTS — high-quality speech generation without sending text/reference audio to a hosted provider.

• Zero-shot voice cloning — synthesize with a target voice using a short reference clip.

• Multilingual support — Chatterbox Multilingual V3 supports 23+ languages.

• Low-latency option — Chatterbox-Turbo is a smaller 350M English model aimed at voice agents and production-ish use.

• Paralinguistic tags — Turbo supports tags like [laugh], [cough], and [chuckle] for more human speech.

• Watermarking — generated audio includes Resemble AI’s PerTh watermark for responsible AI provenance.

• Simple Python API — install via pip install chatterbox-tts and call from Python.

Model zoo

Model	Size	Languages	Best for
Chatterbox-Turbo	350M	English	Low-latency voice agents, English narration, production-style apps.
Chatterbox-Multilingual V3	500M	23+	Multilingual TTS, localization, cross-language voice cloning.
Single Language Pack	500M each	Dedicated language finetunes	Higher language/dialect-specific quality for priority markets.
Chatterbox English	500M	English	General English zero-shot TTS with creative controls.

The upstream README describes Multilingual V3 as the recommended general-purpose multilingual model because it improves speaker similarity, reduces hallucinations/off-prompt speech, and produces more natural conversational speech than earlier multilingual versions.

Supported languages

Chatterbox Multilingual supports:

• Arabic ar

• Danish da

• German de

• Greek el

• English en

• Spanish es

• Finnish fi

• French fr

• Hebrew he

• Hindi hi

• Italian it

• Japanese ja

• Korean ko

• Malay ms

• Dutch nl

• Norwegian no

• Polish pl

• Portuguese pt

• Russian ru

• Swedish sv

• Swahili sw

• Turkish tr

• Chinese zh

Single-language packs exist for some priority language/regional variants, including Chinese, LatAm Spanish, Brazilian Portuguese, Spain Spanish, Portugal Portuguese, and Hindi.

Installation

shell

pip install chatterbox-tts

From source:

shell

git clone https://github.com/resemble-ai/chatterbox.git

cd chatterbox

pip install -e .

Upstream says development/testing was done on Python 3.11 on Debian 11, with dependencies pinned in pyproject.toml.

Basic English TTS

python

import torchaudio as ta

from chatterbox.tts import ChatterboxTTS

model = ChatterboxTTS.from_pretrained(device="cuda")

text = "This is an advanced self-hosted text-to-speech model."

wav = model.generate(text)

ta.save("output.wav", wav, model.sr)

Devices can be cuda, cpu, or mps, but for practical quality/latency use an NVIDIA GPU when possible.

Voice cloning

Pass a reference clip with audio_prompt_path:

python

import torchaudio as ta

from chatterbox.tts import ChatterboxTTS

model = ChatterboxTTS.from_pretrained(device="cuda")

wav = model.generate(

"This sentence will be spoken in the style of the reference voice.",

audio_prompt_path="reference_voice.wav",

)

ta.save("cloned.wav", wav, model.sr)

Reference clip guidance:

• Use clean speech with little background noise.

• 5–15 seconds is a practical target; upstream Turbo example mentions a 10-second reference clip.

• Use a clip in the same language/accent as the target output when possible.

• Avoid copyrighted, private, or non-consensual voice samples.

Chatterbox-Turbo

Turbo is the lower-latency English model. It is useful for voice agents, interactive apps, and narration where response time matters.

python

import torchaudio as ta

from chatterbox.tts_turbo import ChatterboxTurboTTS

model = ChatterboxTurboTTS.from_pretrained(device="cuda")

text = "Hi, I can sound more natural now [chuckle], and include little human reactions."

wav = model.generate(text, audio_prompt_path="reference_10s.wav")

ta.save("turbo.wav", wav, model.sr)

Turbo supports paralinguistic tags such as:

• [laugh]

• [chuckle]

• [cough]

Use these sparingly. Overusing expressive tags can make speech feel artificial or unstable.

Multilingual V3

python

import torchaudio as ta

from chatterbox.mtl_tts import ChatterboxMultilingualTTS

model = ChatterboxMultilingualTTS.from_pretrained(

device="cuda",

t3_model="v3",

)

wav = model.generate(

"Bonjour, ceci est une voix multilingue générée localement.",

language_id="fr",

)

ta.save("french.wav", wav, model.sr)

With a reference voice:

python

wav = model.generate(

"Hola, esta voz fue generada con un modelo autoalojado.",

language_id="es",

audio_prompt_path="spanish_reference.wav",

)

ta.save("spanish.wav", wav, model.sr)

Important: when cloning across languages, the reference clip language/accent can affect the output. Upstream suggests matching the reference clip to the specified language tag when possible. If language transfer inherits too much reference accent, try reducing cfg_weight.

Tuning controls

The upstream README calls out two important controls:

• cfg_weight

• exaggeration

General defaults:

python

wav = model.generate(

text,

audio_prompt_path="reference.wav",

exaggeration=0.5,

cfg_weight=0.5,

)

Guidelines from upstream:

Goal	Suggested tuning
General speech	Defaults around `exaggeration=0.5`, `cfg_weight=0.5`.
Reference speaker talks too fast	Lower `cfg_weight` to around `0.3` to improve pacing.
More dramatic/expressive speech	Increase `exaggeration` toward `0.7+` and lower `cfg_weight` around `0.3`.
Cross-language cloning accent issues	Lower `cfg_weight`; upstream notes `0` can mitigate inherited accent in some cases.

Exact parameter availability can vary by class/model, so check the examples in the repo for the model family you use.

Voice conversion

The repository includes example_vc.py and gradio_vc_app.py, indicating Chatterbox also supports voice-conversion-style workflows. Use this when you want to transform existing speech toward another voice rather than synthesize from text only.

Gradio apps

The repo includes:

• gradio_tts_app.py

• gradio_tts_turbo_app.py

• gradio_vc_app.py

• multilingual_app.py

Typical pattern after installing from source:

shell

python gradio_tts_app.py

python gradio_tts_turbo_app.py

python multilingual_app.py

Use these for manual testing, voice comparison, and prompt/reference-clip iteration before wrapping the model behind an API.

Watermarking

Chatterbox embeds Resemble AI’s PerTh watermark into generated audio. The upstream README says this watermark is imperceptible, survives common transformations such as MP3 compression and editing, and has high detection accuracy.

Watermark extraction example:

python

import librosa

import perth

watermarked_audio, sr = librosa.load("output.wav", sr=None)

watermarker = perth.PerthImplicitWatermarker()

watermark = watermarker.get_watermark(watermarked_audio, sample_rate=sr)

print(watermark) # 0.0 = no watermark, 1.0 = watermarked

This is useful for provenance and abuse mitigation, but do not treat watermarking as the only safety control.

Self-hosted service pattern

Chatterbox does not appear to ship as an OpenAI-compatible /v1/audio/speech server by default. For apps, wrap it behind a small internal HTTP service.

flowchart TD
    App[App or voice agent]
    API[TTS adapter API]
    Profiles[Voice profile registry]
    Chatterbox[Chatterbox model worker]
    Store[Audio storage]
    Audio[WAV or MP3 response]

    App --> API
    API --> Profiles
    API --> Chatterbox
    Chatterbox --> Store
    Store --> Audio
    Audio --> App

Adapter responsibilities:

• keep model(s) warm;

• map voice IDs to reference clips or preset profiles;

• normalize/chunk text;

• enforce max text length;

• select turbo, english, or multilingual-v3 model;

• convert WAV to MP3/Opus if needed;

• add rate limits and auth;

• log consent/provenance for voice cloning;

• return file URLs or streamed audio.

OpenAI-style adapter shape:

http

POST /v1/audio/speech

Content-Type: application/json

Authorization: Bearer sk-local

{

"model": "chatterbox-turbo",

"voice": "sarah-reference",

"input": "Hello from a self-hosted Chatterbox backend.",

"response_format": "wav"

}

Voice assistant pipeline

flowchart TD
    Mic[Microphone]
    STT[whisper.cpp or faster-whisper]
    LLM[LiteLLM chat/tool model]
    TTS[Chatterbox Turbo]
    Speaker[Speaker]

    Mic --> STT --> LLM --> TTS --> Speaker

Use LiteLLM for the chat/tool model and Chatterbox as the TTS sidecar. Keep speech synthesis outside the LLM gateway unless you write an adapter.

Choosing models

Scenario	Model choice
English realtime-ish voice agent	Chatterbox-Turbo
English narration or creative speech	Chatterbox English
One model for many languages	Chatterbox-Multilingual V3
Language/dialect-specific app	Single Language Pack
Internal experiments	Gradio apps first, then wrap API

Safety and legal notes

Voice cloning is sensitive. A self-hosted Chatterbox service should include policy controls:

• require consent for uploaded reference voices;

• reject public-figure or third-party cloning unless authorized;

• store provenance for reference clips and generated audio;

• keep PerTh watermarking enabled;

• rate-limit external users;

• avoid returning raw reference clips;

• define retention/expiry for uploaded audio;

• review model and generated-audio usage terms before commercial deployment.

Relation to other tools

• Piper — simpler, faster, CPU/edge-friendly offline TTS, but less expressive.

• whisper.cpp — local speech-to-text side of a voice assistant.

• LiteLLM — routes the LLM part of voice assistants; Chatterbox handles speech output.

• CosyVoice — stronger production multilingual streaming/deployment story.

• Dia — better fit for generated two-speaker dialogue scenes.

• GPT-SoVITS — better fit when fine-tuning a specific voice from a small dataset through a WebUI workflow.

Caveats

• GPU strongly recommended for practical latency.

• No built-in OpenAI-compatible speech server in the upstream README; plan to wrap it.

• Voice cloning quality depends heavily on reference audio quality.

• Cross-language cloning may inherit unwanted accents from the prompt clip.

• Generated speech can still hallucinate/repeat/off-prompt; Multilingual V3 improves this but does not eliminate it.