MARS5 is CAMB.AI's English text-to-speech model. Given a snippet of text and a reference audio clip of 2 to 12 seconds, it synthesizes speech in that speaker's voice, including prosodically demanding material such as sports commentary and anime dialogue. The model is the open English release; the company's hosted API extends the same family to text-to-speech and dubbing in 140 or more languages.
Synthesis follows a two-stage pipeline. An autoregressive transformer (about 750M parameters) produces coarse L0 Encodec speech features from the text and reference audio; a multinomial DDPM non-autoregressive stage (about 450M parameters) then refines the text, reference, and coarse features into the remaining Encodec codebook values, and a vocoder turns the result into 24 kHz audio.
Because training pairs raw audio with byte-pair-encoded text, you steer prosody through the transcript itself: a comma inserts a pause, and capital letters emphasize a word. Two inference modes are available. A shallow clone needs only the reference audio; a deep clone also takes the reference transcript, trades longer generation time for closer voice matching, and is the mode used when quality matters most. Reference clips around 6 seconds give the strongest results.
You load the AR and NAR models through torch.hub (Python 3.10+, PyTorch 2.0+) or from the Hugging Face checkpoint, then call tts() with text, reference waveform, optional transcript, and an InferenceConfig that exposes sampling controls such as topk, temperature, topp, repetition penalty window, and frequency penalty. Checkpoints ship as both .pt and .safetensors files with the BPE tokenizer embedded; inference needs GPU memory for all 1.2B parameters with 750M active.
Features
- Voice cloning: speaker identity from a 2 to 12 second reference clip, with shallow (audio only) and deep (audio plus transcript) modes
- Prosody control: punctuation and capitalization in the transcript guide pauses and emphasis
- AR-NAR architecture: an autoregressive coarse stage followed by a multinomial diffusion refinement stage over Encodec codes
- Tunable sampling: topk, topp, temperature, repetition-window and frequency penalties via InferenceConfig
- Checkpoint formats: fp16 AR and NAR weights in PyTorch and safetensors formats, loadable from torch.hub or Hugging Face
- Docker base image: a published image to build custom inference services on
- Hosted API: the same model family available as a service for users without local GPU capacity
