Ultravox v0.4.1 is a multimodal speech-language model that accepts speech and text in the same prompt and generates text. A frozen Whisper-large-v3-turbo encoder turns 16 kHz audio into embeddings, a trainable adapter projects them into the input space of a frozen Llama 3.1 70B Instruct backbone, and an placeholder in the text prompt marks where those embeddings are spliced in. Only the adapter (about 58.7M parameters) is trained.
Training is supervised speech-instruction fine-tuning by knowledge distillation: the model learns to match the logits that the text-only Llama produces for the transcript. The data mixes ASR corpora, extended with Llama 3.1 8B continuations, and speech-translation sets. No preference tuning is applied, and voice output is planned for later versions through semantic and acoustic audio tokens.
Intended uses are voice agents, speech-to-speech translation, and analysis of spoken audio. On an A100-40GB the model reaches roughly 150 ms time to first token and 50 to 100 tokens per second. Translation quality (BLEU) moved between v0.4 and v0.4.1 as follows:
| Pair | v0.4 | v0.4.1 |
|---|---|---|
| en→ar | 14.97 | 19.64 |
| en→de | 30.30 | 32.47 |
| es→en | 39.55 | 40.76 |
| ru→en | 44.16 | 45.07 |
| en→ca | 35.02 | 37.58 |
| zh→en | 12.16 | 17.98 |
Features
- Speech plus text input: system and user turns carry text while slots in the audio embeddings
- Frozen backbone and encoder: Whisper-large-v3-turbo and Llama 3.1 70B Instruct stay fixed; only the multimodal adapter is trained
- Distillation objective: the adapter is trained to reproduce the text model's logits for the spoken content
- Text output: responds in text; audio generation is a stated direction for future versions
- Latency figures: about 150 ms to first token and 50–100 tokens/s on an A100-40GB
- BF16 weights: the released adapter ships in bfloat16
