Fish Audio offers a hosted text-to-speech, voice-cloning, and speech-to-text service with a real-time WebSocket API, and publishes the underlying model family as Fish Speech. The current open model, Fish Audio S2 Pro, is a 4-billion-parameter multilingual text-to-speech model trained on more than 10 million hours of audio across more than 80 languages.
S2 Pro uses a dual-autoregressive architecture: a 4B slow autoregressive transformer predicts the primary semantic codebook along the time axis, and a 400M fast autoregressive model fills in the remaining nine residual codebooks of a residual vector quantization (RVQ) codec at about 21 Hz. Post-training uses Group Relative Policy Optimization (GRPO) with reward signals covering semantic accuracy, instruction adherence, acoustic preference, and timbre similarity. Because the architecture matches a standard LLM, it runs on SGLang and vLLM-Omni with continuous batching, paged KV cache, CUDA graphs, and prefix caching.
Control over delivery is expressed inline with bracketed natural-language tags placed anywhere in the text, such as [whisper], [excited], [pause], or free-form descriptions like [professional broadcast tone]. Reference audio containing several speakers is mapped to tokens so one generation can switch between voices, and an extended context lets later turns draw on earlier ones in a dialogue. Voice cloning uses 10 to 30 seconds of reference audio without fine-tuning.
Reported results:
| Benchmark | Fish Audio S2 |
|---|---|
| Seed-TTS Eval, WER (Chinese) | 0.54% |
| Seed-TTS Eval, WER (English) | 0.99% |
| Audio Turing Test (with instruction) | 0.515 posterior mean |
| EmergentTTS-Eval, win rate | 81.88% |
| Fish Instruction Benchmark, TAR | 93.3% |
| Fish Instruction Benchmark, quality | 4.51 / 5.0 |
| Single H200, real-time factor | 0.195 |
| Single H200, time to first audio | ~100 ms |
Features
- Inline emotion and prosody tags: bracketed instructions at any position in the text, with support for free-form descriptions beyond fixed presets
- Multilingual generation: more than 80 languages without phonemes or language-specific preprocessing; tier 1 is Japanese, English, and Chinese
- Multi-speaker generation: one reference clip with several speakers, addressed by speaker ID tokens
- Multi-turn context: earlier dialogue turns condition later generations
- Voice cloning: 10 to 30 second reference samples capture timbre and speaking style
- Streaming inference: SGLang and vLLM-Omni servers with continuous batching and paged KV cache
- Local inference options: command-line, web UI, HTTP server, and Docker setups
- Hosted API: real-time text-to-speech over WebSocket, plus voice cloning and speech-to-text on the Fish Audio platform