Inworld provides real-time voice AI for conversational products: a text-to-speech (TTS) service, a speech-to-text service, a real-time speech-to-speech API, and an LLM router that forwards requests to third-party models. The current speech models are Realtime TTS-2, with a stated time to first byte of about 100 ms, and TTS-2 Flash, with a stated time to first byte of about 20 ms. Target uses include social and companion apps, learning products, health and wellness, agentic workforces, and games and media.
The open-source component is the Inworld Text-To-Speech Trainer, the training and modelling code behind the earlier TTS-1 and TTS-1-Max models. It lets you pre-train, fine-tune, or align with reinforcement learning a SpeechLM-based TTS model on a single GPU or a multi-GPU cluster. The model pairs a decoder-only language model (for example meta-llama/Llama-3.2-1B-Instruct) with a 1D audio codec; the codec is compatible with the public xcodec2 checkpoint, so you can skip training a codec.
The training pipeline runs in stages: prepare a JSONL manifest of transcripts and WAV paths, vectorize audio into codec tokens with data_vectorizer.py, merge the shards, then run supervised fine-tuning with a JSON config through fabric run. A second stage applies reinforcement learning from human feedback (RLHF) with Group Relative Policy Optimization-style reward functions such as word error rate, served by a separate vLLM node. Inference takes text plus a reference audio prompt and its transcription to generate speech in that voice.
Requirements are Python 3.10, CUDA 12.4 or 12.8, and PyTorch 2.6 or 2.7; the code is tested on Ubuntu 22.04 only.
Features
- Realtime TTS-2 and TTS-2 Flash: hosted text-to-speech with stated first-byte latencies of about 100 ms and 20 ms respectively
- Realtime speech-to-speech API: end-to-end voice conversation without a separate transcription and synthesis pipeline
- Speech-to-text and LLM routing: transcription and a router that passes requests to third-party language models
- SpeechLM modelling: decoder-only transformer over 1D audio-codec tokens, with a configurable codebook size and context length
- Distributed training: DDP, DeepSpeed, and FSDP strategies for single-GPU to multi-node runs
- Data pipeline: JSONL manifest format, sharded vectorization, shard merging, and a 100-sample LibriTTS example set
- RLHF alignment: reward functions such as WERRewardFunc, KL penalty, and multi-generation sampling served through vLLM on a second node
- Voice-prompted inference: generates speech from text conditioned on a reference WAV and its transcription
- Monitoring: Weights & Biases metrics, periodic checkpoints, and console logs
