NVIDIA NeMo Speech is a PyTorch toolkit for researchers and developers working on speech models: automatic speech recognition (ASR), text-to-speech (TTS), speech translation, speaker recognition, speaker diarization, and speech-enabled large language models. It provides training and inference code plus pretrained checkpoints so you can create, customize, and deploy models without starting from scratch.
The repository is the speech-focused successor to the original NeMo framework: as of 2026 it concentrates on audio, speech, and multimodal LLMs, with v2.7.3 as the last release covering additional modalities. NeMo Speech 3.0 ships as a PyPI package (nemo-toolkit) and as an NGC container, and runs on Python 3.12+ and PyTorch 2.7+ on top of an existing CUDA stack. Training requires an NVIDIA GPU; the SpeechLM2 Automodel backend runs without compiled dependencies and can optionally use accelerated kernels (Transformer Engine, FlashAttention, Mamba, DeepEP).
Recent model releases include Nemotron-3.5-ASR-Streaming-0.6B (40 languages, 80 ms to 1 s controllable latency), Parakeet-unified-en-0.6b (offline and streaming English recognition in one model), MagpieTTS v2607 (12 languages), Canary-Qwen-2.5B (5.63% word error rate on the English Open ASR Leaderboard), and Parakeet V3 and Canary V2 with recognition and translation for 25 European languages.
Features
- Automatic speech recognition: training, fine-tuning, and inference for offline and streaming models, including cache-aware FastConformer architectures
- ASR customization: GPU-based n-gram language models (NGPU-LM), language model fusion, neural rescoring, and word boosting
- Text-to-speech: Magpie-TTS models with fine-tuning, preference optimization, long-form inference, and grapheme-to-phoneme components
- Speech LLMs: SpeechLM2 models with an Automodel backend and optional accelerated kernels for Hopper, Blackwell, and A100 GPUs
- Speaker tasks: speaker recognition and speaker diarization collections with model configurations and checkpoints
- Speech and audio processing: audio models and configuration files alongside the ASR and TTS collections
- Training utilities: parallelism, mixed-precision training, checkpoint formats, and Lhotse-based data loading
- Pretrained checkpoints: open-weight models published in the NVIDIA Nemotron Speech collection on Hugging Face
- Flexible environment: installs over your own Python, PyTorch, and CUDA versions, or reproduces the tested stack via uv.lock and the NGC container
