WhisperX is a speech recognition pipeline built on OpenAI's Whisper that adds word-level timestamps and speaker labels to transcripts. Whisper on its own produces utterance-level timestamps that can be off by several seconds and does not batch inference. WhisperX addresses both by transcribing voice-activity-detected segments in batches and then aligning the text to the audio with a phoneme-based model.
The pipeline runs in three stages: batched transcription on a faster-whisper backend (reported at 70x realtime with large-v2, under 8 GB of GPU memory with beam size 5), forced alignment with a language-specific wav2vec2 model to get per-word timings, and optional speaker diarization with pyannote.audio to assign speaker IDs. Voice activity detection runs first to cut hallucination and enable batching without a rise in word error rate.
It is usable as a command-line tool and as a Python library, on GPU or CPU. Alignment models for English, French, German, Spanish, and Italian come from torchaudio, with many other languages covered by Hugging Face models. A context-aware batching mode passes context from the previous segment to improve punctuation and proper-noun handling on long audio.
Documented limitations: words with no characters in the alignment model's dictionary (numbers such as "2014." or currency) get no timing, overlapping speech is handled poorly, diarization is imperfect, and each language needs its own wav2vec2 model. The method is described in an INTERSPEECH 2023 paper.
Features
- Batched transcription: voice-activity-based batching on a faster-whisper backend for fast long-form inference
- Word-level timestamps: forced alignment with wav2vec2 phoneme models gives a timing for each word
- Speaker diarization: pyannote.audio assigns speaker ID labels to words and segments, with optional minimum and maximum speaker counts
- Voice activity detection: VAD preprocessing reduces hallucination, with Silero VAD available as an alternative VAD option
- Multilingual alignment: default alignment models for five languages via torchaudio and many more via Hugging Face, picked automatically from the detected language
- Interleaved context: conditions each segment on previous context to improve punctuation and proper nouns
- CLI and Python API: transcribe, align, and diarize from the command line or in code, on GPU or CPU
- Subtitle output: writes .srt files with optional word highlighting
