Source-linked AI summary
WhisperX: Time-Accurate Speech Transcription of Long-Form Audio
Max Bain, Jaesung Huh, Tengda Han, Andrew Zisserman
TL;DR
Long-form transcription with Whisper is challenged by drift, hallucination, repetition, sequential processing, and inaccurate timestamps. WhisperX combines VAD Cut & Merge segmentation with forced phoneme alignment to provide efficient transcription and word-level timestamps, demonstrating strong benchmark performance and a twelve-fold speed increase without sacrificing transcription quality. Its multilingual alignment requires language-appropriate models, and repetition errors remain an area for further work.
Problem
Long-form Whisper transcription suffers from drifting, hallucination, repetition, sequential processing, and inaccurate or unavailable word-level timestamps.
Method
WhisperX pre-segments audio with VAD and Cut & Merge, batches Whisper transcription, and applies external phoneme forced alignment for word-level timestamps.
Results
WhisperX outperforms Whisper and wav2vec2.0 on word segmentation benchmarks, WER, and transcription speed, with nearly twelve-fold faster batched inference without performance loss.
Takeaways & Limitations
VAD Cut & Merge reduces hallucination and repetition while enabling within-audio batched transcription and time-accurate word segmentation.
Takeaways & Limitations
Further work is needed to reduce hallucination and repetition errors, and multilingual alignment requires VAD and phoneme models suited to the target languages.
Abstract
from arXiv · showhide
Large-scale, weakly-supervised speech recognition models, such as Whisper, have demonstrated impressive results on speech recognition across domains and languages. However, their application to long audio transcription via buffered or sliding window approaches is prone to drifting, hallucination & repetition; and prohibits batched transcription due to their sequential nature. Further, timestamps corresponding each utterance are prone to inaccuracies and word-level timestamps are not available out-of-the-box. To overcome these challenges, we present WhisperX, a time-accurate speech recognition system with word-level timestamps utilising voice activity detection and forced phoneme alignment. In doing so, we demonstrate state-of-the-art performance on long-form transcription and word segmentation benchmarks. Additionally, we show that pre-segmenting audio with our proposed VAD Cut & Merge strategy improves transcription quality and enables a twelve-fold transcription speedup via batched inference.
1. Introduction
Long-form transcription exposes limitations not prominent in short-utterance benchmarks, including memory constraints, boundary errors, timestamp drift, and inaccurate word timing. WhisperX addresses these challenges with VAD-based segmentation, Cut & Merge processing, and external phoneme alignment.
- Real-world applications require transcribing audio lasting minutes or hours, unlike most academic benchmarks built from short utterances.
- Whisper is trained on 30-second audio segments, while transformer memory constraints prevent transcription of arbitrarily long inputs.
- Sliding-window and buffered approaches can suffer overlapping or incomplete audio, timestamp drift, and accumulated boundary errors.
- Forced alignment methods provide a basis for aligning transcripts with audio at word or phoneme level, motivating combinations of ASR and phoneme models.
- WhisperX adds VAD pre-segmentation, Cut & Merge chunking for batched Whisper inference, and external phoneme forced alignment for accurate word timestamps.
2. WhisperX
WhisperX organizes long-form transcription around speech-aware segmentation, bounded approximately 30-second chunks, parallel Whisper inference, and phoneme-based word alignment. Its design also includes multilingual scope conditions and a comparison against direct Whisper timestamp extraction.
- Voice Activity Detection: VAD identifies speech regions and supports cheaper processing, boundaries outside active speech, parallel transcription, and locally constrained alignment.
- Voice Activity Detection: VAD models map acoustic feature sequences to binary speech-presence labels, which are post-processed into active speech segments.
- VAD Cut & Merge: WhisperX uses min-cut to bound long active segments and merge to combine neighboring short segments, preserving context while matching ASR input limits.For Whisper, the training and maximum input duration is 30 seconds.
- Batched Transcription: Speech-aware segments approximately matching the model’s training duration can be transcribed in parallel, without previous-text conditioning to preserve batch independence.
- Word Alignment: Phoneme alignment restricts classification to phonemes present in each transcript, applies DTW, and derives word start and end times from the temporal path.
- Multilingual Transcription: Multilingual use requires language-robust VAD and an alignment phoneme model trained for the target languages, while translated transcripts cannot receive phoneme alignment.
3. Evaluation
The evaluation tests WhisperX against Whisper and wav2vec2.0, examines VAD Cut & Merge effects on quality and speed, and studies model choices for word segmentation.
- WhisperX is evaluated against Whisper and wav2vec2.0 on long-form transcription and word-level segmentation.
- The evaluation measures how VAD Cut & Merge pre-processing affects transcription quality and speed.
- The study examines how phoneme-model and Whisper-model choices affect word segmentation performance.
3.1. Datasets
The evaluation uses meeting, telephone-conversation, and long-form audiovisual speech corpora, with manually verified or corrected word alignments where available.
- The AMI-IHM test set contains 16 meeting recordings with manually verified word-level alignments.
- The study could not evaluate multilingual word segmentation with non-English ASR, but reports successful qualitative examples in its open-source repository.
- Switchboard-1 contains approximately 2,400 hours of telephone-conversation speech with manually corrected word alignments; 100 conversations were randomly sampled.
- TEDLIUM-3 comprises 11 TED talks of 20 minutes each, while Kincaid46 contains videos sourced from YouTube for long-form transcription evaluation.
3.2. Metrics
The evaluation measures transcription accuracy and speed, repetition and hallucination, and joint word-transcription and timestamp quality.
- Long-form transcription is evaluated with Word Error Rate and transcription speed.
- Insertion Error Rate and 5-gram duplicate counts quantify hallucination and repetition in predicted transcripts.
- Word segmentation evaluation jointly assesses transcription and timestamp quality using Precision and Recall on datasets with word-level timestamps.
3.3. Implementation Details
Implementation comparisons use WhisperX, Whisper, and wav2vec2.0 configurations alongside benchmark tables describing transcription and segmentation evaluations.
- WhisperX uses the default configuration unless experiments specify otherwise.
- Whisper baselines inherit the default configuration and use the official implementation for word-timestamp inference.
- wav2vec2.0 experiments use default settings unless specified otherwise, with model versions obtained from the official torchaudio repository.
- Table 2 compares long-form transcription and word segmentation across TED-LIUM, Kincaid46, AMI, and SWB using speed, WER, 5-Dup, Precision, and Recall.
- Table 3 compares full-audio input with VAD-CMτ preprocessing and batched transcription on TED-LIUM and AMI.
3.4. Results
WhisperX improves long-form transcription, word segmentation, and speed through VAD-based chunking, batched inference, and external phoneme alignment. Results also show that model choice affects segmentation, while repetition errors remain a limitation.
- WhisperX substantially outperforms Whisper and wav2vec2.0 in word segmentation benchmarks, WER, and transcription speed.
- VAD and Cut & Merge improve transcription-only WER and word segmentation precision and recall, whereas batching without VAD degrades both.
- Nearly twelve-fold speed increase is achieved with VAD-based batched inference without performance loss.
- The optimal Cut & Merge threshold is |Atrain| = 30, providing the fastest transcription speed and lowest WER.
- WhisperX reports the lowest IER on Kincaid46 and TED-LIUM, while VAD operations reduce hallucination and repetition errors.
- Wav2vec2.0 has fewer repetition errors than Whisper and WhisperX despite underperforming in WER and word segmentation.
- Larger Whisper models consistently improve segmentation precision and recall, but the larger phoneme model does not necessarily perform best.
- The LibriSpeech-trained base alignment model performs consistently well, whereas the large alignment model shows no consistent gains.
4. Conclusion
WhisperX is a time-accurate speech recognition system that parallelizes long-form transcription using VAD Cut & Merge and provides word-level segmentations through phoneme alignment. The system achieves a twelve-fold speed increase without sacrificing transcription quality.
- WhisperX enables within-audio parallelized transcription through VAD Cut & Merge preprocessing.
- VAD Cut & Merge reduces hallucination and repetition while enabling batched transcription.
- Twelve-fold speed increase is achieved without sacrificing transcription quality.
- Forced alignment with a phoneme model provides accurate word-level segmentations with minimal inference overhead.
- WhisperX produces time-accurate transcriptions for applications including subtitling and diarisation.