Source-linked AI summary
Context-Aware Interleaved Batching for WhisperX
Carlos Bain, Max Bain
TL;DR
Long-form transcription must balance WhisperX’s fast parallel processing against standard Whisper’s slower but context-preserving decoding. The paper introduces Context-Aware Interleaved Batching, which carries text context across VAD-segmented batches while decoupling audio boundaries from fragile timestamp decoding. On long-form benchmarks, the method reduces WER, improves proper-noun transcription, and maintains high-throughput inference, with a batch-size limitation when too few chunks are available.
Problem
WhisperX’s parallel chunks lose historical context, while standard Whisper preserves context sequentially but is slow and vulnerable to timestamp-related hallucination loops.
Method
Context-Aware Interleaved Batching partitions VAD-segmented audio into contiguous streams and propagates stream-specific rolling text context across parallel batches.
Results
Across Earnings-21 and MedicalLessons, WhisperX+IB improves accuracy while retaining parallel speed, including 8.4× speedup, WER reduction from 3.5% to 3.3%, and PN Score improvement from 83.6% to 84.7%.
Takeaways & Limitations
The framework bridges high-throughput batched transcription and continuous context, improving grammatical, punctuation, and proper-noun accuracy in long-form audio.
Takeaways & Limitations
When the number of VAD chunks is no greater than the batch size, restoring preceding-text context may require reducing B, potentially to 1, trading parallel throughput for contextual accuracy.
Abstract
from arXiv · showhide
While WhisperX accelerates speech transcription via intra-audio batching, it isolates audio segments, losing the historical context needed for coherent punctuation and terminology transcription. Conversely, standard Whisper retains context sequentially but suffers from slow inference and hallucination loops. To achieve the best of both worlds, we propose Context-Aware Interleaved Batching. By using VAD-derived segment boundaries, our algorithm stabilizes Whisper's text conditioning, allowing us to safely maintain continuous historical context across batched audio segments. As demonstrated on long-form audio benchmarks, this approach reduces Word Error Rate (WER) and improves proper noun transcription, all while maintaining high-throughput inference speeds.
1. Introduction
Long-form transcription faces a tension between Whisper’s sequential historical conditioning and WhisperX’s faster but context-isolated batching. Context-Aware Interleaved Batching propagates historical text across parallel batch boundaries, stabilizing conditioning while preserving speed and improving accuracy.
- Standard Whisper processes long-form audio sequentially, making deployment computationally expensive and preventing batched inference.
- Whisper’s previous-text conditioning can degrade WER when errors or hallucinations propagate across sequential forward passes.
- WhisperX segments audio at natural VAD-detected silences, enabling within-audio batching, improved accuracy, and accurate word-level timestamps through CTC forced alignment.
- WhisperX’s isolated batched chunks cannot condition on preceding text, removing autoregressive context needed across long-form audio segments.
- Context-Aware Interleaved Batching propagates historical text across parallel batch boundaries while decoupling chunk boundaries from internal timestamp decoding to stabilize conditioning and improve accuracy.
- The paper contributes a parallel historical-conditioning method, an open-source medical long-form benchmark with Proper Noun and punctuation metrics, and demonstrations of accuracy gains across metrics.
2. Method
The method replaces Whisper’s fragile timestamp-driven sequential windowing with VAD-defined chunks arranged into contiguous streams and processed through interleaved parallel batches. Stream-specific rolling contexts carry decoded text between successive chunks, with a cold-start re-transcription resolving initially unavailable context.
- Baseline architectures: Standard Whisper uses cross-attention for the current audio window and previous-window text through decoder self-attention conditioning.
- Baseline architectures: Whisper’s autoregressive timestamp decoding can misplace audio boundaries, causing erroneous window shifts, loops, and severe text hallucinations.
- WhisperX baseline: WhisperX uses VAD to create temporally ordered variable-length chunks up to 30 seconds, then groups them into batches for parallel transcription.
- Context-Aware Interleaved Batching: Context-Aware Interleaved Batching partitions VAD-segmented chunks into B contiguous streams of length M = ⌈N/B⌉, padding the final stream when needed.
- Context-Aware Interleaved Batching: At each of M iterations, the algorithm extracts the corresponding chunk from every stream to form a parallel batch.
- Context-Aware Interleaved Batching: Each stream maintains a 224-token FIFO context buffer, passing decoded text from the preceding chunk to the next chunk while processing B chunks in parallel.
- Cold-Start Resolution: Because the first batch lacks the final stream’s preceding context, it is initially unconditioned and then re-transcribed after all batches complete.
3. Results
The evaluation compares transcription variants across financial and medical long-form audio using conventional and LLM-based metrics. Context-Aware Interleaved Batching improves accuracy and terminology transcription while retaining high throughput, though punctuation quality and small-batch conditions expose trade-offs.
- Evaluation setup: The evaluation uses Earnings-21 and MedicalLessons, with WER, punctuation F1, speed, LPS, and Proper Noun Score assessed at transcript level.MedicalLessons contains manually verified medical terminology transcripts, while Earnings-21 provides ground-truth punctuation markers.
- Earnings-21: 8.4× speedup lets WhisperX+IB improve accuracy while remaining close to standard WhisperX’s 11.8× speed.The sequential openai/whisper baseline runs at 1.0× speed and lacks parallelisation.
- Earnings-21: WhisperX+IB matches the top LPS of 2.9 but trails openai/whisper in punctuation F1 on Earnings-21.The authors hypothesize that timestamp-conditioned decoding may improve full-stop placement relative to decoding without timestamp tokens.
- MedicalLessons: WER falls from 3.5% to 3.3% and Proper Noun Score rises from 83.6% to 84.7% on MedicalLessons with WhisperX+IB.The improvement is associated with propagating preceding text across VAD-segmented chunks containing specialized terminology.
- Qualitative comparison: Context injection corrects a hallucinated “post-2007” transcription to “upholstery fabric segment” in the qualitative comparison.The comparison attributes the corrected output to leveraging preceding dialogue.
- Limitations: When the number of VAD chunks N is at most batch size B, restoring preceding context may require reducing B, potentially to 1, trading throughput for contextual accuracy.The method reportedly retains an accuracy advantage over standard Whisper even at B = 1.
4. Conclusion
The proposed framework addresses the speed–accuracy trade-off in high-speed audio transcription by preserving context across chunks while reducing grammatical and punctuation errors.
- WhisperX+IB reduces transcription errors in both grammar and punctuation.