Source-linked AI summary
SE-DiCoW: Self-Enrolled Diarization-Conditioned Whisper
Alexander Polok, Dominik Klement, Samuele Cornell, Matthew Wiesner, Jan Černocký, Sanjeev Khudanpur, Lukáš Burget
TL;DR
Multi-speaker speaker-attributed ASR remains difficult to generalize across domains, and DiCoW has ambiguous conditioning during fully overlapped speech. SE-DiCoW adds self-selected target-speaker enrollment through cross-attention and refines training and initialization, reducing macro-averaged tcpWER by 52.4% relative to DiCoW on EMMA MT-ASR.
Problem
Fully overlapped speech can give different target speakers nearly identical STNO conditioning despite differing transcriptions, limiting speaker disambiguation.
Method
SE-DiCoW automatically selects a target-speaker reference segment from the recording and incorporates it as additional conditioning via cross-attention, alongside refined segmentation, initialization, and augmentation.
Results
52.4% lower macro-average tcpWER than DiCoW is reported across the EMMA MT-ASR benchmark, with consistent gains on real conversational data.
Takeaways & Limitations
SE-DiCoW provides a stronger target-speaker ASR system across diverse benchmarks and remains robust to imperfect enrollment segments.
Takeaways & Limitations
The method remains challenged when enrollment is fully overlapped with too many speakers, with degradation reported as speaker count increases.
Abstract
from arXiv · showhide
Speaker-attributed automatic speech recognition (ASR) in multi-speaker environments remains a major challenge. While some approaches achieve strong performance when fine-tuned on specific domains, few systems generalize well across out-of-domain datasets. Our prior work, Diarization-Conditioned Whisper (DiCoW), leverages speaker diarization outputs as conditioning information and, with minimal fine-tuning, demonstrated strong multilingual and multi-domain performance. In this paper, we address a key limitation of DiCoW: ambiguity in Silence-Target-Non-target-Overlap (STNO) masks, where two or more fully overlapping speakers may have nearly identical conditioning despite differing transcriptions. We introduce SE-DiCoW (Self-Enrolled Diarization-Conditioned Whisper), which uses diarization output to locate an enrollment segment anywhere in the conversation where the target speaker is most active. This enrollment segment is used as fixed conditioning via cross-attention at each encoder layer. We further refine DiCoW with improved data segmentation, model initialization, and augmentation. Together, these advances yield substantial gains: SE-DiCoW reduces macro-averaged tcpWER by 52.4% relative to the original DiCoW on the EMMA MT-ASR benchmark.
1. INTRODUCTION
Speaker-attributed ASR remains difficult in multi-speaker conversations because overlapping, spontaneous speech and speaker attribution challenge conventional systems. DiCoW addresses these issues with diarization conditioning, while SE-DiCoW targets ambiguity in fully overlapped regions through self-enrollment and additional model refinements.
- Multi-speaker ASR must recover both transcript content and speaker identity, but single-speaker models struggle with overlap and spontaneous dialogue.
- Modular systems combining diarization, source separation, and ASR are complex, generalize poorly across domains, and can suffer cascading errors.
- Target-speaker ASR conditions recognition on speaker identity through embeddings or enrollment audio, but speaker-specific representations are difficult to generalize.
- DiCoW instead conditions Whisper on frame-level diarization masks, avoiding explicit speaker-identity modeling while supporting unknown speakers and cross-domain conversations.
- In fully overlapped speech, STNO masks can provide nearly identical conditioning for different speakers despite differing transcriptions.
- SE-DiCoW selects target-speaker speech segments from diarization outputs as additional cross-attention conditioning and combines this mechanism with improved initialization, segmentation, and augmentation.
2. METHOD
SE-DiCoW extends DiCoW with self-enrollment to resolve ambiguous STNO conditioning in fully overlapped speech, while adding architectural, training, and augmentation refinements.
- DiCoW: DiCoW conditions target-speaker ASR on frame-level diarization probabilities and four-category STNO masks rather than explicit speaker identities.The four categories represent silence, target activity, non-target activity, and target-speaker overlap.
- DiCoW limitation: Fully overlapped speech can yield nearly identical STNO conditioning for different target speakers, making speaker-specific transcription difficult.The ambiguity is especially problematic when simultaneous conversations produce different transcriptions despite similar conditioning.
- Self-enrollment: SE-DiCoW selects a fixed-length recording segment maximizing target-speaker activity and uses it as additional cross-attention conditioning at every encoder layer.The selected segment is derived from diarization probabilities and supplies speaker-specific reference representations.
- Additional improvements: Additional refinements include a pre-positional-embedding FDDT layer, corrected 30-second training segmentation, and robustness augmentations for diarization errors.The augmentations include Gaussian noise, activity flips, joint SpecAugment, and MUSAN noise; inference follows Whisper’s 30-second sequential windows.
3. EXPERIMENTAL SETUP
The experiments train Whisper-large-v3-turbo on mixed conversational and synthetic overlapping-speech data, then evaluate tcpWER across diarization conditions and recording domains.
- Training data: Training combines AMI, NOTSOFAR-18, Libri2Mix/3Mix, and synthetic LibriSpeech mixtures with up to three partially overlapping segments.Synthetic enrollment mixtures contain one target-speaker segment and two other-speaker segments.
- Training configuration: All models use Whisper-large-v3-turbo with a 2 × 10^-6 learning rate, batch size 96, 2k warmup steps, and 40k training steps.Training uses cosine decay.
- Evaluation metric: Performance is measured with time-constrained minimum-permutation WER using a 5-second collar.Both training and inference code are publicly available.
- Evaluation protocol: Evaluation compares oracle diarization, which provides an upper bound, with real diarization from the DiariZen front-end across multiple domains and recording conditions.AMI evaluation includes SDM and other recording configurations described in the experimental protocol.
4. RESULTS
Across real and synthetic benchmarks, SE-DiCoW achieves the lowest tcpWER, while improved segmentation, initialization, and augmentation strengthen DiCoW performance. Results also show robustness to imperfect enrollment, but real diarization remains a limitation, especially with more than two overlapping speakers.
- Oracle evaluation provides an upper bound, while baseline DiCoW falters most in Libri3Mix-both, where three recordings overlap without temporal offsets.
- Corrected training-data segmentation, refined initialization, and augmentation consistently reduce tcpWER, producing further gains with DiCoW v3.3.Segmentation improvements are particularly noted on AMI and NOTSOFAR-1 with long-form sequential decoding.
- SE-DiCoW achieves the lowest tcpWER across all reported real and synthetic benchmarks.Table 1 reports tcpWER under oracle and DiariZen diarization conditions.
- More than 75% relative tcpWER reduction is achieved on Libri3Mix-clean versus the original DiCoW.
- Real DiariZen diarization causes noticeable degradation, particularly on datasets with more than two simultaneously overlapping speakers.DiariZen models at most two active speakers, and Libri3Mix shows a consistently missing speaker through mean speaker counting error.
- 4.1. Analysis of Self-Enrollment Mixture Composition: SE-DiCoW performs best with three enrollment speakers and 25% target overlap, reaching 9.61% tcpWER on Libri3Mix-clean.With fully overlapped enrollment, tcpWER increases to 12.2% and 12.4% for four and five speakers, respectively.
5. CONCLUSION
SE-DiCoW resolves ambiguity in DiCoW’s STNO conditioning by selecting target-speaker reference segments through self-enrollment. It reduces macro-average tcpWER by 52.4% over DiCoW and maintains gains across diverse datasets, including real conversational data.
- SE-DiCoW addresses STNO ambiguity during fully overlapped speech by selecting target-speaker reference segments and adding them through cross-attention.
- 52.4% macro-average tcpWER reduction over DiCoW is achieved on the EMMA MT-ASR benchmark.The paper also reports over 75% relative improvement on Libri3Mix-clean and consistent gains on real conversational data.
- Enrollment analysis shows robustness to imperfect reference segments, while improved initialization, segmentation, and augmentation contribute to overall effectiveness.
- SE-DiCoW reaches performance on par with the best domain-tuned systems while preserving DiCoW’s cross-domain generalization.