Source-linked AI summary
Mitigating Speaker Leakage in Cascaded Multi-talker ASR with Diarization-based Transcript Correction
Hermann Yepdjio Nkouanga, Minwei Luo, Maggie Wigness, Suresh Singh
TL;DR
Speaker leakage from imperfect separation limits cascaded multi-talker ASR by contaminating separated streams and causing speaker-attribution errors. The paper introduces diarization-based pruning that removes verified leakage artifacts, achieving relative WER improvements of up to 29% on high-leakage AMI subsets.
Problem
Imperfect separation leaves residual interfering speech in output streams, causing significant speaker-attribution errors in cascaded multi-talker ASR.
Method
A pre-trained diarization model verifies acoustic, lexical, and temporal evidence to identify interfering-speaker segments whose transcribed words are pruned.
Results
Relative WER improvements reach up to 29% on high-leakage subsets of the AMI Meeting corpus, with robust gains across synthetic and real-world datasets.
Takeaways & Limitations
The pruning algorithm provides a versatile, computationally efficient enhancement that generalizes to organic meeting environments.
Takeaways & Limitations
The exploratory joint architecture is highly sensitive to training-data acoustics and SNR, degrading on cleaner LibriSpeechMix and AMI IHM conditions.
Abstract
from arXiv · showhide
While cascaded multi-talker ASR (MT-ASR) leverages state-of-the-art foundation models, its performance is often capped by speaker leakage during separation. Prior correction strategies primarily focus on lexical re-labeling for speaker attribution. We propose a complementary pruning-based paradigm that robustly identifies and removes leakage artifacts. Our method utilizes a pre-trained speaker diarization model as a multimodal verifier to prune transcribed segments satisfying a tripartite consensus of temporal containment, lexical cross-validation, and temporal alignment. Results on LibriMix, LibriSpeechMix, and the AMI Meeting corpus show our algorithm consistently reduces cpW ER across diverse overlap conditions. Specifically, on subsets with high speaker leakage, our method achieves relative cpW ER reductions of up to 29%, highlighting its effectiveness in enhancing the reliability of cascaded MT-ASR transcripts in complex acoustic environments.
1. Introduction
Cascaded MT-ASR performance is limited by speaker leakage caused by imperfect separation. This paper introduces a complementary diarization-based pruning approach that identifies and removes leakage artifacts from separated-stream transcripts.
- Problem: Imperfect separation leaves residual interfering speech in separated streams, causing significant final speaker-attribution errors.Speaker leakage is identified as a major limitation of cascaded systems.
- Prior work: Prior post-processing mainly re-labels misattributed words using lexical, LLM-based, or hybrid text-acoustic strategies.The paper positions pruning as complementary to these re-labeling approaches.
- Proposed method: The proposed algorithm uses an independent, pre-trained speaker diarization model to verify each separated audio stream and prune words from segments containing an interfering speaker.This approach removes leakage artifacts rather than relabeling them.
- Contributions: Experiments demonstrate significant cpWER reduction across multiple multi-talker datasets.The paper also evaluates a multi-task architecture with a diarization head integrated into a separation backbone.
2. Related Work
Related work frames multi-talker ASR around end-to-end and cascaded architectures, alongside post-processing methods for correcting speaker attribution. Existing approaches largely re-label transcripts, motivating pruning-based verification for speaker leakage and corrupted segments.
- Architectural paradigms: End-to-end MT-ASR jointly optimizes recognition and diarization, whereas cascaded systems chain independent separation and ASR models.End-to-end systems reduce error propagation but require massive multi-speaker datasets; cascaded systems remain adaptable to foundation models but depend on separation quality.
- Architectural paradigms: Cascaded systems are fundamentally limited by first-stage separation quality, which can produce speaker leakage and attribution errors.This limitation has driven substantial research into post-processing cascaded outputs.
- Post-processing methods: Early lexical correction used text context and first-pass diarization for word-level speaker classification, but weak acoustic grounding could cause over-correction.LLMs have also been explored for transcript re-diarization using semantic logic and Chain-of-Thought reasoning.
- Post-processing methods: Hybrid methods add acoustic evidence through lexical embeddings with frame-level posterior scores or discretized acoustic confidence prompts.AG-LSEC fuses lexical embeddings with frame-level posterior scores, while SEAL prompts LLMs with discretized acoustic confidence scores.
- Alternative strategies: Beyond re-labeling, prior alternatives include constrained diarization guided by speaker-turn tokens and human-in-the-loop interfaces for manual correction.The paper instead evaluates pruning-based verification and pruning of corrupted segments, while also examining a preventative diarization head in a separation backbone.
3. Methods
The proposed method uses diarization to detect speaker leakage and prunes transcript words only when acoustic, lexical, and temporal evidence agree. An exploratory multi-task model jointly performs separation and diarization through shared encoded features.
- Diarization-based pruning: The correction algorithm applies diarization to separated streams A and B to identify intervals containing interfering-speaker speech.For stream A, leakage windows W_A represent detected interference intervals.
- Diarization-based pruning: A transcript word is flagged as leakage only when acoustic containment, lexical cross-validation, and temporal alignment all hold.The word must lie within a leakage window, occur in the parallel transcript, and have an overlapping matching tuple there.
- Diarization-based pruning: Correction is bypassed when global transcript similarity falls below γ, with γ = 0.40 selected after evaluating values from 0.30 to 1.0.Similarity is computed using the Ratcliff/Obershelp pattern recognition algorithm.
- Multi-task architecture: The experimental multi-task framework augments MossFormer2 with a 2-layer transformer diarization head operating on shared 512-dimensional frame-level representations.The encoder produces these features through 24 interleaved FLASH self-attention and Gated Feedforward Sequential Memory Network iterations.
- Multi-task architecture: The shared model uses a weighted separation-and-diarization loss and a phased schedule that freezes the separation backbone for 10 epochs, then unfreezes it for 5 epochs.The diarization-loss weight is λ_diar = 20.0, selected empirically from candidates between 0.5 and 20.
4. Experimental Setup
The experiments evaluate the method on synthetic and real-world multi-talker datasets spanning fully and partially overlapping speech, noise, and meeting scenarios. Performance is measured with cpWER, using pretrained separation, recognition, and diarization models in the testing pipeline.
- Datasets: Evaluation covers synthetic and organic multi-talker datasets to assess robustness across varying acoustic conditions.The datasets include Libri2Mix, LibriSpeechMix, and the AMI Meeting Corpus.
- Datasets: 3,000-sample Libri2Mix test sets represent fully overlapping speech with and without WHAM noise, while LibriSpeechMix provides partially overlapping clean speech.The Libri2Mix both-train subset is used to fine-tune the exploratory diarization head.
- Datasets: 659 two-speaker AMI segments cover Single Distant Microphone and Individual Headset Mix scenarios after utterance-group partitioning.Only two-speaker groups are retained for consistency with the separation backbones.
- Evaluation metric: cpWER evaluates multi-talker ASR by computing WER across all speaker permutations and selecting the minimum.Because n = 2 in this study, the metric resolves speaker-order ambiguity for two references or speakers.
- System components: The pipeline separates speech with Mossformer2 and Sepformer, transcribes sources with Universal-2, and verifies segments with pyannote.audio 3.1.Universal-2 supplies word-level timestamps, while pyannote.audio 3.1 serves as the diarization verifier.
5. Results and Discussions
The speaker leakage fix consistently improves WER across test conditions, with its strongest gains on real-world AMI meeting data and high-leakage subsets. Ablation results show that acoustic grounding and tripartite multimodal consensus are central to robust correction.
- Overall Results: 10.55% relative improvement is achieved on AMI IHM with Sepformer, while Mossformer achieves 7.97% on AMI SDM.The fix improves performance across test conditions, with the most significant gains on real-world meeting data.
- High-Leakage Analysis: 29.26% relative reduction is the peak gain for Mossformer on high-leakage samples with transcript similarity > 0.4.The high baseline, including WER > 70% in AMI-SDM, identifies speaker leakage as a primary driver of ASR failure.
- Architectural Robustness: Despite N = 120 for Mossformer versus N = 84 for Sepformer in LibriSpeechMix, the fix delivers consistent large-scale improvements across backbones.This supports the fix’s versatility as a post-processing enhancement despite variation in identified leaked-sample counts.
- Real-World Generalization: WER drops from 65.58% to 46.39% for Mossformer IHM, showing strong efficacy in complex, partially overlapping meeting environments.The largest absolute gains occur in the AMI corpus rather than fully overlapping simulated data.
- Ablation Study: The full Text + Acoustic system is most robust, requiring Cac ∧ Clex ∧ Ctemp, whereas Text Only over-corrects and Acoustic Only confirms the primacy of signal-level diarization.Acoustic features provide leakage detection, while lexical cues provide essential grounding and temporal alignment constrains pruning.
6. Conclusions
The paper introduces a pruning-based paradigm that uses a pre-trained diarization model as a multimodal verifier to identify and remove speaker-leakage artifacts in cascaded multi-talker ASR. Experiments across synthetic and real-world datasets show robust WER reductions, including up to 29% relative improvement on high-leakage AMI subsets.
- Conclusions: The proposed pruning-based paradigm addresses speaker leakage more directly than ambiguous lexical re-labeling in high-leakage scenarios.It uses a pre-trained diarization model as a multimodal verifier to identify and remove spurious speech artifacts through tripartite consensus.
- Conclusions: The algorithm achieved relative WER improvements of up to 29% on high-leakage subsets of the AMI Meeting corpus.The reported gains are part of experimental results spanning synthetic and real-world datasets.
- Conclusions: The proposed enhancement generalizes to organic meeting environments while remaining computationally efficient.The experiments characterize the algorithm as versatile and robust across synthetic and real-world datasets.