Source-linked AI summary
Integration of speech separation, diarization, and recognition for multi-speaker meetings: System description, comparison, and analysis
Desh Raj, Pavel Denisov, Zhuo Chen, Hakan Erdogan, Zili Huang, Maokui He, Shinji Watanabe, Jun Du, Takuya Yoshioka, Yi Luo, Naoyuki Kanda, Jinyu Li, Scott Wisdom, John R. Hershey
TL;DR
Long, unsegmented multi-speaker recordings pose challenges for diarization and ASR, particularly when speech overlaps. This paper compares a modular separation, diarization, and recognition pipeline on LibriCSS. Its best pipeline with separation had cpWER 46.9% relative better than one without it.
Problem
Long, unsegmented conversations with an unknown number of participants require identifying who spoke what and when, while overlapping speech challenges multi-speaker processing.
Source: §1 · p. 1Method
Results
The best pipeline with separation had cpWER 46.9% relative better than one without it.
Source: §6 · p. 6Takeaways & Limitations
Because the modules interact only at the I/O level, they can be implemented with different tools and optimized independently.
Source: §1 · p. 2TS-VAD uses a fixed number of binary output nodes, so the authors assumed no session had more than eight speakers.
Source: §4.3 · p. 4
Abstract
from arXiv · showhide
Multi-speaker speech recognition of unsegmented recordings has diverse applications such as meeting transcription and automatic subtitle generation. With technical advances in systems dealing with speech separation, speaker diarization, and automatic speech recognition (ASR) in the last decade, it has become possible to build pipelines that achieve reasonable error rates on this task. In this paper, we propose an end-to-end modular system for the LibriCSS meeting data, which combines independently trained separation, diarization, and recognition components, in that order. We study the effect of different state-of-the-art methods at each stage of the pipeline, and report results using task-specific metrics like SDR and DER, as well as downstream WER. Experiments indicate that the problem of overlapping speech for diarization and ASR can be effectively mitigated with the presence of a well-trained separation module. Our best system achieves a speaker-attributed WER of 12.7%, which is close to that of a non-overlapping ASR.
1. INTRODUCTION
Multi-speaker speech recognition must determine who spoke what and when in long, unsegmented conversations, where overlapping speech creates challenges for diarization and ASR. The paper proposes a modular separation→diarization→recognition pipeline and compares methods for LibriCSS.
1. INTRODUCTION
- Almost 20% of multi-speaker recordings may contain overlapped speech, which challenges diarization and ASR on long, unsegmented conversations.Single-speaker diarization can miss an interfering speaker, while ASR trained on clean utterances is more error-prone on overlapped regions; these factors may increase speaker-attributed WER by up to 86% in meetings.Source: §1 · p. 1
- Short, fully overlapped mixture evaluations may miss distortions in overlap-free regions and can assume the number of speakers is known.Continuous speech separation has been explored for long-form conversations with partial overlap.Source: §1 · p. 1
- Prior work includes CHiME-6’s unsegmented dinner-party recognition track and LibriCSS pipelines combining continuous speech separation with hybrid ASR.Related efforts also include LibriMix and end-to-end joint modeling of speaker counting, identification, and ASR.Source: §1 · pp. 1–2
- 12.7% and 23.9% cpWER were achieved by the best system with and without separation, respectively, on the LibriCSS evaluation set.The pipeline’s modules interact only at the I/O level and can be implemented with different tools and optimized independently.Source: §1 · p. 2
2. SYSTEM OVERVIEW
The proposed system processes unsegmented multichannel audio through separation, diarization, and ASR. This ordering permits speaker-biased ASR while keeping separation independent of diarization.
2. SYSTEM OVERVIEW
- The pipeline orders its modules as separation, diarization, then ASR, passing separated streams to diarization and homogeneous single-speaker segments to an ASR decoder.Diarization considers all streams together because a speaker may be split across streams in different windows.Source: §2 · p. 2
- The CHiME-6 pipeline diarizes first, enabling diarization-guided source separation, while CSS separates first and then performs ASR and diarization, allowing streaming deployment.Poor diarization can affect CHiME-6 separation; CSS makes speaker-biased ASR infeasible and may produce spurious insertions from cross-talk.Source: §2 · p. 2
- Unlike the CHiME-6 pipeline, this pipeline’s separation does not depend on diarizer performance and can use speaker-independent methods such as mask-based MVDR or sequential multi-frame separation.The diarization stage sees only single-speaker recordings, avoiding the need for an overlap-aware diarization component.Source: §2 · pp. 2–3
- Experiments use LibriCSS’s 10 approximately one-hour simulated-conversation sessions, with session 0 for development and the remaining nine for evaluation.Diarization is evaluated with DER and recognition with cpWER, which minimizes total WER over speaker permutations.Source: §2 · p. 3
3. SPEECH SEPARATION
The section compares mask-based MVDR and sequential multi-frame separation, then reports their separation and downstream performance on LibriCSS. The sequential system achieves higher SDR, while downstream outcomes also depend on diarization and post-processing.
3.1. Mask-based MVDR
- Mask-based MVDR segments the recording into overlapping chunks, predicts two speech masks and one noise mask per chunk, stitches masks across chunks, then applies adaptive MVDR beamforming.The chunks are 2.4 seconds with a 0.8-second hop; stitching tracks local permutations using mask similarity in the 1.6-second overlap region.Source: §3.1 · p. 3
3.2. Sequential multi-frame separation
- Sequential multi-frame separation uses three untied mask-based beamforming steps, feeding previously beamformed signals into the next mask-prediction step.The model was trained on 10-second three-speaker mixtures and applied to LibriCSS in 8-second blocks with 4 seconds of overlap.Source: §3.2 · p. 3
3.3. Experimental results
- On a simulated evaluation set requiring reference signals, with oracle track mapping, the sequential multi-frame model achieved 14.1 dB average meeting-level SDR versus 5.8 dB for mask-based MVDR.The separation evaluation differed from LibriCSS because reference signals are required; the implemented MVDR beamformer does not directly reconstruct target signals at the reference microphone.Source: §3.3 · p. 3, Table 1 · p. 4
- The two methods had comparable diarization performance, but three-stream sequential separation outperformed two-stream mask-based MVDR in downstream cpWER.A diarization-output post-processing step improved sequential-model cpWER by 15.4% relative, whereas MVDR gained only 0.2% relative; the differing SDR and cpWER outcomes indicate a potential objective mismatch.Source: §3.3 · p. 4
4. SPEAKER DIARIZATION
The study compares diarization methods suited to separated streams, where speakers may appear in different streams over time. On mixed audio, overlap-assigning RPN and TS-VAD outperform clustering methods; on separated audio, spectral clustering (SC) outperforms AHC, while RPN and TS-VAD yield significantly higher cpWER than SC.
4. SPEAKER DIARIZATION
- The selected diarization methods must handle speakers across separated streams because windowed separation can place one speaker in different streams at different times.Overlap awareness is not a strict requirement after separation.Source: §4 · p. 4
4.1. X-vector + clustering
- X-vector clustering uses SAD followed by clustering of subsegment embeddings, with AHC on PLDA scores, SC on cosine similarity, or VBx initialized from AHC.The clustering methods make no assumption about the number of speakers; AHC and VBx use PLDA trained on Librispeech.Source: §4.1 · p. 4
4.2. Region proposal networks (RPN)
- RPN jointly optimizes segmentation and embedding extraction, then clusters region embeddings with K-means and applies non-maximal suppression.It is trained on simulated meeting-style recordings with partial overlaps and assumes the oracle number of speakers is known.Source: §4.2 · p. 4
4.3. Target-speaker voice activity detection (TS-VAD)
- TS-VAD takes conventional speech features and speaker i-vectors as inputs and predicts frame-level activity for each speaker; its output layer assumes at most 8 speakers per session.Initial speaker i-vectors came from the SC system, and training used simulated meeting-style data.Source: §4.3 · p. 4
4.4. Experimental results
- On mixed LibriCSS, RPN and TS-VAD outperformed clustering-based methods, including in low-overlap regions.The authors conjecture the low-overlap difference may arise from mismatch between PLDA training data and the evaluation set; SC performed better than other clustering methods.Source: §4.4 · p. 4, Table 2 · p. 5
- 13.8% absolute DER improvement resulted when AHC used cosine similarity instead of PLDA scores on separated audio; SC also significantly outperformed AHC.On separated streams, SC had better diarization performance than AHC, likely because AHC's PLDA was trained on acoustically different clean Librispeech utterances.Source: §4.4 · p. 5, Table 3 · p. 5
- RPN DER on separated audio improved from 26.9% to 22.4% after filtering non-speech segments with the study's SAD.RPN and TS-VAD cpWER was significantly higher than SC's; TS-VAD degraded from mixed to separated audio across missed speech, false alarms, and speaker confusions.Source: §4.4 · p. 5, Table 3 · p. 5
5. SPEECH RECOGNITION
The section compares hybrid and Transformer-based ASR models on LibriCSS, then evaluates them with diarization and separation in the full pipeline. Its final comparison reports that a strong diarization-and-ASR combination is hamstrung without explicit separation.
5.1. TDNN-F based hybrid HMM-DNN
- The hybrid base model uses a 17-layer factored-TDNN network trained on 960 hours of Librispeech, and its fine-tuned variant uses one epoch of reverberated-data adaptation.The fine-tuned model was used to evaluate downstream ASR performance for separation and diarization methods.Source: §5.1 · p. 5
5.2. Transformer-based end-to-end ASR
- The E2E system is an ESPNet Transformer encoder-decoder trained with joint sequence-to-sequence and CTC objectives; it obtains 2.2% and 5.5% WER on Librispeech test-clean and test-other.Decoding uses beam search combining sequence-to-sequence, CTC, and external Transformer language-model scores.Source: §5.2 · pp. 5–6
5.3. Experimental results
- Fine-tuning the hybrid model on reverberated data yielded a 31.5% relative WER improvement, while a beam size of 30 improved Transformer WER by almost 10% relative.Increasing the Transformer beam size further gave no significant gains.Source: §5.3 · p. 6
- With diarization replacing oracle segments, hybrid and E2E cpWER degraded by 23.4% and 37.6%, respectively; separation improved cpWER, with E2E changing from 27.1% to 13.4%.The increase from replacing oracle segments was more prominent in low- and medium-overlap conditions; the authors attribute this primarily to incorrect speaker assignment.Source: §5.3 · p. 6
- The final comparison evaluates best-model pipelines with and without separation and reports that the no-separation combination is hamstrung.Table 6 compares the two pipeline variants.Source: Table 6 · p. 6, §5.3 · p. 6
6. DISCUSSION
The discussion identifies separation as a major contributor to pipeline performance, while noting that diarization results on mixed audio do not transfer reliably to separated streams. It also finds that clean-trained ASR integrates well with the pipeline.
6. DISCUSSION
- The best pipeline with separation has 46.9% relatively better cpWER than the best pipeline without separation.The authors qualify the chunk-size result: improved performance with larger separation chunks comes through diarization tricks that filter increased cross-talk.Source: §6 · p. 6
- RPN and TS-VAD gain substantially on mixed recordings but do not transfer well to separated streams, where traditional clustering can outperform them.The best diarization without separation was 45.8% relatively better than one with separation, a counterintuitive result the authors flag for further investigation.Source: §6 · p. 6
- ASR performance on mixed recordings strongly indicates performance on separated streams, and a state-of-the-art model trained on clean single-speaker utterances achieves the best available cpWER on this dataset.Source: §6 · p. 6