Source-linked AI summary
Pushing the Boundaries of Streaming Multi-Speaker ASR: A Systematic Study of Architectural Trade-offs
Taejin Park, Ivan Medennikov, Kunal Dhawan, Weiqing Wang, Jagadeesh Balam, Boris Ginsburg
TL;DR
Streaming multi-speaker ASR must balance practical constraints and accurate recognition of overlapping speech, yet architectural trade-offs have not been systematically quantified. The paper compares four approaches built from shared open-source streaming ASR and diarization models, and finds that diarization conditioning currently has the accuracy advantage while PI-DTW addresses an alignment bottleneck for SOT training.
Problem
Architectural trade-offs for streaming multi-speaker ASR have not been comprehensively quantified, despite deployment constraints such as limited training data, restricted model access, and memory limits.
Source: §1 · p. 1Method
The paper compares four architectures based on shared open-source streaming ASR and diarization models and introduces PI-DTW to align speakers and words for SOT training.
Source: §1 · p. 1, §2.3 · p. 2Results
Takeaways & Limitations
PI-DTW provides a path for scaling SOT approaches, while specialized diarization conditioning currently holds the accuracy advantage.
Source: §1 · p. 2, §4 · p. 4Cascaded systems have limited overlap transcription, while masked-input parallel instances may increase computational and memory costs.
Source: §2.1 · p. 2, §2.2 · p. 2
Abstract
from arXiv · showhide
Streaming multi-speaker ASR is a challenging task that must balance accuracy, latency, and efficiency while handling overlapping speech and maintaining coherent long-context modeling over extended conversations in an online fashion. We present a unified framework that categorizes streaming multi-speaker ASR into four architectural strategies based on how diarization and ASR are integrated. Using a shared pair of open-source streaming ASR and diarization models as a common foundation, we derive four multi-speaker ASR systems that differ in whether they employ multiple model instances, fine-tuning, or both. We evaluate these systems across multi-speaker accuracy, single-speaker accuracy degradation, memory footprint, and training complexity. Through this systematic architectural analysis, we clarify the design space for streaming multi-speaker ASR and provide practical guidance for selecting the most suitable approach under diverse deployment constraints.
1. Introduction
Streaming multi-speaker ASR is increasingly needed for responsive voice agents, but architectural trade-offs remain underexplored for real-world deployment. This work compares four streaming designs and evaluates their accuracy, resource, and training trade-offs.
1. Introduction
- Streaming multi-speaker ASR supports low-latency voice agents and speech-to-speech systems, helping maintain responsive conversational flow.The demand is driven by applications such as voice agents powered by duplex models.Source: §1 · p. 1
- Most earlier multi-speaker ASR research focused on offline settings, while token-level SOT later enabled streaming multi-talker ASR.SOT serializes overlapped speech; subsequent work developed enhanced variants of that framework.Source: §1 · p. 1
- Diarization-based modular systems process separated speaker segments with single-speaker ASR, and guided source separation can further estimate spectral masks.Input features may be masked so each ASR instance transcribes one speaker at a time; guided source separation is often applied to improve separation.Source: §1 · p. 1
- A systematic comparison is needed because deployment constraints include limited training or fine-tuning data, restricted model-parameter access, and memory limits.Prior work had not thoroughly quantified trade-offs among architectural choices for multi-speaker ASR.Source: §1 · p. 1
- The study compares four streaming multi-speaker ASR architectures derived from the same open-source ASR and diarization models, evaluating accuracy, memory, and training complexity.It also introduces a transcription-to-token alignment algorithm for SOT training data preparation.Source: §1 · pp. 1–2
2. Architectural Trade-offs
The paper groups streaming multi-talker ASR into four architectural paradigms according to model-instance use and retraining requirements, using open-source streaming ASR and diarization models as a common foundation.
2. Architectural Trade-offs
- The four representative paradigms differ in their use of multiple model instances and whether they require retraining.The shared foundation is Nemotron Speech streaming ASR, based on RNN-T and FastConformer, together with streaming Sortformer v2.1 diarization.Source: §2 · p. 2
- Nemotron Speech is the pretrained streaming ASR model used in the framework, built on an RNN-T and FastConformer encoder.Source: §2 · p. 2
- Streaming Sortformer v2.1 supplies the diarization model for the architectural comparison.Source: §2 · p. 2
2.1. Cascaded ASR and Diarization
The cascaded architecture runs diarization and ASR independently, then attributes recognized words to speakers using their timestamps and diarization logits. It avoids retraining and fits API-based deployments, but is limited on overlapping speech and can misassign speakers at boundaries.
2.1. Cascaded ASR and Diarization
- Cascaded ASR and diarization operate independently, with no interaction between the two systems.Speaker attribution maps ASR word timestamps to diarization logits and assigns each word to the speaker with the highest logit.Source: §2.1 · p. 2
- The approach requires no model retraining and has minimal engineering overhead, making it practical when only API-level recognizer access is available.Its modularity also supports compatibility with existing production systems using standard streaming ASR APIs.Source: §2.1 · p. 2
- The method has limited ability to transcribe overlapping speech because segments are handled independently and typically assume one dominant speaker.Source: §2.1 · p. 2
- Inaccurate ASR word timestamps can misalign segment boundaries and lead to speaker assignment errors.Source: §2.1 · p. 2
2.2. Diarization Masked Input Architecture
The masked-input architecture applies diarization-derived, speaker-dependent masks to acoustic features and decodes each speaker in parallel. It can improve overlap transcription without retraining, though parallel instances may raise computational and memory costs.
2.2. Diarization Masked Input Architecture
- Diarization-derived, speaker-dependent masks are applied to acoustic features to enable parallel decoding for each speaker.Source: §2.2 · p. 2, Figure 3 · p. 2
- Masked feature streams partially separate concurrent speakers and improve transcription of overlapping speech.Source: §2.2 · p. 2
- The method reduces boundary misalignment by avoiding reliance on ASR-generated timestamps for segmentation and requires no retraining.It reuses single-speaker ASR models, making it attractive for data-sparse languages.Source: §2.2 · p. 2
- Parallel multi-instance decoding may increase computational and memory costs as the number of speakers grows.Source: §2.2 · p. 2
2.3. Serialized Output Training (SOT)
The section extends serialized-output training to long-form streaming by using a diarization-driven speaker cache and introduces PI-DTW to align transcript speakers with diarization timestamps. It describes the alignment costs and training setup for this approach.
2.3. Serialized Output Training (SOT)
- PI-DTW jointly aligns speakers and words to resolve transcript–diarization label ambiguity, combining dynamic time warping with a speaker-frequency cost for imbalanced speakers.DTW is suited to similar speaking times, while the frequency cost distinguishes cases where one speaker dominates.Source: §2.3 · p. 2
2.3.1. PI-DTW Preliminaries: Notation and Local Cost
- The local cost measures each word’s speaker mismatch against permuted RTTM activity, normalized by the number of simultaneously active speakers.A matching speaker in a single-speaker frame costs 0; when two speakers overlap and the text speaker is active, it costs 0.5; an inactive text speaker costs 1.Source: §2.3.1 · p. 3
2.3.2. Cost Calculation Mechanisms
- The alignment mechanism uses three DTW transitions and batches all candidate permutations, with total computational complexity O(P · K · T).A speaking-time frequency cost compares normalized text-implied and RTTM speaker ratios; the optimal permutation minimizes the sum of DTW and frequency costs.Source: §2.3.2 · p. 3
2.3.3. Model Training
- Training uses AMI, ICSI, DipCo, and Fisher English data, first training on 8–12-second segments for 50k steps and then fine-tuning on 10–55-second utterances for 20k steps.The encoder is initialized from a checkpoint, while the RNN-T decoder is randomly initialized; fine-tuning uses a learning rate of 2 × 10−4.Source: §2.3.3 · pp. 3–4
2.4. Diarization-Conditioned Multi-Instance Schemes
SSA is a streaming-capable diarization-conditioned approach built on pretrained Nemotron Speech that spawns speaker-specific ASR instances. It requires specialized fine-tuning and has a larger memory footprint than single-instance approaches.
2.4. Diarization-Conditioned Multi-Instance Schemes
- SSA directly ingests diarization predictions to spawn speaker-specific ASR instances and is built on the pretrained Nemotron Speech streaming model.The paper classifies it as diarization conditioning.Source: §2.4 · p. 4
- SSA delivers superior overlapping-speech performance with negligible single-speaker accuracy regression, but its multi-instance design increases memory and computation costs, especially with many speakers.The method also requires specialized fine-tuning to integrate speaker conditioning.Source: §2.4 · p. 4
2.4.1. Model Training
- Training follows the SSA methodology, adding Granary single-speaker data alongside Fisher, LibriSpeechMix, AMI, ICSI, and NOTSOFAR-1, with the SSA study’s configurations unchanged.The shared configurations include tokenizers, optimizers, learning-rate schedulers, and hardware specifications.Source: §2.4.1 · p. 4
3. Experiments and Results
The experiments compare four architectural paradigms on conversational datasets and report multi-speaker cpWER alongside single-speaker WER. SSA leads the reported multi-speaker comparisons and largely preserves the base model’s single-speaker performance.
3. Experiments and Results
- The evaluation covers CH109, a 109-file two-speaker Call-Home subset, Mixer6, and AMI under IHM and SDM conditions.The paper also benchmarks against recently published multi-speaker ASR systems, using identical audio files and annotations for open-source models.Source: §3 · p. 4
- SSA achieves the best multi-speaker performance; WL-SOT slightly outperforms masked input, while the cascaded method has the lowest multi-speaker accuracy.Table 1 reports cpWER across four datasets; its oracle columns use ground-truth diarization to remove degradation from diarization errors.Source: §3 · p. 4
- SSA largely preserves Nemotron Speech ASR’s single-speaker performance, whereas WL-SOT exhibits significant single-speaker accuracy degradation.Table 2 reports WER on the Hugging Face OpenASR leaderboard datasets.Source: §3 · p. 4
4. Conclusion
The paper formalizes streaming multi-talker ASR architectural trade-offs and introduces PI-DTW to resolve label alignment for WL-SOT training. It identifies diarization conditioning as the current accuracy leader while positioning resolved SOT alignment as a basis for future exploration of joint optimization.
4. Conclusion
- Diarization conditioning currently holds the accuracy advantage, while PI-DTW resolves the label-alignment problem for WL-SOT training and enables exploration of end-to-end joint optimization.The conclusion describes joint optimization as future exploration, not an evaluated result.Source: §4 · p. 4
- Future work will scale WL-SOT training data and support speaker tagging natively in Attention-based Encoder-Decoder models.Source: §4 · pp. 4–5
- The work formalizes architectural trade-offs in streaming multi-talker ASR and introduces PI-DTW to resolve label alignment for WL-SOT training.Source: §4 · p. 4
5. Generative AI Use Disclosure
The authors report using generative AI services only for technical assistance and retain responsibility for the final work.
5. Generative AI Use Disclosure
- Generative AI use was limited to grammar, LaTeX, and table-formatting assistance; the authors say they produced the substantive research and interpretations and take responsibility for the final work.Source: §5 · p. 5