Source-linked AI summary
Joint Speech Recognition and Speaker Diarization via Sequence Transduction
Laurent El Shafey, Hagen Soltau, Izhak Shafran
TL;DR
Understanding multi-speaker conversations requires recognizing words and assigning them to speakers, while conventional systems use separate ASR, diarization, and reconciliation stages. The paper casts joint ASR and speaker diarization as sequence transduction with an RNN-T that uses acoustic and linguistic cues to infer speaker roles. Against the strong baseline, WDER dropped from 15.8% to 2.2% (about 86% relative improvement), with about 0.6% degradation in WER.
Problem
Understanding multi-speaker conversations requires recognizing words and assigning them to speakers, while conventional systems use separate ASR, diarization, and reconciliation stages.
Source: §1 · p. 1Method
The paper casts joint ASR and speaker diarization as sequence transduction with an RNN-T that uses acoustic and linguistic cues to infer speaker roles.
Source: §1 · p. 1Results
Against the strong baseline, WDER dropped from 15.8% to 2.2% (about 86% relative improvement), with about 0.6% degradation in WER.
Source: §3.4 · p. 4Takeaways & Limitations
The approach is particularly well-suited to conversations with few speakers and well-defined roles, and requires matched speaker-role-labeled training data.
Source: §4 · p. 4Evaluation was limited by the lack of a public corpus for assessing the model, and the contribution of lexical cues could not be isolated.
Source: §4 · p. 4
Abstract
from arXiv · showhide
Speech applications dealing with conversations require not only recognizing the spoken words, but also determining who spoke when. The task of assigning words to speakers is typically addressed by merging the outputs of two separate systems, namely, an automatic speech recognition (ASR) system and a speaker diarization (SD) system. The two systems are trained independently with different objective functions. Often the SD systems operate directly on the acoustics and are not constrained to respect word boundaries and this deficiency is overcome in an ad hoc manner. Motivated by recent advances in sequence to sequence learning, we propose a novel approach to tackle the two tasks by a joint ASR and SD system using a recurrent neural network transducer. Our approach utilizes both linguistic and acoustic cues to infer speaker roles, as opposed to typical SD systems, which only use acoustic cues. We evaluated the performance of our approach on a large corpus of medical conversations between physicians and patients. Compared to a competitive conventional baseline, our approach improves word-level diarization error rate from 15.8% to 2.2%.
1. Introduction
Conversational understanding requires both recognizing words and determining who spoke when, but conventional systems handle these tasks in separate stages. The paper proposes a joint sequence-transduction approach that uses acoustic and linguistic cues to assign speaker roles.
1. Introduction
- Conventional conversation processing transcribes words, predicts who spoke when, and then reconciles the separate system outputs.The separate stages can be problematic because diarization systems may not respect word boundaries.Source: §1 · p. 1
- Many diarization systems segment speech, extract speaker embeddings, and cluster them, relying primarily on acoustic information to assign labels.Prior work has also explored learned embeddings and supervised alternatives to clustering.Source: §1 · p. 1
- Conversation roles can be reflected in language use, such as physicians asking about symptoms and patients seeking treatment clarification.Earlier work used linguistic cues to associate labels with conventional diarization segments or predict speaker changes over word windows.Source: §1 · p. 1
- The proposed system jointly performs speech recognition and speaker diarization by defining them as a sequence-transduction task implemented with an RNN-T.It uses acoustic and linguistic cues and is designed for conversations with well-defined speaker roles.Source: §1 · p. 1
2. Diarization via Sequence Transduction
The section formulates joint speech recognition and speaker diarization as sequence transduction with an RNN-T, combining acoustic and linguistic cues to predict words and speaker roles. It describes the transducer’s alignment and network components, then details its implementation for clinical conversations.
2.1. Problem Formulation and Proposed Solution
- Unlike CTC, which does not model output dependencies, RNN-T extends CTC with a language-model component.The section identifies these properties as motivation for using RNN-T for joint ASR and diarization.Source: §2.1 · p. 2
- RNN-T jointly predicts words and speaker-role tokens, using acoustic and linguistic cues rather than treating diarization as a separate task.The authors note that RNN-Ts can emit richer target symbols and integrate acoustic and linguistic information; physician and patient roles are represented by tokens such as <spk:dr> and <spk:pt>.Source: §2.1 · p. 2
2.2. Recurrent Neural Network Transducer
- RNN-T defines output-sequence probability by marginalizing over alignments, where blank symbols distinguish alignments that collapse to the same output.The input is a sequence of acoustic frames, and the output sequence includes speaker roles.Source: §2.2 · p. 2
- The encoder maps acoustic frames to representations, the prediction network uses previous nonblank symbols, and the joint network combines their outputs into symbol probabilities.The model is optimized by maximizing the log-likelihood of the output sequence given the input.Source: §2.2 · p. 2
2.3. Model Implementation
- Training uses conversation segments of at most 15 seconds, potentially containing multiple speakers, with 80-dimensional log-mel filterbank energies as acoustic frames.Targets are transcripts decorated with speaker roles.Source: §2.3 · pp. 2–3
- Morphemes provide an intermediate unit between graphemes and words, while TDNN layers reduce time resolution from 10 ms to 80 ms for more efficient training and inference.The encoder uses three blocks, each with a temporal convolutional layer and three bidirectional LSTM layers; training uses Adam on 128 TPUs and converges in under two days.Source: §2.3 · p. 3
3. Experiments
The experiments evaluate joint speech recognition and diarization on clinical conversations using word-level metrics and a conventional baseline. The proposed system substantially lowers WDER, with a small WER degradation.
3.1. Corpus
- The corpus comprised about 100K manually transcribed physician–patient clinical-conversation audio recordings (approximately 15K hours), with 508 conversations held out for development and 404 for evaluation; no physicians overlapped across splits.Source: §3.1 · p. 3
3.2. Evaluation Metric
- WDER measures words carrying the correct speaker tag and must be paired with WER because insertions and deletions cannot be mapped unambiguously to reference speaker labels.WDER counts substitutions with incorrect speaker tokens and correct ASR words with incorrect speaker tokens.Source: §3.2 · p. 3
3.3. Baseline
- The conventional baseline combined an RNN-T ASR model with speech detection, speaker embeddings, change detection, clustering, and word-boundary reconciliation.It used VoxCeleb2 for embedding training and more data than the joint RNN-T; its generic speaker labels were mapped to roles per conversation to minimize diarization errors.Source: §3.3 · pp. 3–4
3.4. Results and Analysis
- For most conversations, proposed-system WDER was under 4%, while baseline WDER had a wide spread around a mode of 11%.In proposed-system outliers above 15%, speaker-change detections rarely failed; incorrect role assignments could persist across a large segment, which the authors suspected was related to chunked-batch inference.Source: Figure 5 · p. 4, §3.4 · p. 4
- WDER fell from 15.8% to 2.2% versus the strong baseline, while WER degraded about 0.6%.This is a relative WDER improvement of about 86%.Source: Table 1 · p. 4, §3.4 · p. 4
- Across 3,000–15,000 training hours, more data seemed more helpful for WER than WDER; model size was not tuned.Source: Figure 6 · p. 4, §3.4 · p. 4
4. Conclusions And Future Work
The joint ASR and speaker diarization system improved word-level diarization error rate by 86% relative to a conventional baseline, without significant word error rate degradation. Its intended setting is conversations with few speakers and well-defined roles; future work includes other role-defined applications and richer transcripts.
4. Conclusions And Future Work
- An 86% relative improvement in word-level diarization error rate over a conventional baseline came without significant degradation in word error rate.Source: §4 · p. 4
- The system is particularly suited to conversations with few speakers and well-defined roles, and requires matched speaker-role-labeled training data.The authors note that no public corpus was available for evaluating the model for the benefit of the larger community.Source: §4 · p. 4
- Future work includes evaluation on other applications with clear speaker roles and richer transcripts with punctuation, capitalization, and potentially non-verbal cues such as emotions.Source: §4 · p. 4