Source-linked AI summary
Serialized Output Training for End-to-End Overlapped Speech Recognition
Naoyuki Kanda, Yashesh Gaur, Xiaofei Wang, Zhong Meng, Takuya Yoshioka
TL;DR
Monaural overlapped multi-speaker ASR remains difficult, while PIT is constrained by fixed output branches, independent speaker outputs, and costly permutation handling. The paper proposes SOT, which serializes speaker transcriptions with one attention-based encoder-decoder output layer and uses start times for efficient training. On LibriSpeech, SOT transcribes variable-speaker overlaps more effectively than PIT and accurately counts speakers.
Problem
Monaural overlapped speech is difficult for ASR, while PIT constrains speaker capacity, cannot model dependencies between speaker outputs, and requires O(S^3) training cost.
Method
SOT uses a single attention-based encoder-decoder output layer to generate multiple speaker transcriptions sequentially, with start-time ordering enabling O(S) execution.
Results
SOT transcribed monaural overlapped speech significantly more effectively than PIT on LibriSpeech while accurately counting the number of speakers.
Takeaways & Limitations
SOT supports overlapped-speech recognition with variable numbers of speakers without the maximum-speaker constraint imposed by PIT output layers.
Takeaways & Limitations
Minimum-loss speaker ordering requires O(S!) training cost, motivating the start-time-based first-in, first-out ordering scheme.
Abstract
from arXiv · showhide
This paper proposes serialized output training (SOT), a novel framework for multi-speaker overlapped speech recognition based on an attention-based encoder-decoder approach. Instead of having multiple output layers as with the permutation invariant training (PIT), SOT uses a model with only one output layer that generates the transcriptions of multiple speakers one after another. The attention and decoder modules take care of producing multiple transcriptions from overlapped speech. SOT has two advantages over PIT: (1) no limitation in the maximum number of speakers, and (2) an ability to model the dependencies among outputs for different speakers. We also propose a simple trick that allows SOT to be executed in $O(S)$, where $S$ is the number of the speakers in the training sample, by using the start times of the constituent source utterances. Experimental results on LibriSpeech corpus show that the SOT models can transcribe overlapped speech with variable numbers of speakers significantly better than PIT-based models. We also show that the SOT models can accurately count the number of speakers in the input audio.
1. Introduction
Overlapped multi-speaker ASR remains difficult, and PIT addresses it with multiple speaker-specific output layers but has important scalability and modeling limitations. SOT instead uses one output layer to serialize multiple speaker transcriptions and is reported to improve recognition while supporting variable speaker counts.
- 84.7% WER for two-speaker overlapped speech contrasts with 5.5% for single-speaker speech, illustrating the difficulty of monaural overlap.
- PIT uses multiple output layers and considers all speaker permutations during training, but its output-layer count limits the maximum number of speakers.
- Independent PIT output layers cannot model dependencies among speakers, allowing duplicated hypotheses and requiring extra treatment to reduce such errors.
- PIT has O(S^3) computational complexity, so most previous work addressed only two-speaker mixtures despite recordings often containing more speakers.
- SOT uses one output layer to generate multiple speaker transcriptions sequentially, avoiding a maximum-speaker constraint and modeling dependencies among outputs.A start-time-based trick reduces SOT execution to O(S).
2. Related Work
Prior single-model approaches separated multiple utterances with special symbols, but they did not address overlapping speech. SOT differs by applying serialized decoding to overlapped monaural speech and by using attention that scans encoder embeddings in both time directions.
- Earlier RNN-T and multilingual AED systems serialized multiple utterances with speaker-role or language tags, but they did not handle speech overlaps.
- Unlike prior AED-based ASR studies that imposed monotonic alignment constraints, SOT attention scans encoder embeddings back and forth along the time dimension.
- SOT separates multiple decoded utterances with the special symbol ⟨sc⟩, which denotes a speaker change.
3. Review: Multi-Speaker ASR Based on AED with PIT
AED-based ASR encodes the input, uses attention and a decoder to produce output symbols, and trains with cross-entropy loss. In PIT-based multi-speaker ASR, multiple output branches correspond to speakers and are matched to references over all speaker permutations, but this creates several theoretical limitations.
- AED-based single-speaker ASR: The AED encoder converts input X into embeddings Henc, after which attention produces context vectors for decoder steps.The attention computation uses the decoder state, previous attention weights, and encoder embeddings.
- AED-based single-speaker ASR: The decoder estimates each output distribution from the attention context and decoder state, using recurrent layers and an affine softmax output layer.Training minimizes cross-entropy against reference labels ending with the special ⟨eos⟩ symbol.
- PIT-based multi-speaker ASR: PIT uses multiple output branches, with shared attention and decoder parameters, to generate one output sequence for each speaker.Each branch index s ranges from 1 to S, where S is the number of speakers.
- PIT-based multi-speaker ASR: PIT training evaluates all possible speaker-to-output permutations when calculating the cross-entropy loss.The permutation function generates every permutation of the given sequence.
- PIT-based multi-speaker ASR: PIT is limited by its fixed number of output layers, independent output branches, possible duplicated hypotheses, and O(S^3) training cost.These limitations constrain the maximum speaker count and hinder application to recordings with many speakers.
4. Serialized Output Training
Serialized Output Training (SOT) replaces PIT’s multiple output branches with one decoder sequence containing multiple speaker transcriptions. It uses speaker start times for O(S) training and offers unrestricted speaker counts while modeling dependencies among utterances.
- SOT formulation: SOT serializes multiple speaker references into one token sequence, inserting ⟨sc⟩ between utterances and using ⟨eos⟩ only at the sequence end.The model therefore uses a single output branch to decode multiple speakers sequentially.
- Speaker ordering: Minimum-loss speaker ordering evaluates all speaker concatenations, but its training cost is O(S!), making it computationally expensive.This ordering strategy selects the permutation with the lowest loss, similarly to PIT.
- Speaker ordering: First-in, first-out SOT sorts references by utterance start time, reducing training cost to O(S).It assumes that two utterances do not start at exactly the same time; ties are randomly ordered.
- Advantages: A Separation after Attention LSTM addresses contamination in attention context vectors caused by overlapped speech.The added decoder-side LSTM acts as a separation module after attention, while one encoder layer is removed for model-size fairness.
- Advantages: Unlike PIT, SOT has no maximum-speaker constraint and can represent dependencies among speaker utterances, preventing duplicated hypotheses.PIT’s independent output layers cannot directly model these dependencies.
- Advantages: SOT can predict the number of speakers by counting ⟨sc⟩ and ⟨eos⟩ occurrences when trained on mixtures with varying speaker counts.Decoding continues until ⟨eos⟩ is predicted.
5.1. Evaluation settings
The evaluation uses simulated multi-speaker LibriSpeech mixtures, with WER measured after optimally matching hypothesis and reference speaker orders. Models use BLSTM encoders, LSTM decoders, and specified acoustic features, while Table 2 and Table 3 vary mixture conditions and model architectures.
- Data: Training uses LibriSpeech’s 960-hour train 960 set, with dev clean and test clean used for evaluation.The corpus contains about 1,000 hours of audiobook data.
- Data: Training mixtures randomly select the speaker count and add randomly delayed utterances while preserving each utterance’s original volume.The resulting average signal-to-interference ratio is about 0 dB.
- Data: Evaluation mixtures are generated similarly, except utterances may start simultaneously because the training delay constraint is not imposed.This creates evaluation cases with coincident utterance starts.
- Evaluation: WER is computed after examining all hypothesis permutations and selecting the ordering with the lowest WER.This accommodates systems that produce different numbers of hypotheses than reference speakers.
- Model settings: The encoder has 6 bidirectional LSTM layers, while the decoder has 2 unidirectional LSTM layers; PIT baselines share the first 5 encoder layers across branches.Encoder dimensions are 512, 724, or 1024.
- Evaluation tables: Table 2 reports WER for varied training and test speaker mixtures, marking unmatched training/testing conditions with parentheses.Test data are generated by mixing test clean.
- Evaluation tables: Table 3 reports WER across parameter counts and architectures for SOT trained on mixtures of 1, 2, and 3 speakers.Its test data are generated by mixing test clean.
5.2. Evaluation results
Experiments show that SOT handles overlapped speech across speaker counts, compares favorably with PIT, and can count speakers accurately, while larger models and SAA improve performance.
- Two-speaker mixed speech: SOT significantly outperformed normal single-speaker ASR in the two-speaker experiment.
- Variable numbers of speakers: SOT recognized overlapped speech with variable numbers of speakers without degradation on single-speaker data.The authors suggest mixing training utterances may provide a data augmentation effect.
- Model size and SAA: Larger model size was essential for good SOT results, and applying SAA further improved performance.The SAA model was smaller than the naive SOT model because an encoder layer was removed rather than adding a decoder-side layer.
- Comparison with PIT: SOT achieved better results than PIT on 1- and 2-speaker evaluation data while also recognizing 3-speaker data with fewer parameters.The 3-speaker SOT trained roughly 30% faster than the 2-output PIT model, including I/O.
- Speaker counting accuracy: 99.8% and 97.0% speaker-counting accuracy were achieved for 1-speaker and 2-speaker mixtures, respectively.The model sometimes underestimated the number of speakers in 3-speaker mixtures.
6. Conclusions
The paper concludes that SOT recognizes monaural overlapped speech with any number of speakers, executes in O(S), outperforms PIT, and accurately counts speakers.
- SOT recognizes overlapped speech consisting of any number of speakers.
- Using utterance start times lets SOT execute in O(S).
- On LibriSpeech, SOT transcribed monaural overlapped speech more effectively than PIT and accurately counted speakers.