Source-linked AI summary
Continuous Speech Separation with Conformer
Sanyuan Chen, Yu Wu, Zhuo Chen, Jian Wu, Jinyu Li, Takuya Yoshioka, Chengyi Wang, Shujie Liu, Ming Zhou
TL;DR
Conversation transcription is challenged by overlapped speech and rapid speaker turns, motivating continuous speech separation that can operate without utterance boundaries. The paper uses Conformer-based separation with mask estimation and streaming chunk-wise processing, achieving superior results to RNN-based systems on LibriCSS and real meetings, while longer-context training remains constrained by overlap handling.
Problem
Overlapped speech and rapid speaker turn-taking limit ASR for conversation transcription, while realistic continuous separation cannot assume utterance boundaries.
Method
The system uses Conformer-based mask estimation for continuous speech separation, with chunk-wise processing and context-aware self-attention for streaming inputs.
Results
Conformer outperformed RNN-based models in utterance-wise and continuous evaluations, with relative WER gains over BLSTM of 4% and 15% for hybrid and transducer ASR systems, respectively.
Takeaways & Limitations
The results indicate that appropriately using context information improves continuous separation and supports gains in single-channel real-meeting transcription.
Takeaways & Limitations
Longer context did not lower WER in large-overlap settings, partly because training did not include overlap regions between adjacent windows.
Abstract
from arXiv · showhide
Continuous speech separation plays a vital role in complicated speech related tasks such as conversation transcription. The separation model extracts a single speaker signal from a mixed speech. In this paper, we use transformer and conformer in lieu of recurrent neural networks in the separation system, as we believe capturing global information with the self-attention based method is crucial for the speech separation. Evaluating on the LibriCSS dataset, the conformer separation model achieves state of the art results, with a relative 23.5% word error rate (WER) reduction from bi-directional LSTM (BLSTM) in the utterance-wise evaluation and a 15.4% WER reduction in the continuous evaluation.
1. INTRODUCTION
Conversation transcription remains difficult when speech overlaps or speakers change quickly, because these conditions violate single-speaker assumptions and create permutation problems. The paper examines Conformer-based continuous speech separation to address this gap in realistic long-form conversations.
- Overlapped speech and rapid speaker turn-taking limit ASR performance in conversation transcription.These conditions violate the usual single-active-speaker assumption.
- Continuous speech separation differs from prior separation settings because it processes an ongoing mixture without relying on utterance boundaries.
- Conformer-based continuous speech separation is evaluated on LibriCSS, a dataset of long-form multi-talker sessions with varied overlap ratios.The system continuously separates mixed speech and routes utterances to overlap-free output channels for downstream ASR.
- The proposed system enables a standard single-speaker ASR system to process each separated output channel directly.
2. APPROACH
The approach estimates speaker and noise masks from multichannel or single-channel mixtures with a Conformer, combining self-attention and convolution for separation. Chunk-wise sliding-window processing supports streaming continuous separation while using history and future context.
- 2.1. Problem Formulation: The model estimates masks with a Conformer and applies them to the mixture STFT to obtain separated source STFTs.For multichannel inputs, adaptive MVDR beamforming is used to recover source signals.
- 2.2. Model structure: The architecture produces three masks: two for speakers and one for noise.
- 2.2. Model structure: Conformer blocks combine self-attention, convolution, and feed-forward modules to model global and local information.The convolution component is inserted into a Transformer block to improve local information modeling.
- 2.2. Model structure: Chunk-wise sliding-window processing computes current-segment masks from history, current, and future context for streaming separation.The window advances by the current-segment length at each step.
- 2.2. Model structure: Previous chunks can be retained in self-attention so keys and values include both prior and current context.The cache size depends on how many history chunks are used.
3. EXPERIMENT
Experiments evaluate Conformer-based separation on LibriCSS and real meetings under utterance-wise and continuous-input conditions. Conformer models outperform BLSTM-based systems, with gains also observed on real conversational recordings.
- 3.1. Datasets: Evaluation used LibriCSS utterance-wise and continuous-input schemes plus a 15.8-hour real conversation corpus.LibriCSS contains seven-channel mixed recordings, while the real corpus comprises daily group discussions with 3–22 speakers per meeting.
- 3.3. Results for utterance wise evaluation: Conformer models achieved state-of-the-art utterance-wise WER results across seven-channel and single-channel settings.Conformer-base substantially improved over BLSTM in seven-channel evaluation and generally outperformed Transformer-base.
- 3.4. Results for continuous input evaluation: 15% relative WER gains over BLSTM were obtained with Conformer-base for continuous evaluation using transducer ASR.The corresponding gain with hybrid ASR was 4%, and the gap narrowed on the large-overlap test set.
- 3.4. Results for continuous input evaluation: Self-attention-based models improved over BLSTM on continuous recordings with rapid speaker turn changes.The 0S dataset was more difficult than 0L in continuous evaluation because it contained more quick speaker-turn changes.
- 3.4. Results for continuous input evaluation: Longer-context ConformerXL models did not reduce WER, particularly at high overlap ratios.The authors associate this degradation with possible noise from longer history and a mismatch caused by excluding adjacent-window overlap during training.
- 3.5. Results on large scale real meetings: 11.8% WER and 18.4% SA-WER relative reductions were achieved on a real meeting dataset versus a system without separation.The improvements followed three described enhancements, and each introduced step consistently improved both metrics.
4. CONCLUSION
The paper investigates Conformer architectures for continuous speech separation and finds them effective across utterance-wise and continuous evaluations. Conformer also outperforms Transformer and yields gains in single-channel real meeting transcription.
- 4. CONCLUSION: Conformer outperformed RNN-based models in utterance-wise and continuous-input evaluations, and also surpassed Transformer.The conclusion attributes the results to appropriately utilizing context information in speech separation.