PaperlayerView exact revision on arXiv

Source-linked AI summary

Continuous Speech Separation with Conformer

Sanyuan Chen, Yu Wu, Zhuo Chen, Jian Wu, Jinyu Li, Takuya Yoshioka, Chengyi Wang, Shujie Liu, Ming Zhou

arXiv:2008.05773v2eess.AScs.CL

TL;DR

Conversation transcription is challenged by overlapped speech and rapid speaker turns, motivating continuous speech separation that can operate without utterance boundaries. The paper uses Conformer-based separation with mask estimation and streaming chunk-wise processing, achieving superior results to RNN-based systems on LibriCSS and real meetings, while longer-context training remains constrained by overlap handling.

  • Problem

    Overlapped speech and rapid speaker turn-taking limit ASR for conversation transcription, while realistic continuous separation cannot assume utterance boundaries.

    Source: §1 · p. 1

  • Method

    The system uses Conformer-based mask estimation for continuous speech separation, with chunk-wise processing and context-aware self-attention for streaming inputs.

    Source: §2.1 · p. 1

  • Results

    Conformer outperformed RNN-based models in utterance-wise and continuous evaluations, with relative WER gains over BLSTM of 4% and 15% for hybrid and transducer ASR systems, respectively.

    Source: §3.4 · p. 3, §4 · p. 4

  • Takeaways & Limitations

    The results indicate that appropriately using context information improves continuous separation and supports gains in single-channel real-meeting transcription.

    Source: §4 · p. 4, §3.5 · p. 4

    Longer context did not lower WER in large-overlap settings, partly because training did not include overlap regions between adjacent windows.

    Source: §3.4 · p. 4

Abstract

from arXiv · show

Continuous speech separation plays a vital role in complicated speech related tasks such as conversation transcription. The separation model extracts a single speaker signal from a mixed speech. In this paper, we use transformer and conformer in lieu of recurrent neural networks in the separation system, as we believe capturing global information with the self-attention based method is crucial for the speech separation. Evaluating on the LibriCSS dataset, the conformer separation model achieves state of the art results, with a relative 23.5% word error rate (WER) reduction from bi-directional LSTM (BLSTM) in the utterance-wise evaluation and a 15.4% WER reduction in the continuous evaluation.

1. INTRODUCTION

Conversation transcription remains difficult when speech overlaps or speakers change quickly, because these conditions violate single-speaker assumptions and create permutation problems. The paper examines Conformer-based continuous speech separation to address this gap in realistic long-form conversations.

  • Overlapped speech and rapid speaker turn-taking limit ASR performance in conversation transcription.These conditions violate the usual single-active-speaker assumption.Source: §1 · p. 1
  • Continuous speech separation differs from prior separation settings because it processes an ongoing mixture without relying on utterance boundaries.Source: §1 · p. 1
  • Conformer-based continuous speech separation is evaluated on LibriCSS, a dataset of long-form multi-talker sessions with varied overlap ratios.The system continuously separates mixed speech and routes utterances to overlap-free output channels for downstream ASR.Source: §1 · p. 1
  • The proposed system enables a standard single-speaker ASR system to process each separated output channel directly.Source: §1 · p. 1

2. APPROACH

The approach estimates speaker and noise masks from multichannel or single-channel mixtures with a Conformer, combining self-attention and convolution for separation. Chunk-wise sliding-window processing supports streaming continuous separation while using history and future context.

2.1. Problem Formulation

  • The model estimates masks with a Conformer and applies them to the mixture STFT to obtain separated source STFTs.For multichannel inputs, adaptive MVDR beamforming is used to recover source signals.Source: §2.1 · p. 1

2.2. Model structure

  • The architecture produces three masks: two for speakers and one for noise.Source: Figure 1 · p. 2
  • Conformer blocks combine self-attention, convolution, and feed-forward modules to model global and local information.The convolution component is inserted into a Transformer block to improve local information modeling.Source: §2.2 · p. 2
  • Chunk-wise sliding-window processing computes current-segment masks from history, current, and future context for streaming separation.The window advances by the current-segment length at each step.Source: §2.3 · p. 2, Figure 2 · p. 2
  • Previous chunks can be retained in self-attention so keys and values include both prior and current context.The cache size depends on how many history chunks are used.Source: §2.3 · p. 2

3. EXPERIMENT

Experiments evaluate Conformer-based separation on LibriCSS and real meetings under utterance-wise and continuous-input conditions. Conformer models outperform BLSTM-based systems, with gains also observed on real conversational recordings.

3.1. Datasets

  • Evaluation used LibriCSS utterance-wise and continuous-input schemes plus a 15.8-hour real conversation corpus.LibriCSS contains seven-channel mixed recordings, while the real corpus comprises daily group discussions with 3–22 speakers per meeting.Source: §3.1 · pp. 2–3, §3.5 · p. 4, Table 3 · p. 4

3.3. Results for utterance wise evaluation

  • Conformer models achieved state-of-the-art utterance-wise WER results across seven-channel and single-channel settings.Conformer-base substantially improved over BLSTM in seven-channel evaluation and generally outperformed Transformer-base.Source: §3.3 · p. 3, Table 1 · p. 3

3.4. Results for continuous input evaluation

  • 15% relative WER gains over BLSTM were obtained with Conformer-base for continuous evaluation using transducer ASR.The corresponding gain with hybrid ASR was 4%, and the gap narrowed on the large-overlap test set.Source: §3.4 · p. 3, Table 2 · p. 4
  • Self-attention-based models improved over BLSTM on continuous recordings with rapid speaker turn changes.The 0S dataset was more difficult than 0L in continuous evaluation because it contained more quick speaker-turn changes.Source: §3.4 · p. 3
  • Longer-context ConformerXL models did not reduce WER, particularly at high overlap ratios.The authors associate this degradation with possible noise from longer history and a mismatch caused by excluding adjacent-window overlap during training.Source: §3.4 · p. 4

3.5. Results on large scale real meetings

  • 11.8% WER and 18.4% SA-WER relative reductions were achieved on a real meeting dataset versus a system without separation.The improvements followed three described enhancements, and each introduced step consistently improved both metrics.Source: §3.5 · p. 4, Table 3 · p. 4

4. CONCLUSION

The paper investigates Conformer architectures for continuous speech separation and finds them effective across utterance-wise and continuous evaluations. Conformer also outperforms Transformer and yields gains in single-channel real meeting transcription.

4. CONCLUSION

  • Conformer outperformed RNN-based models in utterance-wise and continuous-input evaluations, and also surpassed Transformer.The conclusion attributes the results to appropriately utilizing context information in speech separation.Source: §4 · p. 4

Related papers

Loading 2008.05773v2…