Source-linked AI summary
Continuous speech separation: dataset and analysis
Zhuo Chen, Takuya Yoshioka, Liang Lu, Tianyan Zhou, Zhong Meng, Yi Luo, Jian Wu, Xiong Xiao, Jinyu Li
TL;DR
The paper addresses the mismatch between conventional speech-separation evaluations and continuous conversational audio, where overlap is partial and signal metrics correlate weakly with ASR accuracy. It introduces LibriCSS and ASR-based protocols, then evaluates a speaker-independent CSS system. CSS improves WER across continuous-input conditions, while residual overlap degradation and microphone-dependent limitations remain.
Problem
Existing evaluations often use fully overlapped, pre-segmented mixtures and signal metrics that correlate weakly with ASR accuracy, limiting assessment of conversational separation.
Method
The paper constructs the real-recorded LibriCSS dataset and evaluates speaker-independent CSS with Kaldi-based ASR protocols on utterance-wise and continuous inputs.
Results
CSS improved WERs in all seven-channel continuous-input conditions, including non-overlap cases, while MVDR outperformed TF masking in utterance-wise evaluation.
Takeaways & Limitations
LibriCSS and its evaluation pipeline provide a practical benchmark for studying continuous speech separation under varied overlap, silence, and microphone conditions.
Takeaways & Limitations
Even seven-channel MVDR leaves nontrivial overlap-related degradation, and single- or three-microphone separation can fail to improve or degrade WER when one speaker is active.
Abstract
from arXiv · showhide
This paper describes a dataset and protocols for evaluating continuous speech separation algorithms. Most prior studies on speech separation use pre-segmented signals of artificially mixed speech utterances which are mostly \emph{fully} overlapped, and the algorithms are evaluated based on signal-to-distortion ratio or similar performance metrics. However, in natural conversations, a speech signal is continuous, containing both overlapped and overlap-free components. In addition, the signal-based metrics have very weak correlations with automatic speech recognition (ASR) accuracy. We think that not only does this make it hard to assess the practical relevance of the tested algorithms, it also hinders researchers from developing systems that can be readily applied to real scenarios. In this paper, we define continuous speech separation (CSS) as a task of generating a set of non-overlapped speech signals from a \textit{continuous} audio stream that contains multiple utterances that are \emph{partially} overlapped by a varying degree. A new real recorded dataset, called LibriCSS, is derived from LibriSpeech by concatenating the corpus utterances to simulate a conversation and capturing the audio replays with far-field microphones. A Kaldi-based ASR evaluation protocol is also established by using a well-trained multi-conditional acoustic model. By using this dataset, several aspects of a recently proposed speaker-independent CSS algorithm are investigated. The dataset and evaluation scripts are available to facilitate the research in this direction.
1. INTRODUCTION
Natural conversational overlap challenges speech technologies, while existing separation evaluations often miss continuous, partially overlapped speech and practical ASR relevance. The paper addresses these gaps with LibriCSS and ASR-based evaluation.
- Overlapping speech challenges ASR and speaker diarization because these technologies usually assume one or zero active speakers.
- Existing studies often separate fully overlapped, pre-segmented signals rather than continuous recordings containing both overlap and non-overlap.Natural meetings typically have overlap ratios below 20%, so non-overlap distortion also matters.
- Signal-level metrics such as SDR and SISNR are only weakly correlated with ASR accuracy and perceptual sound quality.
- Continuous speech separation generates multiple overlap-free signals from an audio stream with occasional overlapped utterances.
- The paper creates LibriCSS, explores a speaker-independent CSS method, and releases the dataset with Kaldi-based ASR evaluation scripts.
2. CONTINUOUS SPEECH SEPARATION
Continuous speech separation processes long conversational streams containing overlapping and overlap-free speech, producing output streams with at most one active speaker per stream. The paper contrasts this with approaches requiring diarization, speaker information, or prior utterance boundaries.
- CSS generates overlap-free speech signals from potentially hours-long continuous audio containing both overlapped and overlap-free parts.In meeting conversations, adjacent utterances overlap by 10–15% on average.
- Alternative CSS strategies require online diarization, offline diarization with speaker embeddings, or prior utterance boundaries for detect-then-separate processing.
- Detect-then-separate must maintain speaker consistency when concatenating separated overlap segments with preceding and following signals.Overlap detection and separation are inter-dependent problems, making the sequential approach sub-optimal according to the paper.
- The investigated speaker-independent CSS method outputs a fixed number of streams, each containing at most one active speaker at any time.During non-overlap, speech is routed to one channel while the others produce zero or negligible noise.
3. LIBRICSS
LibriCSS is a real-room, multichannel dataset built from replayed LibriSpeech utterances to evaluate both utterance-wise and continuous-input speech separation. Its protocols combine varied overlap conditions, streaming separation, and ASR-based scoring.
- 3.1. Dataset: LibriCSS contains simulated conversational sessions made from LibriSpeech utterances replayed by loudspeakers in a room and captured as multichannel audio.The dataset is intended to support continuous-input evaluation while remaining broadly accessible.
- 3.1. Dataset: The dataset records real room acoustics, varies overlap ratios and silence settings, and provides continuous audio plus ground-truth segmentation.These features support both continuous-input and conventional utterance-wise evaluation.
- 3.1. Dataset: LibriCSS comprises 10 hours across 10 approximately one-hour sessions, with mini-session overlap ratios ranging from 0 to 40%.
- 3.1. Dataset: Recordings use a seven-channel circular microphone array in a regular meeting room, with loudspeaker-to-microphone distances from 33 cm to 409 cm.
- 3.3.1. ASR setup: The evaluation uses a multi-condition BLSTM acoustic model trained on 960 hours of LibriSpeech data, with Kaldi-based alignments and decoding.
- 3.3. Evaluation: Utterance-wise evaluation uses ground-truth segmentation, whereas continuous-input evaluation processes unsegmented audio and evaluates CSS algorithms.
- 3.3.3. Continuous input evaluation: Streaming CSS estimates time-frequency masks with a sliding window whose past, current, and future subwindows provide context.Neighboring chunks align output orders using their shared frames.
- 3.3.3. Continuous input evaluation: Continuous-input decoding uses 60–120-second segments bounded during silence, with Asclite2 estimating speaker-agnostic WERs.
4. EXPERIMENTAL RESULTS
Experiments evaluate a speaker-independent CSS system using utterance-wise and continuous-input ASR tests, varying overlap, chunking, and microphone configuration. Multi-channel beamforming improves continuous-input WER, while overlap and short turn-taking gaps remain challenging.
- Utterance-wise evaluation: A seven-channel system produced more accurate TF masks than a single-channel system across utterance-wise conditions.However, TF masking did not improve ASR accuracy in non-overlap conditions, whereas MVDR significantly outperformed TF masking.
- Continuous-input evaluation: The CSS algorithm improved WER in all seven-channel continuous-input conditions, including 0S and 0L non-overlap cases, through beamforming.The evaluated model estimates speech and noise TF masks with a sliding-window bidirectional architecture for streaming processing.
- Continuous-input evaluation: Keeping 0.4 s of future acoustic context improved results when inherent latency was reduced to 0.8 s.The comparison examined 1.6-0.8-0.0 and 0.8-0.4-0.4 chunking configurations.
- Continuous-input evaluation: Reducing the inter-utterance gap increased baseline WER by 33.9% relative, while CSS reduced the increase to 22.7%.CSS mitigated this turn-taking degradation by routing adjacent utterances to different output channels.
- Microphone configuration: Microphone count strongly affected continuous-input performance, while three microphones yielded no meaningful non-overlap improvement and single-channel processing could degrade WER.The experiments compared single-channel, linear-array, triangular-array, and six-channel microphone inputs.
5. CONCLUSION
The paper introduces LibriCSS and evaluation protocols for continuous speech separation, then uses a PIT-based speaker-independent method to investigate CSS. Results identify remaining degradation from overlap and limitations in low-microphone non-overlap settings.
- Conclusion: LibriCSS contains multi-channel meeting-room recordings of concatenated and replayed LibriSpeech utterances for continuous speech separation evaluation.The dataset is designed to bridge research evaluation and real-world conversational processing while remaining broadly accessible.
- Conclusion: A PIT-based speaker-independent CSS method was used to investigate multiple aspects of continuous speech separation on LibriCSS.The paper presents the dataset and protocols as resources for evaluating continuous-input separation algorithms.
- Conclusion: Seven-channel MVDR still showed nontrivial performance degradation caused by speech overlap.Single- and three-microphone separation sometimes failed to improve or degraded WER when only one speaker was active.