Source-linked AI summary
LibriMix: An Open-Source Dataset for Generalizable Speech Separation
Joris Cosentino, Manuel Pariente, Samuele Cornell, Antoine Deleforge, Emmanuel Vincent
TL;DR
Existing speech-separation datasets can produce strong benchmark results while generalizing poorly across datasets and underrepresenting sparsely overlapping speech. This paper introduces LibriMix, an open-source family built from LibriSpeech and WHAM! noise, with VCTK-based cross-dataset evaluation and sparse-overlap tests. Conv-TasNet achieves competitive LibriMix performance, and LibriMix-trained models generalize better than WHAM!-trained models in clean and noisy conditions.
Problem
Models trained on wsj0-2mix show important performance drops on comparable datasets, while fully overlapping mixtures do not adequately represent typical real-world overlap.
Method
The paper introduces open-source LibriMix datasets with two- or three-speaker LibriSpeech mixtures, optional WHAM! noise, VCTK-based evaluation, and sparse-overlap test sets.
Results
LibriMix-trained models generalize better than WHAM!-trained models in clean and noisy conditions, with WHAM!-trained models showing a 4 dB SI-SDR drop on LibriMix.
Takeaways & Limitations
LibriMix provides a broader open-source benchmark for evaluating speech separation generalization across datasets and overlap conditions.
Takeaways & Limitations
WSJ0 and derived data remain proprietary, whereas only WHAM! noises are released under CC BY-NC 4.0.
Abstract
from arXiv · showhide
In recent years, wsj0-2mix has become the reference dataset for single-channel speech separation. Most deep learning-based speech separation models today are benchmarked on it. However, recent studies have shown important performance drops when models trained on wsj0-2mix are evaluated on other, similar datasets. To address this generalization issue, we created LibriMix, an open-source alternative to wsj0-2mix, and to its noisy extension, WHAM!. Based on LibriSpeech, LibriMix consists of two- or three-speaker mixtures combined with ambient noise samples from WHAM!. Using Conv-TasNet, we achieve competitive performance on all LibriMix versions. In order to fairly evaluate across datasets, we introduce a third test set based on VCTK for speech and WHAM! for noise. Our experiments show that the generalization error is smaller for models trained with LibriMix than with WHAM!, in both clean and noisy conditions. Aiming towards evaluation in more realistic, conversation-like scenarios, we also release a sparsely overlapping version of LibriMix's test set.
1. Introduction
Existing speech-separation benchmarks can yield strong performance yet poor generalization and do not adequately represent sparsely overlapping real-world speech. LibriMix is introduced as an open-source dataset designed to address these gaps.
- Conv-TasNet trained on wsj0-2mix shows important performance drops on comparable datasets, indicating limited generalization across speakers and recording conditions.
- Fully overlapping wsj0-2mix mixtures are unnatural because real-world speech overlap is typically around 20% or less.
- Models trained on fully overlapping mixtures may not generalize well to sparsely overlapping speech mixtures.
- Shared speakers between wsj0-2mix training and validation sets make overfitting difficult to detect for speaker-identity-based models.
- LibriMix provides open-source two- or three-speaker mixtures with or without noise, plus VCTK-based cross-dataset evaluation and sparsely overlapping test sets.
2. Datasets
The paper develops open-source speech-separation datasets from LibriSpeech and WHAM! noise, expanding speaker, vocabulary, mixture, and overlap diversity beyond WSJ0-based benchmarks. It also adds VCTK-based evaluation and sparse-overlap test sets for more realistic scenarios.
- WSJ0, wsj0-2mix and WHAM!: WSJ0-based datasets use read speech from about 119 speakers and support two- and three-speaker mixtures, while WSJ0 and derived data are proprietary.
- WSJ0, wsj0-2mix and WHAM!: WHAM! extends wsj0-2mix with coffee-shop, restaurant, and bar noises, using loudest-speaker-to-noise SNRs from -6 to +3 dB.
- LibriSpeech, LibriMix and sparse LibriMix: LibriMix uses about 470 hours of LibriSpeech from 1,252 speakers and combines it with WHAM! noises in entirely open-source datasets.
- LibriSpeech, LibriMix and sparse LibriMix: Libri2Mix and Libri3Mix provide clean and noisy two- and three-speaker mixtures, with Libri2Mix following WHAM!’s structure and train-100, train-360, dev, and test splits.
- LibriSpeech, LibriMix and sparse LibriMix: LibriMix training uses each utterance once, producing about 280 hours of noisy mixtures, around 1,000 distinct speakers, and 60k unique words.
- LibriSpeech, LibriMix and sparse LibriMix: SparseLibri2Mix and SparseLibri3Mix create conversation-like test mixtures by concatenating MFA-delimited sub-utterances, allowing alternating dominant speakers.
- LibriSpeech, LibriMix and sparse LibriMix: SparseLibri test sets contain 500 mixtures for each of six nominal overlap levels from 0% to 100%, with three-speaker overlap counted separately from two-speaker overlap.
- LibriSpeech, LibriMix and sparse LibriMix: VCTK-2mix is an unmatched open-source test set based on 109 native English speakers, with silent portions removed using energy-based voice activity detection.
3. Results
Conv-TasNet achieves competitive separation performance on LibriMix, while sparse-overlap and cross-dataset evaluations expose remaining generalization and robustness challenges. LibriMix-trained models generalize better than WHAM!-trained models across clean and noisy conditions.
- Results on LibriMix: Conv-TasNet outperforms ideal masks for two-speaker clean separation and matches them in noisy conditions, but remains below oracle performance for three-speaker mixtures.Results are reported using SI-SDR improvement over the input mixture.
- Results on SparseLibriMix: Higher speech overlap produces lower SI-SDRi on both two- and three-speaker SparseLibriMix test sets.At 100% overlap, performance is similar to non-sparse 8 kHz LibriMix, with slightly higher values because mixtures are not truncated.
- Results on SparseLibriMix: Conv-TasNet performs worse than IRM at smaller overlaps, indicating remaining room for improvement in sparsely overlapping speech separation.This comparison is reported for both clean and noisy sparse-overlap conditions.
- Dataset comparisons: 4 dB SI-SDR separates WHAM!-trained from LibriMix-trained models on LibriMix, while the reverse gap on WHAM! is only 0.8 dB.The cross-dataset comparison covers clean and noisy separation and evaluates models on Libri2Mix, WHAM!, and VCTK2Mix test sets.
- Dataset comparisons: WHAM!-trained models perform around 3–4 dB worse than LibriMix train-360 models on clean and noisy VCTK-2mix.The authors associate LibriMix's stronger generalization with differences in speaker count, vocabulary, recording conditions, and training-data volume.
- Dataset comparisons: Differences from earlier VCTK results are attributed to silence handling, SNR computation, and the LibriTTS-derived data used in the earlier study.Silences can substantially increase effective SNR ranges and make one source nearly inaudible in some mixtures.
4. Conclusions
LibriMix is introduced as a dataset family for generalizable single-channel speech separation, with clean and noisy two- and three-speaker variants. Independent cross-dataset evaluation and sparse-overlap test sets show improved generalization over WHAM! while leaving room for further improvement.
- Libri2Mix and Libri3Mix support two- and three-speaker separation in clean and noisy conditions.
- Conv-TasNet achieves competitive results across all LibriMix conditions.
- VCTK-2mix provides an independent test set for reproducible cross-dataset evaluation.
- Models trained on Libri2Mix generalize better to VCTK-2mix than models trained with WHAM!.
- Libri3Mix is the first open-source dataset enabling three-speaker noisy separation.
- SparseLibri2Mix and SparseLibri3Mix provide sparsely overlapping test sets, and initial results indicate room for improvement.