Source-linked AI summary
Libri-Light: A Benchmark for ASR with Limited or No Supervision
Jacob Kahn, Morgane Rivière, Weiyi Zheng, Evgeny Kharitonov, Qiantong Xu, Pierre-Emmanuel Mazaré, Julien Karadayi, Vitaliy Liptchinsky, Ronan Collobert, Christian Fuegen, Tatiana Likhomanenko, Gabriel Synnaeve, Armand Joulin, Abdelrahman Mohamed, Emmanuel Dupoux
TL;DR
ASR progress is constrained by the cost of annotating ever larger datasets and by limited coverage of low-resource languages and speech varieties. Libri-light introduces over 60K hours of LibriVox speech with benchmarks and baselines for unsupervised, semi-supervised, and distant supervision. Larger unsupervised pretraining improves representations and recognition with limited labels, while pseudo-labels help when initial models are strong.
Problem
Annotation costs and limited labelled resources make it difficult to scale ASR datasets and support low-resource languages and speech varieties.
Method
Libri-light builds a segmented LibriVox corpus with metadata and evaluates CPC-based systems across ABX, PER/CER, and WER settings using limited labels and unaligned text.
Results
Unsupervised pretraining improves ABX, PER, and WER outcomes as training speech increases, while pseudo-labels help only when generating and fine-tuned models are initially good.
Takeaways & Limitations
Libri-light supplies an open benchmark for comparing ASR systems trained with limited or no supervision across three evaluation settings.
Takeaways & Limitations
The baselines are proof-of-concept systems with a significant performance gap from fully supervised systems.
Abstract
from arXiv · showhide
We introduce a new collection of spoken English audio suitable for training speech recognition systems under limited or no supervision. It is derived from open-source audio books from the LibriVox project. It contains over 60K hours of audio, which is, to our knowledge, the largest freely-available corpus of speech. The audio has been segmented using voice activity detection and is tagged with SNR, speaker ID and genre descriptions. Additionally, we provide baseline systems and evaluation metrics working under three settings: (1) the zero resource/unsupervised setting (ABX), (2) the semi-supervised setting (PER, CER) and (3) the distant supervision setting (WER). Settings (2) and (3) use limited textual resources (10 minutes to 10 hours) aligned with the speech. Setting (3) uses large amounts of unaligned text. They are evaluated on the standard LibriSpeech dev and test sets for comparison with the supervised state-of-the-art.
1. INTRODUCTION
Large-scale supervised ASR faces annotation costs and limited language coverage, motivating weakly supervised approaches. Libri-light addresses the lack of a common benchmark by providing open speech, metrics, and baselines across three supervision settings.
- Annotation costs limit scaling supervised ASR to larger datasets and many languages, dialects, accents, and registers.
- Existing work spans semi-supervised learning, distant supervision, pretraining, and zero-resource discovery from raw speech.
- Libri-light provides 60K hours of unlabelled speech and common ABX, PER/CER, and WER metrics for three supervision settings.
- The benchmark uses 10-minute, 1-hour, and 10-hour limited-resource sets, with large in-domain and out-of-domain text for distant supervision.
- Libri-light uses LibriSpeech dev and test sets and provides open-source datasets, metrics, and baseline systems for comparison.
3. DATASET AND METRICS
Libri-light combines large unlabelled speech, limited labelled subsets, standard evaluation sets, and unaligned text. It evaluates unsupervised, semi-supervised, and distant-supervision ASR with ABX, PER/CER, and WER.
- The dataset includes unlabelled speech, limited labelled speech, LibriSpeech dev/test sets, and unaligned text.
- LibriVox English audiobooks are converted to 16kHz FLAC, filtered for corrupted or unsuitable speakers, and segmented with voice activity detection.
- 62.2K hours of speech are distributed across matched-genre unlab-600, unlab-6k, and unlab-60k splits, including duplicates.
- Limited supervision uses 10-hour, 1-hour, and six 10-minute LibriSpeech subsets with orthographic and phonetic transcriptions.
- The LibriSpeech language-model corpus supplies 800M tokens and a 200k-word vocabulary for distant supervision.
- ABX measures unsupervised phonetic discriminability, while semi-supervised and distant-supervision systems use PER/CER and WER, respectively.
4. BASELINE SYSTEMS
The baselines use CPC representations for unsupervised and limited-label settings, while distant supervision combines CTC models, language models, and pseudo-label retraining.
- The unsupervised baseline is CPC, whose encoder, sequence model, and predictor learn speech representations from future-frame prediction.
- The WER evaluation table compares supervised systems, CPC with limited-data fine-tuning, and pseudo-label-trained TDS systems.
- Semi-supervised CPC fine-tunes a linear CTC classifier on limited phone labels, with an end-to-end from-scratch control.
- Distant supervision fine-tunes CPC with orthographic labels and decodes using wav2letter++ with a 4-gram KenLM language model.
- A separate TDS pipeline trains small phoneme or grapheme models, generates pseudo-labels for 60k hours, and retrains larger TDS systems.
5. RESULTS
Across the three settings, larger unsupervised pretraining improves representations and downstream recognition with limited labels. Pseudo-labels help when both the generating and fine-tuned models are initially strong.
- CPC achieves good ABX scores relative to MFCCs and reaches the range of the 2017 Zero Resource Speech Challenge English state of the art.
- Unsupervised pretraining improves PER across several amounts of semi-supervised fine-tuning.
- Increasing unsupervised pretraining improves distant-supervision results, although they remain far from supervised state of the art.
- Pseudo-labels benefit distant and semi-supervised systems only when the generating and fine-tuned models are initially good.
6. CONCLUSION
The paper introduces a large dataset for benchmarking ASR with limited or no supervision. Unsupervised training improves representations and boosts limited-label phoneme and word recognition, while the baselines remain below fully supervised systems.
- The dataset benchmarks ASR systems trained with limited or no supervision.
- Unsupervised training on increasingly larger datasets yields better features.
- Unsupervised training significantly boosts systems using 10 minutes to 10 hours of labels for semi-supervised phoneme recognition and distant-supervision word recognition.
- The provided baselines are proof-of-concept systems with substantial remaining distance from fully supervised performance.
- Potential improvements include larger models, speaker-adversarial losses, full-system fine-tuning, pseudo-label retraining, active learning, and language modeling on unlabelled audio.
S1. SUPPLEMENTARY DATASETS AND METRICS
The dataset construction pipeline converts downloaded audio into segmented, annotated files. It uses a trained acoustic model for voice activity detection and records posterior speech probabilities as metadata.
- The construction pipeline downloads data, excludes bad files, converts audio to FLAC, extracts VAD, SNR, and perplexity, builds JSON files, and creates three cuts.
- Voice activity detection uses a TDS acoustic model trained with CTC loss on orthographic LibriSpeech transcriptions.
- The VAD model performs greedy frame-by-frame decoding by mapping letters to SPEECH and silence to NONSPEECH.
- Each file’s JSON metadata includes the posterior probability of SPEECH produced by the VAD model.
- The VAD model contains 100 million parameters and uses clusters of 2, 3, 4, and 5 TDS blocks separated by 2D convolutions.
S1.1.2. SNR
The dataset annotates audio with SNR, perplexity, book and speaker metadata, genre tags, and voice-activity intervals. SNR is derived from frame-level VAD posteriors using explicit speech and noise thresholds.
- SNR is computed from 80 ms frame-level VAD posteriors using thresholds of < 0.8 for speech and > 0.995 for noise.
- When a speech frame is detected, two subsequent frames are included to compensate for spiky VAD predictions.
- Perplexity is computed as the mean log posterior probability for each file using beam search with a LibriSpeech-trained 4gram language model.
- Each file receives SNR, perplexity, macro-genre, book, and speaker metadata, plus voice-activity onset and offset lists.
- Macro-genres are folded into seven categories: Literature, Science, Craft & Essay, Ancient, Religion, Poetry, Theater, and Undefined.
- The three dataset cuts are designed to preserve the same macro-genre distribution.
S1.2. The ABX metric
The ABX metric evaluates whether representations make same-category phonemes more similar than different-category phonemes. It averages minimal-pair errors across categories.
- The minimal-pair ABX task compares whether features for the same phoneme are more similar than features for different phonemes.
- θ(x, y) is the probability that a sample from category x is closer to another x element than to an element from category y.
- ABX error is computed by averaging 1 − θ over all categories.
S2. SUPPLEMENTARY BASELINE MODELS
The unsupervised baseline uses contrastive predictive coding to learn speech representations by predicting future feature representations and discriminating them from in-batch negatives.
- CPC predicts k future representations from an input feature sequence.The model distinguishes each true future feature from randomly sampled negative examples in the batch.
- The waveform is transformed into a feature sequence by a convolutional network gc.
- A recurrent network gar converts each feature xt into the current phoneme representation zt.
- A prediction network gp predicts future features from recurrent representations up to time t0.
- The recurrent-layer output zt is used as the feature when the representation is transferred to another task.
S2.1.2. Architecture Details
The feature model uses a five-layer convolutional encoder followed by a one-layer LSTM and a one-layer transformer predictor, processing 16 kHz audio at 10 ms feature resolution.
- The convolutional encoder gc has five layers with strides [2], filter sizes [8], and 256 ReLU hidden units.
- A downsampling factor of 160 makes each feature represent 10 ms of 16 kHz audio.
- The recurrent model gar is a one-layer LSTM with a 256-dimensional hidden state, while predictor gp is a one-layer transformer.
- Training uses 1280 ms input sequences in batches of 32 across 128 GPUs, with each batch sampled from one speaker.
S2.2. TDS model and training
The supplementary baseline uses compact TDS acoustic models and pseudo-labeling configurations for limited-label training, with decoding controlled by a shared LibriSpeech language model.
- The limited-supervision baseline uses a smaller 20-million-parameter TDS model.Its architecture includes an initial stride-2 convolution and three TDS block groups.
- The models use 1500 training epochs for supervised training and 150 epochs for semi-supervised training, with learning-rate reductions during training.
- A beam size of 1000 is used for all decoding experiments with the official LibriSpeech 4-gram language model.
- Table S1 reports PER/CER for acoustic models trained on pseudo-labels generated over the 60K dataset and then used to retrain larger TDS models.