PaperlayerView exact revision on arXiv

Source-linked AI summary

Libri-Light: A Benchmark for ASR with Limited or No Supervision

Jacob Kahn, Morgane Rivière, Weiyi Zheng, Evgeny Kharitonov, Qiantong Xu, Pierre-Emmanuel Mazaré, Julien Karadayi, Vitaliy Liptchinsky, Ronan Collobert, Christian Fuegen, Tatiana Likhomanenko, Gabriel Synnaeve, Armand Joulin, Abdelrahman Mohamed, Emmanuel Dupoux

arXiv:1912.07875v1cs.CLcs.SDeess.AS

TL;DR

ASR progress is constrained by the cost of annotating ever larger datasets and by limited coverage of low-resource languages and speech varieties. Libri-light introduces over 60K hours of LibriVox speech with benchmarks and baselines for unsupervised, semi-supervised, and distant supervision. Larger unsupervised pretraining improves representations and recognition with limited labels, while pseudo-labels help when initial models are strong.

  • Problem

    Annotation costs and limited labelled resources make it difficult to scale ASR datasets and support low-resource languages and speech varieties.

    Source: §1 · p. 1

  • Method

    Libri-light builds a segmented LibriVox corpus with metadata and evaluates CPC-based systems across ABX, PER/CER, and WER settings using limited labels and unaligned text.

    Source: §1 · p. 1, §3.1 · pp. 2–3, S1.2. The ABX metric · p. 6

  • Results

    Unsupervised pretraining improves ABX, PER, and WER outcomes as training speech increases, while pseudo-labels help only when generating and fine-tuned models are initially good.

    Source: §5 · p. 4

  • Takeaways & Limitations

    Libri-light supplies an open benchmark for comparing ASR systems trained with limited or no supervision across three evaluation settings.

    Source: §1 · p. 1, §6 · p. 4

    The baselines are proof-of-concept systems with a significant performance gap from fully supervised systems.

    Source: §6 · p. 4

Abstract

from arXiv · show

We introduce a new collection of spoken English audio suitable for training speech recognition systems under limited or no supervision. It is derived from open-source audio books from the LibriVox project. It contains over 60K hours of audio, which is, to our knowledge, the largest freely-available corpus of speech. The audio has been segmented using voice activity detection and is tagged with SNR, speaker ID and genre descriptions. Additionally, we provide baseline systems and evaluation metrics working under three settings: (1) the zero resource/unsupervised setting (ABX), (2) the semi-supervised setting (PER, CER) and (3) the distant supervision setting (WER). Settings (2) and (3) use limited textual resources (10 minutes to 10 hours) aligned with the speech. Setting (3) uses large amounts of unaligned text. They are evaluated on the standard LibriSpeech dev and test sets for comparison with the supervised state-of-the-art.

1. INTRODUCTION

Large-scale supervised ASR faces annotation costs and limited language coverage, motivating weakly supervised approaches. Libri-light addresses the lack of a common benchmark by providing open speech, metrics, and baselines across three supervision settings.

  • Annotation costs limit scaling supervised ASR to larger datasets and many languages, dialects, accents, and registers.Source: §1 · p. 1
  • Existing work spans semi-supervised learning, distant supervision, pretraining, and zero-resource discovery from raw speech.Source: §1 · p. 1
  • Libri-light provides 60K hours of unlabelled speech and common ABX, PER/CER, and WER metrics for three supervision settings.Source: §1 · p. 1
  • The benchmark uses 10-minute, 1-hour, and 10-hour limited-resource sets, with large in-domain and out-of-domain text for distant supervision.Source: §1 · p. 1, §2 · p. 2
  • Libri-light uses LibriSpeech dev and test sets and provides open-source datasets, metrics, and baseline systems for comparison.Source: §1 · p. 1

3. DATASET AND METRICS

Libri-light combines large unlabelled speech, limited labelled subsets, standard evaluation sets, and unaligned text. It evaluates unsupervised, semi-supervised, and distant-supervision ASR with ABX, PER/CER, and WER.

  • The dataset includes unlabelled speech, limited labelled speech, LibriSpeech dev/test sets, and unaligned text.Source: §3.1 · p. 2
  • LibriVox English audiobooks are converted to 16kHz FLAC, filtered for corrupted or unsuitable speakers, and segmented with voice activity detection.Source: §3.1 · p. 2
  • 62.2K hours of speech are distributed across matched-genre unlab-600, unlab-6k, and unlab-60k splits, including duplicates.Source: §3.1 · p. 2
  • Limited supervision uses 10-hour, 1-hour, and six 10-minute LibriSpeech subsets with orthographic and phonetic transcriptions.Source: §3.1 · p. 2
  • The LibriSpeech language-model corpus supplies 800M tokens and a 200k-word vocabulary for distant supervision.Source: §3.1 · p. 3
  • ABX measures unsupervised phonetic discriminability, while semi-supervised and distant-supervision systems use PER/CER and WER, respectively.Source: §3.2 · p. 3

4. BASELINE SYSTEMS

The baselines use CPC representations for unsupervised and limited-label settings, while distant supervision combines CTC models, language models, and pseudo-label retraining.

  • The unsupervised baseline is CPC, whose encoder, sequence model, and predictor learn speech representations from future-frame prediction.Source: §4 · p. 3
  • The WER evaluation table compares supervised systems, CPC with limited-data fine-tuning, and pseudo-label-trained TDS systems.Source: Table 4 · p. 4
  • Semi-supervised CPC fine-tunes a linear CTC classifier on limited phone labels, with an end-to-end from-scratch control.Source: §4 · p. 4
  • Distant supervision fine-tunes CPC with orthographic labels and decodes using wav2letter++ with a 4-gram KenLM language model.Source: §4 · p. 4
  • A separate TDS pipeline trains small phoneme or grapheme models, generates pseudo-labels for 60k hours, and retrains larger TDS systems.Source: §4 · p. 4

5. RESULTS

Across the three settings, larger unsupervised pretraining improves representations and downstream recognition with limited labels. Pseudo-labels help when both the generating and fine-tuned models are initially strong.

  • CPC achieves good ABX scores relative to MFCCs and reaches the range of the 2017 Zero Resource Speech Challenge English state of the art.Source: §5 · p. 4
  • Unsupervised pretraining improves PER across several amounts of semi-supervised fine-tuning.Source: §5 · p. 4
  • Increasing unsupervised pretraining improves distant-supervision results, although they remain far from supervised state of the art.Source: §5 · p. 4
  • Pseudo-labels benefit distant and semi-supervised systems only when the generating and fine-tuned models are initially good.Source: §5 · p. 4

6. CONCLUSION

The paper introduces a large dataset for benchmarking ASR with limited or no supervision. Unsupervised training improves representations and boosts limited-label phoneme and word recognition, while the baselines remain below fully supervised systems.

  • The dataset benchmarks ASR systems trained with limited or no supervision.Source: §6 · p. 4
  • Unsupervised training on increasingly larger datasets yields better features.Source: §6 · p. 4
  • Unsupervised training significantly boosts systems using 10 minutes to 10 hours of labels for semi-supervised phoneme recognition and distant-supervision word recognition.Source: §6 · p. 4
  • The provided baselines are proof-of-concept systems with substantial remaining distance from fully supervised performance.Source: §6 · p. 4
  • Potential improvements include larger models, speaker-adversarial losses, full-system fine-tuning, pseudo-label retraining, active learning, and language modeling on unlabelled audio.Source: §6 · p. 4

S1. SUPPLEMENTARY DATASETS AND METRICS

The dataset construction pipeline converts downloaded audio into segmented, annotated files. It uses a trained acoustic model for voice activity detection and records posterior speech probabilities as metadata.

  • The construction pipeline downloads data, excludes bad files, converts audio to FLAC, extracts VAD, SNR, and perplexity, builds JSON files, and creates three cuts.Source: S1.1. Datasets and meta-data · p. 6
  • Voice activity detection uses a TDS acoustic model trained with CTC loss on orthographic LibriSpeech transcriptions.Source: S1.1.1. VAD · p. 6
  • The VAD model performs greedy frame-by-frame decoding by mapping letters to SPEECH and silence to NONSPEECH.Source: S1.1.1. VAD · p. 6
  • Each file’s JSON metadata includes the posterior probability of SPEECH produced by the VAD model.Source: S1.1.1. VAD · p. 6
  • The VAD model contains 100 million parameters and uses clusters of 2, 3, 4, and 5 TDS blocks separated by 2D convolutions.Source: S1.1.1. VAD · p. 6

S1.1.2. SNR

The dataset annotates audio with SNR, perplexity, book and speaker metadata, genre tags, and voice-activity intervals. SNR is derived from frame-level VAD posteriors using explicit speech and noise thresholds.

  • SNR is computed from 80 ms frame-level VAD posteriors using thresholds of < 0.8 for speech and > 0.995 for noise.Source: S1.1.2. SNR · p. 6
  • When a speech frame is detected, two subsequent frames are included to compensate for spiky VAD predictions.Source: S1.1.2. SNR · p. 6
  • Perplexity is computed as the mean log posterior probability for each file using beam search with a LibriSpeech-trained 4gram language model.Source: S1.1.3. Perplexity · p. 6
  • Each file receives SNR, perplexity, macro-genre, book, and speaker metadata, plus voice-activity onset and offset lists.Source: S1.1.4. JSON files and splits · p. 6
  • Macro-genres are folded into seven categories: Literature, Science, Craft & Essay, Ancient, Religion, Poetry, Theater, and Undefined.Source: S1.1.4. JSON files and splits · p. 6
  • The three dataset cuts are designed to preserve the same macro-genre distribution.Source: S1.1.4. JSON files and splits · p. 6

S1.2. The ABX metric

The ABX metric evaluates whether representations make same-category phonemes more similar than different-category phonemes. It averages minimal-pair errors across categories.

  • The minimal-pair ABX task compares whether features for the same phoneme are more similar than features for different phonemes.Source: S1.2. The ABX metric · p. 6
  • θ(x, y) is the probability that a sample from category x is closer to another x element than to an element from category y.Source: S1.2. The ABX metric · p. 6
  • ABX error is computed by averaging 1 − θ over all categories.Source: S1.2. The ABX metric · p. 6

S2. SUPPLEMENTARY BASELINE MODELS

The unsupervised baseline uses contrastive predictive coding to learn speech representations by predicting future feature representations and discriminating them from in-batch negatives.

S2.1.2. Architecture Details

The feature model uses a five-layer convolutional encoder followed by a one-layer LSTM and a one-layer transformer predictor, processing 16 kHz audio at 10 ms feature resolution.

S2.2. TDS model and training

The supplementary baseline uses compact TDS acoustic models and pseudo-labeling configurations for limited-label training, with decoding controlled by a shared LibriSpeech language model.

Related papers

Loading 1912.07875v1…