Source-linked AI summary
TED-LIUM 3: twice as much data and corpus repartition for experiments on speaker adaptation
François Hernandez, Vincent Nguyen, Sahar Ghannay, Natalia Tomashenko, Yannick Estève
TL;DR
TED-LIUM 3 addresses the need for more English speech-recognition training data and better corpus organization for speaker-adaptation research. It expands and realigns TED data, adds a speaker-adaptation distribution, and shows that the larger corpus benefits end-to-end systems more than HMM-based systems, although HMM remains stronger at the largest training size.
Problem
Earlier TED-LIUM releases provided less training data and did not include a distribution specifically designed for speaker-adaptation experiments.
Method
The paper creates TED-LIUM 3 by realigning existing audio and transcripts, adding TED talks, and defining legacy and speaker-adaptation corpus distributions.
Results
452 hours of transcribed speech make increased training data more useful for end-to-end ASR than HMM-based ASR, while HMM-based ASR still achieves lower WER at that training size.
Takeaways & Limitations
TED-LIUM 3 provides a larger open corpus for ASR research and a balanced evaluation setup for speaker-adaptation experiments.
Abstract
from arXiv · showhide
In this paper, we present TED-LIUM release 3 corpus dedicated to speech recognition in English, that multiplies by more than two the available data to train acoustic models in comparison with TED-LIUM 2. We present the recent development on Automatic Speech Recognition (ASR) systems in comparison with the two previous releases of the TED-LIUM Corpus from 2012 and 2014. We demonstrate that, passing from 207 to 452 hours of transcribed speech training data is really more useful for end-to-end ASR systems than for HMM-based state-of-the-art ones, even if the HMM-based ASR system still outperforms end-to-end ASR system when the size of audio training data is 452 hours, with respectively a Word Error Rate (WER) of 6.6% and 13.7%. Last, we propose two repartitions of the TED-LIUM release 3 corpus: the legacy one that is the same as the one existing in release 2, and a new one, calibrated and designed to make experiments on speaker adaptation. Like the two first releases, TED-LIUM 3 corpus will be freely available for the research community.
1 Introduction
TED-LIUM 3 expands the corpus through improved alignment and additional TED talks, reaching 452 hours of aligned audio. It also introduces a speaker-adaptation-oriented distribution alongside the legacy release structure.
- The corpus remains connected to earlier TED-LIUM releases that provided 118 and 207 hours of audio for ASR research.
- TED-LIUM 3 is made available through the LIUM website.
- 452 hours of aligned audio result from a new realignment engine combined with additional TED talks.
- TED-LIUM 3 proposes a new data distribution designed for speaker adaptation experiments, alongside the legacy distribution.
2 TED-LIUM 3 Corpus Description
TED-LIUM 3 is constructed by realigning TED audio and transcripts, then organizing the resulting corpus into legacy and speaker-adaptation distributions. The new organization preserves prior evaluation compatibility while targeting more balanced speaker-adaptation experiments.
- Data, Alignment and Filtering: The Kaldi toolkit automatically aligns each transcript with its corresponding audio file after decoding with a biased language model.
- Data, Alignment and Filtering: Equivalent models showed no clear improvement or degradation when trained on new rather than original TED-LIUM 2 alignments.
- Corpus Distribution: Legacy Version: The legacy version keeps the same development and test sets as the first TED-LIUM releases.
- Corpus Distribution: Legacy Version: 82.6% of the 540-hour audio duration is aligned speech, including some noises and silences.
- Corpus Distribution: Speaker Adaptation Version: Speaker-adaptation datasets balance speakers, gender, and duration more representatively than the original development and test sets.
- Corpus Distribution: Speaker Adaptation Version: The proposed development and test sets exclude speakers appearing in the training data through other talks.
- Corpus Distribution: Speaker Adaptation Version: Table 4 reports statistics for the speaker-adaptation training, development, and test datasets using pure-speech duration.
3 Experiments with State-of-the-art HMM-based ASR System
The HMM-based experiments compare TED-LIUM releases using Kaldi chain models, augmentation, language-model lattice rescoring, and TDNN variants. Increasing data supports larger models and improved WER, while factorized TDNNs further improve results.
- The experiments use Kaldi chain models with TDNN and TDNN-F architectures, plus speed and volume augmentation.
- The language-model experiments rescore lattices with either a pruned 4-gram model or an RNNLM trained on approximately 255 million words.
- More than doubling the training data enables larger models within the same chain-TDNN architecture.
- 6.2% WER was obtained on the development set and 6.7% on the test set after RNNLM lattice-rescoring.
- Factorized TDNN experiments produced significant WER improvements on both TED-LIUM release 2 and release 3 corpora.
4 Experiments with Fully Neural End-to-end ASR System
The fully neural system predicts character sequences with a Deep Speech 2-like CTC architecture and is evaluated across three training-data sizes and three decoding configurations. Increasing training data substantially improves transcription quality, with augmentation and beam decoding producing the strongest reported results.
- 4.1 Model Architecture: The end-to-end system uses convolutional and recurrent layers, CTC training, and character-sequence output rather than a word vocabulary.
- 4.2 Experimental Results: The three training sets contain 207, 290, and 452 hours of speech from TED-LIUM 2 and TED-LIUM 3.
- 4.2 Experimental Results: The evaluated configurations are Greedy, Greedy+augmentation, and Beam+augmentation, with the last applying a language model through beam-search decoding.
- 4.2 Experimental Results: 13.7% WER and 6.1% CER are achieved by the Beam+augmentation configuration.
- 4.2 Experimental Results: 28.1% WER with original TED-LIUM 2 data falls to 20.3% with TED-LIUM 3 in Greedy mode.
5 Experiments with the Speaker Adaptation Distribution
The speaker-adaptation experiments compare speaker-independent and speaker-adaptive acoustic models using i-vectors and fMLLR on the adaptation distribution. The associated results are reported for the speaker-adaptation task in Table 7.
- The experiments train three pairs of acoustic models, each containing a speaker-independent model and a corresponding speaker-adaptive-trained model.
- The study evaluates i-vectors for TDNN-LSTM models and fMLLR for both GMM-HMM and TDNN-LSTM models.
- Table 7 reports speaker-adaptation results using MFCC-39 and hires MFCC-40 feature configurations.
6 Discussion and Conclusion
TED-LIUM 3 doubles aligned training audio, but the larger dataset benefits end-to-end ASR more than HMM-based systems. HMM-based ASR remains stronger at the reported 452-hour scale.
- 452 hours of aligned audio doubles the quantity available for acoustic-model training.
- 6.7% WER on legacy test data with release 3 improved only slightly over 6.8% with release 2 for the HMM-based system.
- 13.7% WER was achieved by end-to-end ASR with TED-LIUM 3 data and a classical 3-gram language model, versus 20.3% with TED-LIUM 2 data.
- Without external knowledge, end-to-end ASR reached 17.4% WER using only TED-LIUM 3 aligned audio.
- The release includes funding from the French ANR Agency and Google Digital News Innovation Fund.