Source-linked AI summary
Libriheavy: a 50,000 hours ASR corpus with punctuation casing and context
Wei Kang, Xiaoyu Yang, Zengwei Yao, Fangjun Kuang, Yifan Yang, Liyong Guo, Long Lin, Daniel Povey
TL;DR
Open ASR corpora can lack full-format text and utterance context, despite those being useful for punctuation prediction and contextualized recognition. Libriheavy labels Librilight audio with formatted transcripts and context using a reusable alignment pipeline; the authors report that experiments indicate corpus quality and pipeline effectiveness.
Problem
Many large open ASR corpora provide only normalized transcripts, and most existing corpora lack the utterance-level text context usually needed to train contextualized recognition systems.
Source: §1 · p. 1Method
The paper aligns Librilight audio to original book text and releases a 50,000-hour corpus with formatted transcripts, preceding text context, and a general alignment pipeline.
Source: §1 · p. 1, §2.2 · p. 1, §3 · p. 2Results
The authors report that experiments on Libriheavy show the corpus is high quality and the creation pipeline is effective.
Source: §5 · p. 4Takeaways & Limitations
The corpus offers punctuation, casing, and text context for varied ASR tasks, and the released alignment pipeline is intended to make corpus construction easier.
Source: §1 · p. 1The segmentation pipeline normalizes component scores so that no score dominates the final score.
Source: §3.4 · p. 3
Abstract
from arXiv · showhide
In this paper, we introduce Libriheavy, a large-scale ASR corpus consisting of 50,000 hours of read English speech derived from LibriVox. To the best of our knowledge, Libriheavy is the largest freely-available corpus of speech with supervisions. Different from other open-sourced datasets that only provide normalized transcriptions, Libriheavy contains richer information such as punctuation, casing and text context, which brings more flexibility for system building. Specifically, we propose a general and efficient pipeline to locate, align and segment the audios in previously published Librilight to its corresponding texts. The same as Librilight, Libriheavy also has three training subsets small, medium, large of the sizes 500h, 5000h, 50000h respectively. We also extract the dev and test evaluation sets from the aligned audios and guarantee there is no overlapping speakers and books in training sets. Baseline systems are built on the popular CTC-Attention and transducer models. Additionally, we open-source our dataset creatation pipeline which can also be used to other audio alignment tasks.
1. INTRODUCTION
The paper motivates Libriheavy by the need for larger open ASR corpora and training text that retains punctuation, casing, and utterance context. It introduces a 50,000-hour corpus and a reusable audio-alignment pipeline.
1. INTRODUCTION
- Neural-network speech-recognition models are described as more data-hungry than traditional systems.The introduction situates this need alongside the move from hybrid HMMs to end-to-end architectures.Source: §1 · p. 1
- Existing open datasets are often small or medium-sized, while large-scale corpora commonly provide only normalized transcripts.The paper notes that normalized text precludes training systems that need full-format text, such as punctuation prediction.Source: §1 · p. 1
- Contextualized speech recognition uses cross-utterance context, but training it usually requires utterance-level context unavailable in most existing ASR corpora.The paper therefore identifies a dataset with textual context as desirable.Source: §1 · p. 1
- Libriheavy provides 50,000 hours of labeled audio with punctuation, casing, and preceding text, alongside an open-source general audio-alignment pipeline.The authors also report experiments intended to demonstrate corpus quality and pipeline robustness.Source: §1 · p. 1
2. LIBRIHEAVY CORPUS
Libriheavy labels Librilight audio and segments it for ASR, while retaining richer text and context metadata. Its training and evaluation partitions are designed to support corpus use and speaker- and book-disjoint evaluation.
2. LIBRIHEAVY CORPUS
- The corpus section covers audio files, metadata, data partitions, text styles, and other aspects of Libriheavy.Instructions and scripts are available in the Libriheavy GitHub repository.Source: §2 · p. 1
2.1. Librilight
- Librilight is a free collection of more than 60,000 hours of unlabeled spoken English audio derived from LibriVox audiobooks.It is intended for speech-recognition training under limited or no supervision.Source: §2.1 · p. 1
2.2. Libriheavy
- Libriheavy aligns Librilight audio to its original book text, segments it into 2–30-second pieces, and provides small, medium, and large training subsets plus evaluation sets.The training subsets preserve Librilight splits; the evaluation subsets are dev, test-clean, and test-other.Source: §2.2 · pp. 1–2
2.2.1. Metadata
- Each Lhotse cut is a self-contained JSON-lines segment with transcript and audio source, plus text context and byte offsets for slicing context from the original book.The default preceding-text context is 1,000 bytes.Source: §2.2.1 · p. 2
2.2.2. Evaluation Sets
- Evaluation sets are constructed without speaker or book overlap with training; 20 hours are sampled from each clean and other pool for test and dev allocation.Half of each pool’s 20 hours forms its test set, and the other half is added to dev.Source: §2.2.2 · p. 2
3. AUDIO ALIGNMENT
The alignment pipeline locates audiobook text corresponding to long audio, refines the match, and segments the audio into shorter training examples. It combines transcript-based search, monotonic match chains, constrained Levenshtein alignment, and punctuation- and score-based segment selection.
3. AUDIO ALIGNMENT
- The general pipeline aligns audio to corresponding text, segments it, and excludes audio that does not exactly correspond to the aligned text.The authors state that the pipeline can also be applied to other data-generation tasks and make its implementation and scripts public.Source: §3 · p. 2
3.1. Downloading text
- Scripts obtain each audiobook’s original text from its source URL and clean redundant spaces and lines before alignment.The audio-to-text source URL is obtained from Librilight metadata.Source: §3.1 · p. 2
3.2. First alignment stage
- The first alignment stage treats an automatic audio transcript as a query against the original book and locates its text segment using close matches and longest increasing pairs.The method avoids Librilight’s VAD segmentation because it requires relatively long text to ensure accuracy.Source: §3.2 · p. 2
3.2.1. Transcribe audios
- Long audio is split into overlapping 30-second pieces, transcribed with a Librispeech-trained ASR model, and recombined using recognized-word timestamps.Adjacent pieces overlap by 2 seconds on each side.Source: §3.2.1 · p. 2
3.2.2. Close matches
- For each query position, close matches are the two target positions neighboring it in suffix-array order within the target portion.Texts are uppercased and punctuation is removed for matching while indices into the original text are retained.Source: §3.2.2 · p. 2, Figure 1 · p. 3
3.2.3. Longest increasing pairs
- The approximate alignment is the longest monotonic chain of query–target index pairs, with both indices nondecreasing.The query and its corresponding target segment are expected to align monotonically.Source: §3.2.3 · p. 3
3.3. Second alignment stage
- The second stage uses the longest chain to locate a target region, then applies Levenshtein alignment in blocks defined by chain traceback.Blockwise alignment limits the quadratic-time alignment computation; concatenating the blocks yields the whole-query alignment.Source: §3.3 · p. 3
3.4. Audio segmentation
- Candidate segments pair sentence-punctuation BOS/EOS positions scored by silence and alignment errors; selection favors 2–30-second durations, especially 5–20 seconds, and permits limited overlap.Scores are normalized so none dominates the final score; the toolkit also supports splitting at a silence threshold.Source: §3.4 · p. 3
4. EXPERIMENTS
The experiments benchmark CTC-Attention and transducer systems, then compare normalized training text with text retaining casing and punctuation. The reported WER gap between text styles is large on small training sets but becomes negligible as training data grows.
4.1. CTC-Attention baseline system
- The CTC-Attention baseline uses Wenet with a 12-layer Conformer encoder, 6-layer Transformer decoder, and 500-class BPE word pieces.Its combined objective weights CTC loss at 0.3 and attention loss at 0.1.Source: §4.1 · p. 3
- Table 2 reports CTC-Attention WER on LibriSpeech and Libriheavy test sets, with text normalization applied to LibriSpeech hypotheses.The CTC branch generates N-best hypotheses for rescoring by the attention branch.Source: §4.1 · pp. 3–4, Table 2 · p. 4
4.2. Transducer baseline system
- The transducer baseline uses Zipformer in icefall, with an encoder and stateless decoder plus an added 1-D convolution layer.Models are trained for 90, 60, and 15 epochs on the small, medium, and large subsets, respectively.Source: §4.2 · p. 4
4.3. Training with punctuation and casing
- As training data grows, the WER gap between normalized and cased, punctuated training text becomes negligible; it is large on small sets.Table 5 compares the systems after uppercasing transcripts and decoding results and removing punctuation for WER calculation.Source: Table 5 · p. 4, §4.3 · p. 4
- The cased, punctuated-text transducer uses 756-class BPE pieces, adding 256 byte positions to handle rare characters.This experiment compares normalized-text and unnormalized-text training while scoring WER after normalizing both transcripts and decoding results.Source: §4.3 · p. 4
5. CONCLUSION
The conclusion presents Libriheavy as a 50,000-hour corpus with punctuation, casing, and text context, alongside an open-source audio-alignment toolkit. Experiments on the corpus support its quality and the effectiveness of the creation pipeline.
5. CONCLUSION
- Libriheavy provides 50,000 hours of speech with punctuation, casing, and text context, alongside an open-source alignment toolkit and experiments supporting corpus quality and pipeline effectiveness.The authors state that the corpus can be used in various ASR tasks and that the toolkit makes speech-corpus construction easier.Source: §5 · p. 4