Source-linked AI summary
The People's Speech: A Large-Scale Diverse English Speech Recognition Dataset for Commercial Usage
Daniel Galvez, Greg Diamos, Juan Ciro, Juan Felipe Cerón, Keith Achorn, Anjali Gopi, David Kanter, Maximilian Lam, Mark Mazumder, Vijay Janapa Reddi
TL;DR
High-quality speech recognition requires large, diverse, and appropriately licensed training data. The paper introduces The People’s Speech, a large supervised English dataset collected from licensed web audio with transcripts, and describes its alignment and maintenance pipeline. The dataset is designed for academic and commercial use, while its split construction remains limited by unresolved speaker and duplicate-audio overlap.
Problem
High-quality ASR requires large, diverse speech data, while proper licensing is an important constraint on usable training corpora.
Method
The paper constructs The People’s Speech from appropriately licensed web audio with existing transcripts, uses forced alignment, and releases the collection pipeline under Apache 2.0.
Results
The People’s Speech is a 30,000-hour supervised English dataset spanning diverse scenarios and available for commercial use.
Takeaways & Limitations
The work demonstrates that large-scale, diverse, open, appropriately licensed speech recognition datasets can be curated from web resources.
Takeaways & Limitations
The dataset lacks train, development, and test splits because speaker overlap and duplicated audio cannot be reliably excluded.
Abstract
from arXiv · showhide
The People's Speech is a free-to-download 30,000-hour and growing supervised conversational English speech recognition dataset licensed for academic and commercial usage under CC-BY-SA (with a CC-BY subset). The data is collected via searching the Internet for appropriately licensed audio data with existing transcriptions. We describe our data collection methodology and release our data collection system under the Apache 2.0 license. We show that a model trained on this dataset achieves a 9.98% word error rate on Librispeech's test-clean test set.Finally, we discuss the legal and ethical issues surrounding the creation of a sizable machine learning corpora and plans for continued maintenance of the project under MLCommons's sponsorship.
1 Introduction
The paper addresses the need for large, diverse, properly licensed speech data by introducing The People’s Speech, a 30,000-hour dataset built from Internet audio with transcripts. It argues that openly licensed web sources and automated alignment make commercially usable speech dataset construction feasible.
- Motivation: Large, diverse training data and proper licensing are both important for building high-quality speech recognition systems.The introduction identifies training-data quality and licensing as central constraints on ASR development.
- Contribution: The People’s Speech is a 30,000-hour supervised speech dataset released under CC-BY and CC-BY-SA licenses permitting academic and commercial reuse.The collection focuses primarily on English speech and includes diverse scenarios beyond audiobook recordings.
- Method: The authors collect and force-align existing audio and transcripts from Internet Archive sources, releasing the collection pipeline under Apache 2.0.Forced alignment supports timestamp assignment and transcript cleanup when source timestamps are insufficiently precise.
- Data Sources: Internet audio with transcripts provides abundant, searchable, and diverse material for creating open speech datasets, although the available Creative Commons content is concentrated in English.The paper identifies many non-audiobook sources and notes the scarcity of comparable non-English Creative Commons data.
- Licensing: Commercial dataset construction is presented as feasible because CC-BY and CC-BY-SA works permit commercial use and can be detected programmatically.The paper frames licensing constraints as solvable through software-supported identification of appropriately licensed sources.
2 Related Work
The paper contrasts The People’s Speech with existing speech datasets across scale, language coverage, source diversity, licensing, and intended use. Its distinguishing combination is diverse English audio from web resources with transcripts and licensing that permits commercial usage.
- 2 Related Work: Earnings21 provides 39 hours of expert-transcribed earnings-call speech for evaluation, whereas The People’s Speech uses forced alignment of existing audio against transcripts.Earnings21 targets named entity recognition on challenging industry-specific jargon and is licensed under CC-BY-SA.
- 2 Related Work: The compared datasets differ in intended scope: Earnings21 targets industry-specific named entity recognition, while The People’s Speech is built as a large-scale speech recognition dataset.The comparison spans datasets designed for standard speech recognition, multilingual coverage, read speech, and specialized evaluation.
- 2 Related Work: Unlike Librispeech’s 1,000 hours of clean audiobook speech from single-speaker settings, The People’s Speech comprises a variety of settings.Librispeech’s narrow setting is identified as a significant criticism in the comparison.
- 2 Related Work: The People’s Speech combines diverse English sources with commercial-use licensing, unlike Gigaspeech’s non-commercial usage restriction and Common Voice’s volunteer-recorded speech.The People’s Speech uses existing audio and transcripts from appropriately licensed web resources, while Gigaspeech may include copyrighted sources and Common Voice solicits volunteers.
- 2 Related Work: The People’s Speech differs from MLS by focusing on English while using a more diverse set of sources than its multilingual Librivox-derived corpus.MLS contains 50,000 hours and is primarily multilingual, whereas The People’s Speech focuses only on English.
3 Dataset Description
The dataset combines public-domain and Creative Commons audio, with most hours in English and coverage across diverse content, locations, sampling rates, and acoustic backgrounds. Its characterization highlights both commercial-use licensing and suitability for speech recognition in varied environments.
- CC-BY and CC-BY-SA data permit sharing and adaptation, while attribution is recorded and CC-BY-SA sources can be filtered for commercial users.
- The dataset spans public-domain, CC-BY, and CC-BY-SA sources, with Figure 1 separating US Government public-domain works from other public-domain audio by license type.
- 51,890 of 52,500 hours are English, while 23 other languages contribute substantially fewer hours, with Malay the next largest at 40 hours.
- Content categories are detected with BART large MNLI, with government and interviews most frequent while less common categories remain non-negligible at this scale.
- Entity analysis finds considerable foreign geographic diversity despite American English, including many locations and appearances of the British political term “Tory.”
- Almost all audio has at least a 16 kHz sampling rate, supporting downsampling for common 8 kHz or 16 kHz speech-recognition systems.
4 Dataset Construction
The dataset is built by collecting commercially licensed audio with transcripts and converting it into training segments through forced alignment. Because source subtitles can be inaccurate, translated, descriptive, or poorly timed, the pipeline filters and verifies alignments before use.
- Collection and Alignment: The collection uses archive.org because it provides 52,500 hours of audio with transcripts under commercial-use licenses.The system considered Vimeo and archive.org but relied only on archive.org for this work.
- Collection and Alignment: Forced alignment converts web audio and existing transcripts into usable training data by assigning timestamps to transcript words.The pipeline transcribes audio, partitions hypotheses into roughly 15-second chunks, and aligns each chunk against the ground-truth transcript.
- Alignment Challenges: Subtitle sources require filtering because they may be translations, inaccurate transcripts, speaking notes, descriptive text, or poorly timed captions.These issues can cause transcripts to diverge from the spoken audio or provide no usable correspondence for alignment.
- Alignment Challenges: DSAlign successfully forced-aligned 90% of the data, while 10% timed out under a 200-second per-source limit.The timeout handles cases where the hypothesis transcript has no correspondence with the ground-truth transcript.
- Alignment Verification: Alignments with CER above 50% are removed, leaving 31,400 hours to the left of the threshold versus 2,050 hours to the right.The figure uses CER to assess whether aligned samples are sufficiently consistent with their ground-truth transcripts.
- Alignment Verification: Manual investigation found that boundary errors can omit transcript starts or ends, producing extra or missing words and high insertion and deletion counts.The authors filter some such cases and report ongoing work to improve boundary detection.
5 System Implementation
The implementation accelerates forced alignment by moving both acoustic inference and language-model decoding onto GPUs. Four NVIDIA T4 GPUs processed the source data in roughly three days.
- Motivation: The CPU-based DSAlign pipeline ran at 0.5 real-time factor, with about 99% of runtime spent on acoustic-model inference.AVX512 recompilation did not provide meaningful speedup, motivating accelerator use.
- Motivation: Prior profiling suggested that GPU acceleration of only the acoustic model would leave roughly 90% of runtime in CPU language-model decoding.This motivated accelerating both stages rather than moving only acoustic inference to a GPU.
- GPU Acceleration: The system uses a GPU-based Kaldi decoder for both acoustic-model inference and external language-model decoding.This avoids saving acoustic-model logits to disk for later CPU decoding.
6 Dataset Limitations
The dataset lacks internally constructed train, development, and test splits because speaker overlap and duplicate audio cannot be reliably controlled. It also has limited speaker metadata and is primarily American English.
- Evaluation Splits: No train, development, or test splits are provided because speaker overlap and duplicated audio could make evaluation sets artificially easy.The authors instead evaluate on external data without known overlap.
- Speaker Metadata: Speakers are rarely annotated, and the speaker population is open rather than a closed, identified set.This makes automatic speaker identification important for constructing reliable development and test sets.
- Language Coverage: The dataset is primarily American English, with limited evidence about the availability of non-English Creative Commons audio with transcripts.The authors suggest weakly supervised methods as one possible future response if suitable unlabeled audio exists.
7 Evaluation
The dataset is evaluated on Librispeech splits to avoid overlap with its missing internal test set. Although performance is below current Librispeech state of the art, the authors interpret the result as evidence that the dataset contains meaningful learning signal.
- Evaluation Setup: The acoustic model was evaluated on Librispeech test and development splits because the corpus has no internally constructed test or development sets.Training used a 20,000-hour subset filtered to at most 20% CER against pretrained-model outputs.
- Evaluation Context: Librispeech provides a non-overlapping evaluation setting, but its read-audiobook distribution is not a close match for conversational speech.The paper notes that train and test distributions should match and emphasizes generalization beyond read speech.
- Results: The results are worse than current Librispeech state of the art, but the authors interpret them as showing meaningful signal in the dataset.Librispeech is an audiobook benchmark, whereas the dataset is intended to support broader speech generalization.
8 Discussion
The project addresses the mismatch between industry speech-data needs and existing benchmark datasets by supporting commercial reuse and ongoing dataset maintenance. MLCommons plans to keep the dataset current while handling licensing disputes and content withdrawals.
- MLCommons plans periodic revisions to keep the dataset current and reduce concept drift as new Archive.org data becomes available.The collection pipeline is designed to update the dataset by rerunning on newly uploaded data.
- MLCommons plans processes for removing mis-licensed or creator-withdrawn content despite the selected licenses permitting dataset use.The discussion notes possible background copyrighted material and creator objections as reasons for removal mechanisms.
- The dataset targets industry use because Librispeech is not representative of industrial dataset size or domains, while purchasable alternatives restrict licenses to single entities.Non-commercial licenses may also limit benchmarking for commercial organizations.
9 Conclusions
The paper introduces a large, diverse supervised speech recognition dataset assembled from openly licensed and public-domain sources for commercial use. It also releases the data-download and force-alignment code under a commercial-use license and discusses future directions.
- The dataset sources supervised speech data from CC-BY, CC-BY-SA, and public-domain sources.
- The dataset is large, diverse in scenario, and available for commercial use.
- The authors provide source code for downloading and force-aligning the data under a commercial-use license.