Source-linked AI summary
WenetSpeech: A 10000+ Hours Multi-domain Mandarin Corpus for Speech Recognition
Binbin Zhang, Hang Lv, Pengcheng Guo, Qijie Shao, Chao Yang, Lei Xie, Xin Xu, Hui Bu, Xiaoyu Chen, Chenchen Zeng, Di Wu, Zhendong Peng
TL;DR
Open-source Mandarin ASR research lacks corpora with both large scale and broad domain coverage. WenetSpeech constructs and validates a 22435-hour corpus from YouTube and Podcast data, providing labeled subsets, evaluation sets, and toolkit baselines. The reported benchmarks show strong performance on AIShell-1, challenging results on the designed tests, and steadily improving performance as training-data scale increases, while the baselines are not intended as state-of-the-art results.
Problem
Many open-source ASR corpora remain small and lack domain diversity, unlike industrial systems trained on tens of thousands of hours.
Method
WenetSpeech combines OCR-based YouTube caption recognition, high-quality Podcast ASR transcription, and CTC-based end-to-end force alignment for candidate validation.
Results
22435 hours of raw audio are organized into 10005 hours of Strong Label data and 2478 hours of Weak Label data, while benchmark performance rises steadily with increasing data scale.
Takeaways & Limitations
WenetSpeech provides a large, domain-diverse Mandarin corpus with evaluation sets and Kaldi, ESPnet, and WeNet baselines for speech-recognition research.
Takeaways & Limitations
The reported toolkit results are baseline systems and may not represent state-of-the-art performance for those toolkits.
Abstract
from arXiv · showhide
In this paper, we present WenetSpeech, a multi-domain Mandarin corpus consisting of 10000+ hours high-quality labeled speech, 2400+ hours weakly labeled speech, and about 10000 hours unlabeled speech, with 22400+ hours in total. We collect the data from YouTube and Podcast, which covers a variety of speaking styles, scenarios, domains, topics, and noisy conditions. An optical character recognition (OCR) based method is introduced to generate the audio/text segmentation candidates for the YouTube data on its corresponding video captions, while a high-quality ASR transcription system is used to generate audio/text pair candidates for the Podcast data. Then we propose a novel end-to-end label error detection approach to further validate and filter the candidates. We also provide three manually labelled high-quality test sets along with WenetSpeech for evaluation -- Dev for cross-validation purpose in training, Test_Net, collected from Internet for matched test, and Test\_Meeting, recorded from real meetings for more challenging mismatched test. Baseline systems trained with WenetSpeech are provided for three popular speech recognition toolkits, namely Kaldi, ESPnet, and WeNet, and recognition results on the three test sets are also provided as benchmarks. To the best of our knowledge, WenetSpeech is the current largest open-sourced Mandarin speech corpus with transcriptions, which benefits research on production-level speech recognition.
1. INTRODUCTION
WenetSpeech addresses the limited size and domain diversity of open-source Mandarin ASR corpora by releasing a large, diverse corpus collected from the Internet. It combines high-quality transcription pipelines with baseline systems for three major speech-recognition toolkits.
- Motivation: Open-source ASR corpora are often small and lack domain diversity compared with industrial systems trained on tens of thousands of hours.The paper identifies this mismatch as a motivation for a larger, more diverse Mandarin resource.
- Corpus release: WenetSpeech is a non-commercially licensed, large multi-domain Mandarin speech corpus whose data are collected from the Internet.Its name reflects Internet-based collection and data sharing.
- Corpus scope: The corpus covers multiple speaking styles, scenarios, domains, topics, and noisy conditions, with extensible metadata for future expansion.The authors position this diversity as useful for various speech-recognition tasks.
- Construction and benchmarks: OCR-based YouTube caption recognition, ASR-based Podcast transcription, and end-to-end label-error detection support high-quality transcription generation.The paper also provides baselines using Kaldi, ESPnet, and WeNet.
2. CREATION PIPELINE
The corpus construction pipeline collects categorized YouTube and Podcast audio, generates audio-text candidates with OCR or ASR, and validates them using confidence-based end-to-end label-error detection. Candidates are then separated into strong, weak, or rejected data according to estimated label quality.
- Audio collection: WenetSpeech audio is manually organized into 10 domain categories and collected from YouTube and Podcast playlists.YouTube videos support OCR-based candidate generation, while Podcast audio is prepared for transcription by a high-quality ASR system.
- YouTube OCR: YouTube candidates are generated by detecting subtitle text, validating subtitle regions, detecting subtitle changes, and recognizing text with a CRNN-CTC system.The pipeline uses subtitle timing and visual changes to define candidate boundaries.
- YouTube OCR: 5000 manually annotated YouTube subtitle transcriptions yielded 98% Text Recognition accuracy, supporting the reliability of the OCR pipeline.The benchmark samples subtitles from videos with different themes.
- YouTube OCR: Long or asynchronous YouTube subtitle phrases are merged consecutively until the audio duration exceeds 8 seconds.This step addresses inaccurate subtitle boundary detection caused by fragmented annotations.
- Podcast ASR: Podcast audio is segmented by VAD and transcribed by a commercial ASR system reported to exceed 95% accuracy in most tested scenarios.The system generates short audio-text pair candidates after VAD segmentation.
- Candidate validation: Candidate errors include subtitle annotation, timestamp, OCR, transcription, and text-normalization errors, motivating confidence-based validation and filtering.The validation stage classifies candidates by confidence and removes extremely poor candidates.
- Force alignment: The proposed CTC-based end-to-end force-alignment method builds an alignment graph for each candidate and permits deletion, insertion, and substitution operations.The graph includes the oracle transcription path and filler-state arcs for modeling discrepancies between audio and text.
- Force alignment: The force-alignment graph is composed with the CTC topology graph to form the final force-decoding graph used for label-error detection.The resulting graph is applied to validate candidate transcriptions.
3. THE WENETSPEECH CORPUS
WenetSpeech provides a large, confidence-partitioned Mandarin corpus with diverse domains and dedicated evaluation sets. Its metadata, training subsets, and test sets support varied speech recognition uses.
- Metadata and Audio Format: Metadata records source paths, public URLs, domain tags, checksums, segments, timestamps, labels, confidence, and subset information in an extensible JSON file.Audio is converted to 16 kHz, single-channel, 16-bit format and Opus-compressed at 32 kbps.
- Size and Confidence: 22435 hours of raw audio are partitioned into 10005 hours of Strong Label data, 2478 hours of Weak Label data, and invalid Others.Strong Label data has confidence greater than 0.95, while Weak Label data has confidence between 0.60 and 0.95.
- Training Data Diversity and Subsets: Training data from YouTube and Podcast is classified into 10 categories spanning speaking styles, topics, and scenarios.About 4k hours come from drama, a broad domain covering varied themes and situations.
- Training Data Diversity and Subsets: Three training subsets, S, M, and L, are provided for building ASR systems at different data scales.Subsets S and M are sampled from training data with oracle confidence 1.0.
- Evaluation Sets: Dev supports cross-validation, Test Net provides matched Internet speech, and Test Meeting provides mismatched far-field meeting speech from 197 real meetings.Test Meeting includes conversational, spontaneous speech recorded in varied rooms.
4. EXPERIMENTS
The paper supplies Kaldi, ESPnet, and WeNet baselines trained on WenetSpeech and evaluates them across multiple test sets. Results cover toolkit benchmarks and data-scale effects, while the authors caution that these are baseline rather than state-of-the-art systems.
- Kaldi Baseline: The Kaldi baseline uses a classical chain model trained with different amounts of WenetSpeech data.Its architecture includes CNN, TDNN-F, time-restricted attention, and LSTMP components.
- ESPnet Benchmark: The ESPnet baseline uses a Conformer encoder and Transformer decoder with a combined CTC and attention objective.The Conformer captures global context through self-attention and local correlations through convolution.
- WeNet Benchmark: The WeNet baseline implements a U2 model that unifies streaming and non-streaming end-to-end speech recognition in one model.Its basic setup largely follows the ESPnet baseline with modified feature extraction and training settings.
- Experimental Results: The reported systems are intended only as toolkit baselines and may not represent state-of-the-art performance.This qualification limits interpretation of the benchmark results as toolkit ceilings.
- Experimental Results: Kaldi, ESPnet, and WeNet baselines are evaluated on three designed test sets and AIShell-1 using Mixture Error Rate (MER).MER treats Mandarin characters and English words as tokens in edit-distance calculation.
- Experimental Results: As WenetSpeech training data increases across subsets, Kaldi baseline performance improves steadily.Table 6 reports MER% for different WenetSpeech training subsets.