Source-linked AI summary
WenetSpeech4TTS: A 12,800-hour Mandarin TTS Corpus for Large Speech Generation Model Benchmark
Linhan Ma, Dake Guo, Kun Song, Yuepeng Jiang, Shuai Wang, Liumeng Xue, Weiming Xu, Huan Zhao, Binbin Zhang, Lei Xie
TL;DR
Large-scale Mandarin TTS lacks comparably extensive open datasets and public benchmarks, while WenetSpeech has noisy, fragmented, and speaker-mixed segments. This paper refines WenetSpeech into a large TTS corpus with quality-based subsets and evaluates VALL-E and NaturalSpeech 2 on them. The evaluations report better performance from higher-quality subsets, and the corpus and benchmark are publicly available.
Problem
Open large-scale TTS datasets are English or multilingual, and public benchmarks for large-scale Chinese TTS are lacking; WenetSpeech also has noise, fragmented segments, and speaker mixing that limit its TTS suitability.
Source: §1 · p. 1Method
The authors refine WenetSpeech through segment merging and boundary extension, audio enhancement, speaker diarization, improved transcription, and quality filtering.
Source: §1 · pp. 1–2Results
Takeaways & Limitations
Abstract
from arXiv · showhide
With the development of large text-to-speech (TTS) models and scale-up of the training data, state-of-the-art TTS systems have achieved impressive performance. In this paper, we present WenetSpeech4TTS, a multi-domain Mandarin corpus derived from the open-sourced WenetSpeech dataset. Tailored for the text-to-speech tasks, we refined WenetSpeech by adjusting segment boundaries, enhancing the audio quality, and eliminating speaker mixing within each segment. Following a more accurate transcription process and quality-based data filtering process, the obtained WenetSpeech4TTS corpus contains $12,800$ hours of paired audio-text data. Furthermore, we have created subsets of varying sizes, categorized by segment quality scores to allow for TTS model training and fine-tuning. VALL-E and NaturalSpeech 2 systems are trained and fine-tuned on these subsets to validate the usability of WenetSpeech4TTS, establishing baselines on benchmark for fair comparison of TTS systems. The corpus and corresponding benchmarks are publicly available on huggingface.
1. Introduction
Large-scale Mandarin TTS lacks comparably extensive open datasets and public benchmarks, while WenetSpeech has quality and segmentation problems for synthesis. The paper refines WenetSpeech into WenetSpeech4TTS and evaluates TTS systems on its quality-based subsets.
Motivation
- Open large-scale TTS datasets are English or multilingual, and public benchmarks for large-scale Chinese TTS are lacking, making fair system comparison difficult.The cited Chinese dataset, DIDISPEECH, contains nearly 800 hours of reading-style speech; reproducing and training other systems is costly.Source: §1 · p. 1
- WenetSpeech’s noisy audio, fragmented or truncated segments, and lack of speaker homogeneity reflect its ASR-oriented design and limit its suitability for TTS.Its video subtitles can split sentences across short segments, and subtitle timing errors can truncate words.Source: §1 · p. 1
Contributions
- WenetSpeech4TTS provides 12,800 hours of TTS-tailored data in Basic, Standard, and Premium subsets containing 7,226, 4,056, and 945 effective hours, respectively.The subsets are categorized by speech quality.Source: §1 · p. 1
Method and evaluation
- The pipeline merges segments, extends boundaries, enhances audio, scores quality, clusters speakers, and improves transcription; evaluations of VALL-E and NaturalSpeech 2 report better performance on higher-quality subsets.The corpus and corresponding benchmark are publicly available.Source: §1 · pp. 1–2
2. Processing pipeline
The processing pipeline adapts WenetSpeech for TTS by restructuring segments, improving audio and transcription, detecting speaker mixing, and filtering by quality. It combines these operations to produce refined training segments.
2.1. Adjacent segments merging
- Adjacent segments are merged when their gap is below 0.55 seconds and speaker-embedding cosine similarity exceeds 0.65, repeating until sentence duration reaches 20 seconds.The procedure uses Resemblyzer embeddings and applies speech enhancement before calculating similarity.Source: §2.1 · p. 2, Figure 1 · p. 2
2.2. Boundary extension
- Segment starts and ends are extended after merging to reduce truncated words, without crossing a neighboring interval’s midpoint or extending more than 0.5 seconds.The operation is applied to both segment starts and ends.Source: §2.2 · p. 2, Figure 2 · p. 2
2.3. Speech enhancement
- DNSMOS P.808 scores improved significantly after enhancement on a random sample of 10,000 segments.MBTFNet is used to remove noise, background music, and other interference before downstream processing.Source: §2.3 · p. 2, Figure 3 · p. 2
2.4. Multi-speaker detection
- Segments are flagged as multi-speaker when embeddings fall into different clusters or when more than half are marked centrifugal.The process uses spectral clustering, merges cluster centers with cosine similarity above 0.75, and labels the 10% of embeddings farthest from each center as centrifugal.Source: §2.4 · pp. 2–3
2.5. Speech recognition
- Paraformer-large supplies new transcriptions and per-segment confidence scores; it achieved 6.74% character error rate on WenetSpeech’s Test Net set.New transcripts address restructuring, OCR deletion errors, and the outdated Podcast-data ASR system.Source: §2.5 · p. 3
2.6. Quality filtering
- The final filtering excludes multi-speaker segments and segments with speech-recognition confidence below 0.7, yielding 12,800 hours of speech segments.The merging operation also incorporates some intervals between segments that were not included in WenetSpeech.Source: §2.6 · p. 3
3. The WenetSpeech4TTS corpus
The corpus is divided into speech-quality subsets, and its segment-length distribution differs from that of WenetSpeech. Corpus materials, scores, and test-set details are also reported.
Quality subsets
- DNSMOS P.808 scores above 4.0 define Premium, above 3.8 define Standard, and above 3.6 define Basic; lower-scoring segments are Rest.Figure 4 plots DNSMOS P.808 scores against the scale of data at those scores.Source: §3 · p. 3, Figure 4 · p. 3
Segment lengths
- Under 3-second segments decreased and segments over 5 seconds increased in the Basic subset compared with WenetSpeech after merging.Figure 5 compares the segment-length distributions in WenetSpeech and the Basic subset.Source: §3 · p. 3, Figure 5 · p. 3
Test set and release
- The test set contains 150 sentences and 26 target speakers: 10 in-set and 16 out-of-set, with equal numbers of male and female speakers.The out-of-set speakers include participants from WenetSpeech Test Net, Aishell3, and amateur speakers.Source: §3 · p. 3
- The open-sourced corpus includes segments, transcripts, and DNSMOS scores, with audio samples available on the demo page.Source: §3 · p. 3
4. Experiments
The experiments train and fine-tune VALL-E and NaturalSpeech 2 on WenetSpeech4TTS subsets and evaluate objective and subjective speech quality. Both models show better objective and subjective results with higher-quality training data, while some outcomes differ by model and speaker familiarity.
4.1. VALL-E and NaturalSpeech 2
- VALL-E and NaturalSpeech 2 are trained on Basic and successively fine-tuned on Standard and Premium to establish benchmark baselines.VALL-E uses AudioDec, while NaturalSpeech 2 uses Encodec and external duration information; their training strategies otherwise follow the same subset progression.Source: §4 · p. 3, §4.1 · pp. 3–4
4. Experiments
- The experiments train and fine-tune two popular large-scale TTS models on WenetSpeech4TTS subsets, and the model weights are open-sourced with the dataset.Table 2 reports results with 95% confidence intervals for Basic training and successive Standard and Premium fine-tuning.Source: §4 · p. 3, Table 2 · p. 4
4.2. Evaluation metrics
- The evaluation measures speaker similarity with SECS, intelligibility with CER, naturalness with NMOS, and speaker similarity MOS with SMOS.Objective tests synthesize 150 utterances per speaker; subjective tests use 80 cases from five seen and five unseen speakers, rated by 20 native Mandarin listeners on a 1–5 scale.Source: §4.2 · p. 4
4.3. Objective evaluation
- Higher-quality training data significantly decreases CER for both models; NaturalSpeech 2 has lower CER than VALL-E across training sets and speakers.VALL-E’s SECS remains relatively stable as data quality improves, while NaturalSpeech 2’s SECS trends upward.Source: §4.3 · p. 4
- CER and SECS are worse for unseen than seen speakers, indicating better intelligibility and similarity on speakers encountered during training.The authors also report relatively high SECS across seen and unseen speakers.Source: §4.3 · p. 4
4.4. Subjective evaluation
- As data quality improves, NMOS generally rises, and most SMOS scores gradually improve; VALL-E’s in-set SMOS is relatively stable.VALL-E has higher naturalness than NaturalSpeech 2 under the same conditions; the authors attribute this to errors from NaturalSpeech 2’s external duration alignment model using automatically transcribed data.Source: §4.4 · p. 4
5. Conclusion
WenetSpeech4TTS is a multi-domain Mandarin corpus derived from WenetSpeech for large TTS model training. Experiments with VALL-E and NaturalSpeech 2 support its usability, with higher-quality subsets achieving better performance.
5. Conclusion
- WenetSpeech4TTS is a multi-domain Mandarin corpus derived from WenetSpeech, refined and divided into speech-quality subsets for large TTS model training.The authors trained and fine-tuned VALL-E and NaturalSpeech 2 on the subsets to provide a benchmark.Source: §5 · p. 4