Source-linked AI summary
WenetSpeech-Yue: A Large-scale Cantonese Speech Corpus with Multi-dimensional Annotation
Longhao Li, Zhao Guo, Hongjie Chen, Yuhang Dai, Ziyu Zhang, Hongfei Xue, Tianlun Zuo, Chengyou Wang, Shuiyuan Wang, Jie Li, Jian Kang, Xin Xu, Hui Bu, Binbin Zhang, Ruibin Yuan, Ziya Zhou, Wei Xue, Lei Xie
TL;DR
Cantonese lacks sufficiently large, diverse, and richly annotated speech resources for robust ASR and TTS. The paper introduces WenetSpeech-Pipe, releases the 21,800-hour WenetSpeech-Yue corpus and WSYue-eval benchmark, and reports competitive or SOTA performance across multiple evaluations. A stated scope boundary is that the commercial TTS system uses a single fixed speaker without speaker-similarity evaluation.
Problem
Cantonese has limited speech resources in scale, style, evaluation coverage, and speaker or acoustic metadata despite its linguistic complexity and large speaker population.
Method
The paper builds WenetSpeech-Yue with WenetSpeech-Pipe, a modular pipeline for multi-dimensional annotation, and releases WSYue-eval for ASR and TTS evaluation.
Results
Models trained on WenetSpeech-Yue achieve performance comparable to or exceeding SOTA systems across multiple ASR and TTS test sets.
Takeaways & Limitations
WenetSpeech-Yue and WenetSpeech-Pipe provide open resources for research on multi-domain Cantonese speech understanding and generation.
Takeaways & Limitations
The commercial TTS system uses a single fixed speaker, and speaker similarity is not considered.
Abstract
from arXiv · showhide
The development of speech understanding and generation has been significantly accelerated by the availability of large-scale, high-quality speech datasets. Among these, ASR and TTS are regarded as the most established and fundamental tasks. However, for Cantonese (Yue Chinese), spoken by approximately 84.9 million native speakers worldwide, limited annotated resources have hindered progress and resulted in suboptimal ASR and TTS performance. To address this challenge, we propose WenetSpeech-Pipe, an integrated pipeline for building large-scale speech corpus with multi-dimensional annotation tailored for speech understanding and generation. It comprises six modules: Audio Collection, Speaker Attributes Annotation, Speech Quality Annotation, Automatic Speech Recognition, Text Postprocessing and Recognizer Output Voting, enabling rich and high-quality annotations. Based on this pipeline, we release WenetSpeech-Yue, the first large-scale Cantonese speech corpus with multi-dimensional annotation for ASR and TTS, covering 21,800 hours across 10 domains with annotations including ASR transcription, text confidence, speaker identity, age, gender, speech quality scores, among other annotations. We also release WSYue-eval, a comprehensive Cantonese benchmark with two components: WSYue-ASR-eval, a manually annotated set for evaluating ASR on short and long utterances, code-switching, and diverse acoustic conditions, and WSYue-TTS-eval, with base and coverage subsets for standard and generalization testing. Experimental results show that models trained on WenetSpeech-Yue achieve competitive results against state-of-the-art (SOTA) Cantonese ASR and TTS systems, including commercial and LLM-based models, highlighting the value of our dataset and pipeline.
1 Introduction
Cantonese remains under-resourced despite its large speaker population and linguistic complexity. The paper introduces a scalable pipeline, corpus, and benchmark that address these resource gaps for ASR and TTS.
- Motivation: Cantonese speech research is constrained by limited scale, linguistic diversity, and annotation richness despite more than 84.9 million speakers.Its nine-tone system, literary–colloquial variation, and frequent English code-switching create distinctive modeling challenges.
- Motivation: Existing Cantonese corpora and evaluations provide limited domain coverage, mostly short utterances, and little speaker or acoustic metadata.These limitations restrict uses including self-supervised learning, style modeling, and multi-task training.
- Contributions: WenetSpeech-Pipe is a large-scale, multi-domain, multi-label pipeline designed for Cantonese speech understanding and generation.The paper presents it as a response to the scale, diversity, and annotation gaps in existing resources.
- Contributions: WenetSpeech-Yue provides 21,800 hours of Cantonese speech with rich multi-dimensional annotations as an open-source research resource.The corpus is described as the largest open-source Cantonese speech resource.
- Contributions: Models trained on WenetSpeech-Yue achieve SOTA performance across multiple ASR and TTS benchmarks.The paper also releases WSYue-eval to support evaluation of both tasks.
2 Related Work
Prior work has expanded multilingual speech resources and Cantonese model support, but Cantonese corpora remain substantially smaller, narrower, and less richly annotated than resources for major languages.
- Large-scale Speech Corpora: Large multilingual corpora such as GigaSpeech2 and Multilingual LibriSpeech demonstrate scalable speech-data construction across languages and modalities.These resources use automated refinement or large audiobook collections to reach broad coverage.
- Cantonese Resource Scarcity: Public Cantonese corpora remain limited in scale, domain diversity, and metadata, with examples ranging from 73.6 to 311 hours.Existing datasets are often read speech or single-speaker expressive recordings with narrow coverage.
- Cantonese Modeling: Cantonese modeling remains under-resourced relative to major languages even as multilingual ASR systems such as SenseVoice and Whisper provide strong baselines.SenseVoice is reported as trained with 9,600 hours of Cantonese within a larger multilingual corpus.
- Cantonese Modeling: Open-source TTS systems including CosyVoice2 and Step-Audio-TTS-3B increasingly support Cantonese and controllable multilingual synthesis.This reflects growing model capability, but does not remove the need for broader Cantonese training and evaluation data.
3 WenetSpeech-Pipe
WenetSpeech-Pipe combines large-scale collection, metadata annotation, multi-system transcription, normalization, voting, and alignment to produce reliable, richly annotated Cantonese speech data.
- Pipeline Overview: The pipeline comprises six modules: Audio Collection, Speaker Attributes Annotation, Speech Quality Annotation, Automatic Speech Recognition, Text Postprocessing, and Recognizer Output Voting.These stages jointly target corpus scale, annotation richness, and transcription quality.
- Audio Collection: In-the-wild long-form recordings are automatically segmented with voice activity detection into short clips suitable for transcription and quality assessment.The collection spans domains including storytelling, drama, commentary, vlogs, food, entertainment, news, and education.
- Speaker Attributes Annotation: Speaker annotation uses diarization plus age and gender estimation to support multi-speaker modeling and style-aware synthesis.The stage separates speakers within recordings and enriches segments with speaker-level metadata.
- Speech Quality Annotation: Speech quality annotation combines noise-level, perceptual-quality, and related acoustic assessments for high-fidelity generation tasks.The described measures include SNR and DNSMOS-based MOS prediction.
- Transcription and Alignment: Three ASR outputs are normalized, filtered, aligned, and fused with Cantonese-adapted ROVER to produce consensus transcriptions and confidence scores.Qwen3-4B performs minimal context-aware refinement, followed by character-level forced alignment for precise timestamps.
4 WenetSpeech-Yue
WenetSpeech-Yue is a broad Cantonese corpus with standardized metadata and confidence-aware labels, accompanied by a benchmark covering varied ASR and TTS conditions.
- 4.1 Dataset: Metadata covers audio properties, text confidence, speaker identity, SNR, DNSMOS, age, gender, and character-level timestamps.The metadata is stored in a standardized JSON file with extensible fields.
- 4.1 Dataset: The corpus organizes speech across ten domains, including storytelling, entertainment, drama, culture, vlogs, commentary, education, podcasts, news, and others.The domain distribution is presented in Figure 2.
- 4.1 Dataset: WenetSpeech-Yue contains 21,800 hours of audio with an average segment duration of 11.40 seconds.The corpus includes both short and long recordings.
- 4.2 Benchmark: WSYue-eval integrates ASR and TTS evaluation for critical dimensions of Cantonese processing.Its ASR component uses manually labeled data spanning short and long utterances, code-switching, and multi-domain conditions.
- 4.2 Benchmark: WSYue-TTS-eval contains Base and Coverage subsets for standard real-world assessment and broader linguistic or domain coverage.The Base subset contains 1,000 prompt-text pairs sampled from CommonVoice.
5 Experiments
Experiments evaluate WenetSpeech-Yue for Cantonese ASR and TTS against competitive baselines using dedicated benchmarks and objective and subjective metrics. Models trained on the corpus achieve strong ASR performance across model scales and improve pretrained TTS systems.
- 5.1 ASR Task: WenetSpeech-Yue models achieve the best performance on most ASR evaluation sets across small, medium, and LLM-augmented configurations.SenseVoice-small-Yue outperforms all baselines despite being the smallest model, while U2pp-Conformer-LLM-Yue consistently attains SOTA accuracy within the LLM group.
- 5.1 ASR Task: High-confidence data fine-tuning produces significant MER gains on both WSYue-ASR-eval test subsets.Stage 1 uses mixed-confidence data, whereas Stage 2 fine-tunes on high-confidence data; the results identify high-confidence labels as the primary driver of improvement.
- 5.2 TTS Task: CosyVoice2-Yue reaches 10.33% MER on the base set and 9.49% on the coverage set, with the highest objective SIM scores of 0.821 and 0.834.Llasa-1B-Yue also reduces MER to 10.89% and 12.78% relative to Llasa-1B, and both fine-tuned systems improve naturalness over zero-shot baselines.
- 5.2 TTS Task: CosyVoice2-Yue achieves the highest intelligibility at I-MOS 4.45 ± 0.16, while Llasa-1B-Yue leads speaker similarity at 4.11 ± 0.37 and accent nativeness at 4.34 ± 0.34.Both fine-tuned models improve all MOS metrics over zero-shot baselines; objective and perceived speaker similarity differ between the systems.
- 5.3 Conclusion: The study combines ASR and TTS evaluation with WenetSpeech-Yue, a large open-source Cantonese corpus, and WSYue-eval benchmarks for rigorous assessment.The released resources support evaluation across multiple Cantonese speech understanding and generation settings.
A Meta Data Example
The corpus stores audio metadata in standardized JSON, combining transcription, confidence, speaker, duration, and audio-quality information. Figure 4 illustrates the annotation structure.
- A Meta Data Example: Standardized JSON metadata records transcription outputs, confidence scores, duration, speaker attributes, and audio-quality metrics.Core fields include utt_id, rover_result, confidence, jyutping_confidence, duration, speaker_id, gender, and age.
- A Meta Data Example: Figure 4 presents sample annotations in JSON format.
B.1 Text Postprocessing
The text postprocessing framework standardizes heterogeneous ASR outputs before fusion. It normalizes punctuation, scripts, numbers, and word spacing.
- B.1 Text Postprocessing: Text postprocessing removes punctuation, converts Traditional Chinese to Simplified Chinese, normalizes numbers, and standardizes word spacing.The framework uses regular expressions, OpenCC, an2cn, and Pangu to make outputs consistent across ASR systems.
- B.1 Text Postprocessing: Figure 5 shows an example of the text postprocessing process.
B.2 LLM Corrector
The LLM corrector refines voted Cantonese ASR transcriptions using the original system outputs as context. It also produces confidence scores and correction analyses.
- B.2 LLM Corrector: Qwen3-4B serves as a Cantonese ASR correction expert that processes the voted transcript alongside outputs from three ASR systems.The module generates refined transcriptions with confidence scores from 0 to 100 and detailed correction analyses.
- B.2 LLM Corrector: Figure 6 illustrates the prompt used by the LLM corrector.
C.1 Coverage Subset Construction
The Coverage Subset is systematically constructed to test models across diverse domains and Cantonese linguistic phenomena. It combines curated seed texts, LLM-based sentence expansion, and human review.
- C.1 Coverage Subset Construction: The construction pipeline targets broad coverage of domains, linguistic phenomena, and conversational Cantonese through manually curated seed texts.Categories include genres such as news and literature, phenomena such as numerals, named entities, polyphonic characters, and code-switching, and colloquial features such as discourse particles and slang.
- C.1 Coverage Subset Construction: Representative seed examples are either retained as complete sentences or expanded with an LLM into natural, fluent sentences suitable for TTS input.The expansion process uses a dedicated prompt, illustrated in Figure 8.
- C.1 Coverage Subset Construction: All generated sentences undergo human review for correctness and adherence to Cantonese linguistic norms.
C.2 Coverage Subset Examples
The Coverage Subset provides examples spanning varied domains and Cantonese linguistic phenomena. Figure 10 illustrates this breadth through representative sentences.
- C.2 Coverage Subset Examples: Representative examples in Figure 10 span multiple domains and Cantonese linguistic phenomena, illustrating the subset’s intended diversity and representativeness.