PaperlayerView exact revision on arXiv

Source-linked AI summary

Summary of the ChinaVoices Challenge 2026: Data, Tasks, Baseline, and Methods

Yujie Liao, Bingshen Mu, Shuiyuan Wang, Liumeng Xue, Hexin Liu, Xian Shi, Jie Hu, Lei Xie

arXiv:2609.03471v1eess.AS

TL;DR

Chinese dialect identification and transcription still lack a unified, public evaluation platform covering a broad range of dialects. ChinaVoices establishes two tasks and shared evaluation data across 16 categories; its best restricted-data systems outperform the unified baseline on both tasks.

  • Problem

    Chinese dialect speech processing lacks a unified, public, reproducible platform and a standardized evaluation set covering a broad range of dialects.

    Source: §1 · p. 2

  • Method

    ChinaVoices defines identification and ASR tasks for 16 dialect categories, using shared evaluation audio, three speaker-disjoint sets, and restricted-data and open-data tracks.

    Source: §1 · p. 2

  • Results

    The best restricted-data systems reach 83.19% ACC and 11.08% CER, compared with the Qwen3-ASR-1.7B baseline's 53.62% macro-averaged ACC and 18.10% macro-averaged CER.

    Source: §5 · p. 13

  • Takeaways & Limitations

    The platform provides a common evaluation setting for comparing dialect identification and ASR across 16 categories.

    Source: §1 · p. 2

    Because leaderboard comparisons involve complete systems from different teams, they cannot estimate the independent contribution of individual components.

    Source: Multi-Dialect Identification Systems · p. 12

Abstract

from arXiv · show

This paper summarizes the ChinaVoices Challenge 2026, which aims to establish unified task definitions and evaluation conditions for Chinese dialect speech processing and to advance multi-dialect identification and automatic speech recognition. The challenge covers 16 dialect categories and defines two tasks: Chinese Multi-Dialect Identification and Chinese Multi-Dialect Automatic Speech Recognition (ASR). It uses approximately 320 hours of speech across the Reference Set, Open Evaluation Set, and Hidden Evaluation Set. The two tasks use the same evaluation audio, and each includes restricted-data and open-data tracks. We describe the task settings, data, evaluation metrics, and Qwen3-ASR-1.7B baseline, and analyze the leaderboard results and submitted systems. In total, 28 teams submit results, 17 provide system reports, and systems from 15 teams pass the compliance review and are included in the analysis. Most eligible systems outperform the baseline, and the official top-three order remains unchanged on the Hidden Evaluation Set for both tasks. Dialect-level results show that categories with higher identification accuracy generally have lower ASR error rates, although the tasks assess related but distinct capabilities. Leading identification systems commonly exploit dialect-discriminative acoustic representations, whereas leading ASR systems emphasize data normalization, augmentation, and auxiliary CTC objectives. These results provide practical guidance for developing and evaluating Chinese multi-dialect speech processing systems.

1 Introduction

The paper motivates a unified evaluation platform for Chinese dialect identification and ASR, where broad, reproducible comparison remains difficult. ChinaVoices Challenge 2026 defines both tasks across 16 dialect categories and supplies shared evaluation data and procedures.

1 Introduction

  • Although Mandarin ASR is widely deployed, stable dialect classification and accurate transcription remain challenging under multi-dialect and low-resource conditions.Source: §1 · pp. 1–2
  • Prior work spans acoustic representations, data resources, and model architectures, but differing dialect inventories, resources, and evaluation sets impede direct comparison.Source: §1 · p. 2
  • Chinese dialect speech processing lacks a unified, public, reproducible platform and broad standardized evaluation sets, limiting fair comparison across regional varieties.Source: §1 · p. 2
  • ChinaVoices defines identification and ASR tasks over 16 dialect categories, with speaker-disjoint Reference, Open Evaluation, and Hidden Evaluation sets and manually quality-checked transcripts.Both tasks use the same evaluation audio, macro-averaged ACC and CER, and restricted-data and open-data tracks; restricted tracks determine official rankings.Source: §1 · pp. 2–3

2 Challenge Tasks and Tracks

ChinaVoices defines two tasks on shared evaluation audio, each with restricted-data and open-data tracks. The official metric for each task macro-averages performance across dialect categories.

2 Challenge Tasks and Tracks

  • The two tasks share evaluation audio and 16 dialect categories, but differ in output and metric; teams may enter either or both tasks and use separate or unified models.Source: §2 · p. 3

2.1 Multi-Dialect Identification

  • Identification assigns one of 16 official dialect labels to an utterance without producing a transcript, and its official metric is macro-averaged ACC.Macro-averaging gives every dialect category equal weight.Source: §2.1 · p. 3

2.2 Multi-Dialect ASR

  • ASR transcribes each utterance into a character sequence, and its official metric is macro-averaged CER across dialect categories.CER counts substitutions, deletions, and insertions relative to reference characters; lower values indicate better transcription.Source: §2.2 · p. 4

2.3 Tracks and Leaderboards

  • Each task has restricted-data and open-data tracks; the restricted track sets official rankings, while open-data results explore performance with broader legally usable data.Source: §2.3 · p. 4

3 Challenge Data and Baseline System

The challenge provides speaker-disjoint data for 16 dialects and a unified Qwen3-ASR-1.7B baseline for identification and transcription. Its evaluation sets and training rules support controlled evaluation and reproducibility.

3.1 Challenge Data

  • The three challenge sets provide about 3, 7, and 10 hours per dialect for Reference, Open Evaluation, and Hidden Evaluation, respectively, with speaker-disjoint partitions.Open Evaluation audio determines public leaderboard scores; Hidden Evaluation data independently verifies leading restricted-data systems, and neither evaluation set may be used for training or targeted data construction.Source: §3.1 · pp. 4–5
  • Restricted-data teams can choose among permitted public resources, so system reports are needed to interpret training-mixture differences while preserving public reproducibility.Source: §3.1 · p. 5

3.2 Unified Baseline

  • The Qwen3-ASR-1.7B baseline uses one conditional-generation process to output a dialect label and transcript, adapting the language model with LoRA while freezing the audio encoder and aligner.On the Open Evaluation Set, it achieves 53.62% macro-averaged ACC and 18.10% macro-averaged CER; its manifests, configurations, and reproduction scripts are available.Source: §3.2 · p. 5

4 Results and Analysis

The results compare eligible systems across restricted-data and open-data leaderboards for dialect identification and ASR. Most eligible systems outperform the baseline, and the official top-three order remains unchanged on the Hidden Evaluation Set for both tasks.

4 Results and Analysis

  • Of 28 teams submitting leaderboard entries, 17 provide reports and systems from 15 eligible teams enter the analysis.Source: §4 · p. 5

4.1 Overall Results

  • Restricted-data leaderboards determine official placements, whereas open-data leaderboards report only Open Evaluation Set results.Source: §4.1 · p. 6
  • 83.19% ACC for scy919 exceeds the 53.62% baseline, while TeleASR reaches 11.08% CER against the 18.10% baseline.Source: Table 1 · p. 6, §4.1 · p. 6, Table 2 · p. 7
  • Identification systems show a 27.07-percentage-point best-to-worst gap, while the ASR top-three CER values span 0.67 percentage points.The leading ASR systems are more closely grouped than the leading identification systems, although the full ASR leaderboard still has a substantial performance range.Source: §4.1 · pp. 6–7
  • The official top-three order remains unchanged on the Hidden Evaluation Set for both tasks, establishing leaderboard-head stability but not overall generalization comparisons between tasks.Source: §4.1 · pp. 7–8
  • Open-data submissions do not consistently outperform restricted-data counterparts, and fewer eligible entrants limit estimates of average external-data benefit.Open-data results are more informative about attainable complete-system performance under broader resource rules.Source: §4.1 · p. 7

4.2 Dialect-Level Metrics and Error Analysis

The dialect-level analysis presents ACC by official challenge label for the top-three restricted-data identification systems on the Open Evaluation Set.

4.2 Dialect-Level Metrics and Error Analysis

  • Table 3 reports dialect-level ACC (%) by official challenge label.Source: Table 3 · p. 8
  • The table covers the top-three restricted-data multi-dialect identification systems on the Open Evaluation Set.Source: Table 3 · p. 8
  • Systems in both dialect-level tables are ordered by official rank, and the mean column uses unrounded scores.Source: §4.2 · p. 8

Multi-Dialect Identification Difficulty and Confusions

Identification difficulty is concentrated in a few dialect categories, and the leading systems make directional errors between particular dialects. The observed confusions suggest category-specific design priorities rather than simply increasing training samples uniformly.

Multi-Dialect Identification Difficulty and Confusions

  • Kejia is hardest to identify at 43.25% mean ACC; dongbei, cantonese, and shan3xi are easiest at 99.48%, 98.62%, and 95.56%.Shan1xi, nanjing, chaoshan, and nanchang also have mean ACC below 68.00%.Source: Multi-Dialect Identification… · p. 8
  • The mean row-normalized confusion matrix highlights directional errors, including minnan→chaoshan at 19.90% versus 12.73% in reverse.Kejia is classified as chaoshan at 16.93% and nanchang at 19.73%; changsha→wuhan is 18.55% versus 6.34% in reverse.Source: Figure 1 · p. 9, Multi-Dialect Identification… · p. 9
  • The difficulty of weak categories suggests hard negatives among similar dialects, intermediate or multi-layer acoustic representations, and stronger category-boundary modeling.The passage cautions that improvement is unlikely to require only a uniform increase in training samples.Source: Multi-Dialect Identification… · p. 9

Multi-Dialect ASR Difficulty and Error Composition

ASR error varies substantially by dialect, with kejia the most difficult and substitutions the dominant error type. Similar overall CER can reflect different error compositions, while dialect-level identification accuracy and ASR CER are strongly associated but not causally linked.

Multi-Dialect ASR Difficulty and Error Composition

  • Kejia has the highest mean CER at 31.01%; chaoshan, shan1xi, wuyu, and minnan follow at 21.89%, 17.71%, 17.37%, and 15.04%.Shan3xi, wuhan, and dongbei are relatively easy at 4.95%, 5.09%, and 5.24% mean CER.Source: Multi-Dialect ASR Difficulty and Error… · p. 10
  • Substitutions make up 82.80–87.30% of edit errors and dominate; insertions are only 5.30–5.60% of errors, with overall rates of 0.61–0.63%.Difficult-dialect substitution rates reach 24.22–31.72% for kejia, 16.04–20.64% for chaoshan, and 14.79–17.50% for shan1xi.Source: Multi-Dialect ASR Difficulty and Error… · pp. 10–11
  • TeleASR’s lower deletion rate offsets its higher substitution rate relative to Mlslabs, yielding slightly lower overall CER.Similar total CER values can therefore arise from different error compositions.Source: Multi-Dialect ASR Difficulty and Error… · pp. 10–11
  • Across 16 dialects, identification ACC and dialect-level CER have a strong negative association (Pearson’s r = −0.76; Spearman’s ρ = −0.74), but the relationship is descriptive, not causal.Higher identification ACC generally accompanies lower CER.Source: Multi-Dialect ASR Difficulty and Error… · p. 11

relationship between the two tasks.

The two tasks are related but distinct: some dialects diverge from the general relationship between identification accuracy and transcription error. Identity cues alone do not guarantee lexical or pronunciation knowledge, so identification cannot replace direct ASR evaluation.

relationship between the two tasks.

  • Wuyu combines high mean ACC (87.57%) with high mean CER (17.37%), while nanjing combines 67.08% ACC with 7.36% CER and wuhan 75.07% ACC with 5.09% CER.These exceptions show that the tasks capture related but distinct aspects of dialect speech.Source: relationship between the two tasks · p. 11
  • Salient dialect-identity cues may not provide enough lexical and pronunciation knowledge for transcription, and category confusion does not always imply poor ASR.Identification information may complement ASR but cannot replace direct transcription evaluation.Source: relationship between the two tasks · pp. 11–12

4.3 System Analysis

The leading systems use distinct identification and ASR pipelines, with recurring design patterns across teams. Because these are comparisons of complete systems, they do not isolate the contribution of individual components.

Multi-Dialect Identification Systems

  • Identification systems combine dialect-specific and intermediate or multi-layer representations with complementary classifiers, objectives, or generative category coding.scy919 uses FireRedLID and Dolphin-CN-Dialect features with TabPFN; Optima combines FireRedLID features and multiple objectives; zenava.ai adapts Qwen3-Omni with LoRA.Source: Multi-Dialect Identification Systems · p. 12
  • Recurring identification design choices include dialect-specific representations, intermediate or multi-layer features, and complementary decisions, but complete-system comparisons cannot isolate any component’s contribution.The patterns are not explained by model scale alone.Source: Multi-Dialect Identification Systems · p. 12
  • The three leading ASR systems share a FireRedASR2-AED foundation but differ in normalization, augmentation, training constraints, difficult-dialect sampling, and auxiliary CTC use.TeleASR emphasizes cross-corpus normalization and alignment; Mlslabs uses staged freezing and AED–CTC constraints; Kyoto-Tsinghua uses layer-wise learning rates, auxiliary CTC, and SeedVC.Source: Multi-Dialect ASR Systems · p. 12

Multi-Dialect ASR Systems

Competitive ASR systems improve the foundation model through coordinated data processing, training objectives, and decoding. Foundation-model choice shapes the performance starting point, but task-matched methods also contribute to final system variation.

Multi-Dialect ASR Systems

  • Competitive ASR systems jointly optimize data processing, training objectives, and decoding rather than relying only on a stronger foundation model.TeleASR uses transcription normalization, context expansion, and augmentation; Mlslabs and the Kyoto-Tsinghua Team use auxiliary CTC objectives for sequence alignment.Source: Multi-Dialect ASR Systems · p. 12

Foundation Model Choice and Final Performance

  • FireRedASR2-AED systems score 11.08%–13.92% CER, with a 11.75% median; five of the top six restricted-data systems use it.Among five systems using this foundation model, the best and worst results still differ by 2.84 percentage points in CER.Source: Foundation Model Choice and Final… · pp. 12–13
  • Qwen3-ASR restricted-data systems span 13.48%–16.36% CER, and identification results show no monotonic relation between model size or updated-parameter fraction and rank.The foundation model influences the performance starting point but does not determine final results; task-matched representations, data processing, objectives, and inference also matter.Source: Foundation Model Choice and Final… · p. 13

5 Conclusion

ChinaVoices Challenge 2026 provides a unified platform for 16 dialect categories across identification and ASR, with shared evaluation audio and restricted- and open-data tracks. Most top-three rankings persist on the hidden set, while results associate dialect-level identification accuracy with ASR error rates and show foundation models do not alone determine system performance.

5 Conclusion

  • The challenge covers 16 dialect categories and two tasks, using shared audio across speaker-disjoint Reference, Open Evaluation, and Hidden Evaluation Sets.Each task has restricted-data and open-data tracks.Source: §5 · p. 13
  • On the Open Evaluation Set, the Qwen3-ASR-1.7B baseline achieves 53.62% macro-averaged ACC and 18.10% macro-averaged CER; best restricted-data systems reach 83.19% ACC and 11.08% CER.The official top-three order remains unchanged on the Hidden Evaluation Set for both tasks.Source: §5 · p. 13
  • Dialect categories with higher identification ACC generally have lower dialect-level CER, though the tasks measure related but distinct capabilities.High-ranking systems use dialect-aware or multi-layer acoustic representations, data processing and augmentation, difficult-dialect sampling, and auxiliary CTC objectives.Source: §5 · p. 13
  • Same-team submissions across data tracks show mixed performance changes, so these results do not isolate the benefit of broader training resources.Future editions could broaden real-world scenarios, refine transcription guidelines, standardize reporting, and provide more comprehensive dialect-level results and error statistics.Source: §5 · p. 13
Loading 2609.03471v1…