PaperlayerView exact revision on arXiv

Source-linked AI summary

Dialect-Robust Speech Language Models with Synthetic Pseudo-Dialect Augmentation

Shunsuke Mitsumori, Tomoya Mizumoto, Yusuke Fujita

arXiv:2610.09321v1cs.CLeess.AS

TL;DR

Dialect robustness in SLMs is constrained by scarce real dialect speech, while conventional TTS augmentation requires dialect speech for synthesis. This paper generates pseudo-dialect speech from LLM-rewritten text using standard TTS and adds intermediate standard-text prediction; it reports improved translation results across languages and architectures.

  • Problem

    SLMs degrade on dialect speech amid a lack of real dialect training data, and conventional TTS augmentation requires sufficient real speech from each target dialect.

    Source: §I · p. 1

  • Method

    The approach converts LLM-generated dialect text into pseudo-dialect speech with standard-language TTS and adds intermediate standard-text prediction during training.

    Source: §I · p. 1

  • Results

    26.24 from 25.38 for Japanese BLEU and 32.47 from 31.57 for German; intermediate standard-text prediction raised Japanese to 28.26 and Chinese from 11.67 to 16.37.

    Source: §I · p. 1

  • Takeaways & Limitations

    The authors present the approach as a practical, scalable way to expand SLM dialect coverage without extensive dialect speech collection.

    Source: §VII · p. 6

Abstract

from arXiv · show

Speech Language Model (SLM) performance often degrades on dialects due to data scarcity. Conventional text-to-speech (TTS) augmentation struggles to cover diverse dialects as it requires a certain amount of real dialect speech. We propose synthesizing pseudo-dialect speech by converting LLM-generated dialect text via a standard-language TTS model, requiring zero real dialect speech. Additionally, we introduce intermediate standard-text prediction during training, acting as semantic normalization for downstream tasks. We evaluate dialect understanding via dialect-to-English speech translation across Japanese, German, and Chinese dialects. Compared to synthetic standard speech baselines, pseudo-dialect augmentation improves scores for Japanese (from 25.38 to 26.24) and German (from 31.57 to 32.47). Furthermore, the intermediate standard-text prediction effectively bridges the semantic gap, boosting performance to 28.26 for Japanese and from 11.67 to 16.37 for Chinese. These results suggest that our approach scales to various languages without requiring speech resources specific to each dialect.

I. INTRODUCTION

Dialect speech data scarcity undermines SLM robustness, while conventional TTS augmentation requires real dialect speech. The paper proposes pseudo-dialect augmentation and intermediate standard-text prediction, and reports translation improvements across languages and model architectures.

I. INTRODUCTION

  • The framework rewrites standard text into dialectal text with an LLM, then uses standard-language TTS to synthesize pseudo-dialect speech without real dialect speech resources.The approach reuses standard-language speech resources because dialects share acoustic features with their standard language.Source: §I · p. 1
  • Intermediate standard-text prediction uses the text paired with pseudo-dialect speech as an explicit output before the downstream task.The authors describe this as mapping dialect audio into a semantic space for backend LLM reasoning.Source: §I · p. 1
  • The evaluation measures dialect understanding through dialect-to-English translation across Japanese, German, and Chinese, and tests augmentation with end-to-end and cascade models on Japanese data.The authors report that pseudo-dialect augmentation benefited both direct Whisper and Whisper+LLM over synthetic standard speech baselines.Source: §I · p. 1
  • Pseudo-dialect augmentation raised English-translation BLEU over synthetic standard-speech baselines for Japanese (25.38 to 26.24) and German (31.57 to 32.47); intermediate standard-text prediction raised Japanese to 28.26 and Chinese from 11.67 to 16.37.Pseudo-dialect speech augmentation also improved direct Whisper from 18.44 to 18.98 and the cascade system from 29.65 to 30.44 over their respective synthetic standard-speech baselines.Source: §I · p. 1

II. PROPOSED METHOD

The method creates pseudo-labeled speech-translation examples from standard speech-text data, then augments them with pseudo-dialect speech synthesized from LLM-generated dialect text using standard TTS.

II. PROPOSED METHOD

  • The framework starts from standard speech-text pairs and a translation model to produce pseudo-labeled speech-translation data, addressing the scarcity of dialect speech inputs.Figure 1 depicts LLM-generated dialect text and standard TTS producing pseudo-dialect speech for single-task and multi-task SLM training.Source: §II · pp. 1–2, Figure 1 · p. 2

A. Pseudo-Dialect Speech Augmentation

  • An LLM-based dialect generation model rewrites standard text as pseudo-dialect text, which a standard-language TTS model converts into pseudo-dialect speech.The synthesized speech is paired with the pseudo translation label for SLM training.Source: A. Pseudo-Dialect Speech Augmentation · p. 2
  • Although standard-trained TTS produces typically non-native-sounding dialect speech, its dialectal lexical and syntactic cues provide supervision for dialect robustness.The input transcript and translation label are retained alongside pseudo-dialect speech in the multi-task training format.Source: A. Pseudo-Dialect Speech Augmentation · p. 2, Figure 1 · p. 2

B. Intermediate Standard-Text Prediction for SLM

The model uses standard text as an intermediate representation between dialect speech and translation, then jointly trains on pseudo-dialect and original standard data.

B. Intermediate Standard-Text Prediction for SLM

  • Standard text is used as an intermediate representation that maps dialect audio into a semantic space before the final translation.The authors describe this step as semantic denoising that bridges dialect input and the target translation.Source: B. Intermediate Standard-Text… · p. 2
  • The model approximates marginalization over standard-text sequences by decoding the most probable sequence before producing the final translation.This forms a chain in which standard text is predicted as a semantic bridge, followed by the translation.Source: B. Intermediate Standard-Text… · p. 2
  • Training jointly uses pseudo-dialect triplets and original standard-data triplets, each containing speech, standard text, and translation.The pseudo-dialect triplet is (˜xd,ts,ys), while the original-data triplet is (xs,ts,ys).Source: B. Intermediate Standard-Text… · p. 2

III. EXPERIMENTAL SETUP

The experiments assess dialect-to-English translation from real Japanese, Chinese, and German dialect speech using BLEU, with SLM, end-to-end, and cascade-model evaluations.

III. EXPERIMENTAL SETUP

  • Experiments cover Japanese, Chinese, and German to test the approach across languages with differing dialect-to-standard-language relationships.Japanese data is also used to test augmentation with end-to-end Whisper and Whisper+LLM cascade models.Source: §III · p. 2

A. Evaluation Tasks

  • Dialect understanding is evaluated by translating real dialect speech into English, with BLEU scores computed using SacreBLEU.The task tests whether the model understands the underlying semantics of dialect speech.Source: A. Evaluation Tasks · p. 2

B. Model and Training Setup

  • The SLM connects a speech encoder to an LLM through a projector; Whisper large-v3 is used across languages, and only projector parameters are updated.The speech encoder and LLM remain frozen during training.Source: B. Model and Training Setup · p. 2
  • Table I describes output formats for each multi-task configuration, with the transcription target determined by the input speech.The target is pseudo-dialect text or standard text, depending on the input.Source: Table I · p. 3
  • Training runs for 10 epochs on eight NVIDIA H100 GPUs, with effective batch size 512 and peak learning rate 1e-4.Training time varies by language and is longer when pseudo-dialect augmentation is used.Source: B. Model and Training Setup · p. 3

C. Datasets

Evaluation uses real dialect speech from Japanese, German, and Chinese corpora, while training and augmentation draw on standard-language speech and text. English labels for selected training datasets were generated with a different LLM from the one used for evaluation references.

1) Evaluation Datasets:

  • Evaluation covers 20 Japanese dialects in CPJD, 8 German dialects in SwissDial, and 8 Chinese dialect subsets in KeSpeech, with real dialect speech excluded from training.English reference labels were generated from standard-language parallel texts using GPT-4o.Source: 1) Evaluation Datasets · p. 3

2) Training Datasets:

  • Training data use ReazonSpeech large v2 (about 2.6M utterances) for Japanese, Multilingual LibriSpeech (about 0.4M) for German, and 3M WenetSpeech utterances for Chinese.Japanese training also includes Speech-BSD and CoVoST2 standard speech with human-annotated Japanese transcripts and English translations.Source: 2) Training Datasets · p. 3
  • Qwen2.5-32B-Instruct generated English labels for datasets lacking translations, distinct from GPT-4o evaluation references to limit reliance on one translation system's stylistic biases.The choice follows previous work.Source: 2) Training Datasets · p. 3

D. Experimental Configurations

The experiments compare no augmentation, synthetic standard speech, and pseudo-dialect speech. The synthetic conditions match added data volume, while pseudo-dialect text is rewritten across a range of regional dialects.

1) Data Augmentation Configurations:

  • All three settings are compared as standard single-task models.Source: 1) Data Augmentation Configurations · p. 3
  • The no-augmentation baseline uses only real speech: approximately 2.6M Japanese, 0.4M German, and 3.0M Chinese utterances.Source: 1) Data Augmentation Configurations · p. 3
  • Synthetic standard speech doubles the baseline training volume with a 1:1 match of synthesized utterances, isolating data-volume effects from dialectal linguistic features.Totals are 5.2M Japanese, 0.8M German, and 6.0M Chinese utterances.Source: 1) Data Augmentation Configurations · p. 3
  • Pseudo-dialect augmentation rewrites standard transcripts into randomly selected dialects—20 Japanese, 8 German, and 8 Chinese—then synthesizes them using the same TTS setup as the standard-speech condition.Its total training volume exactly matches the synthetic standard speech setting.Source: 1) Data Augmentation Configurations · p. 3

2) Multi-task Learning Configurations:

Multi-task models are trained on the full pseudo-dialect-augmented dataset with target formats that may include intermediate predictions. At evaluation, the English translation is extracted from tagged output.

2) Multi-task Learning Configurations:

  • Evaluation extracts the final English translation from text enclosed by <en> and </en> tags.Source: 2) Multi-task Learning Configurations · p. 3
  • The multi-task training configurations use concatenated target sequences, optimizing their joint probability through standard autoregressive next-token cross-entropy loss.Source: 2) Multi-task Learning Configurations · p. 3
  • Table II compares dialect-to-English translation BLEU across languages, augmentation settings, and output formats; “ASR” and “Standard” denote intermediate transcription and standard-text prediction.Source: Table II · p. 4
  • Tables III and IV report average English-translation BLEU across augmentation types and intermediate-prediction configurations, with Table IV also covering 20 individual results.Source: Table IV · p. 4, Table III · p. 4

IV. EXPERIMENTAL RESULTS

Dialect-to-English results vary by language and intervention: Japanese benefits from both approaches, German from pseudo-dialect augmentation, and Chinese from intermediate standard-text prediction. Japanese dialect-level gains are broad, while Chinese pseudo-dialect speech and additional intermediate predictions expose limitations.

IV. EXPERIMENTAL RESULTS

  • The study uses paired bootstrap resampling with 1,000 iterations (p < 0.05), but Japanese dialect subsets contain only about 250 utterances, so non-significant results may reflect low power.Source: §IV · p. 4
  • Japanese rose from 24.21 to 28.26; German pseudo-dialect augmentation beat synthetic standard speech by +0.90 BLEU, while Chinese intermediate prediction rose from 11.67 to 16.37 (+4.70 BLEU).German multi-task learning was not effective, while Chinese pseudo-dialect augmentation showed no noticeable effect.Source: §IV · p. 4
  • Chinese dialect character forms do not reflect phonetic variation, and standard TTS failed to synthesize distinct dialectal speech, potentially limiting audio-augmentation benefits.ASR+Standard+English also degraded performance across all three languages versus Standard+English, likely due to amplified error propagation from two intermediate predictions.Source: §IV · p. 5
  • Pseudo-dialect augmentation raised Japanese average BLEU to 26.24 versus 25.38 for synthetic standard speech, with gains over both baselines in 15 of 20 dialects.Four dialects showed statistically significant gains (p < 0.05); the average comparison was significant (p < 0.001).Source: §IV · p. 5
  • Standard+English significantly outperformed ASR+English and improved performance over direct English translation in 19 of 20 Japanese dialects.Nine dialects showed statistically significant improvements (p < 0.05); ASR+English had larger fluctuations in Hiroshima, Izumo, and Tsugaru.Source: §IV · p. 5

V. ANALYSIS

Japanese examples illustrate how pseudo-dialect speech helps the model recognize dialect-specific grammar and vocabulary. The analysis also considers the method’s benefits and limitations.

A. Improvement by Pseudo-Dialect Speech

B. Challenges of Pseudo-Dialect Speech

The analysis identifies lexical and acoustic challenges, shows how standard Japanese can help bridge dialect meaning to English, and tests pseudo-dialect augmentation across architectures.

B. Challenges of Pseudo-Dialect Speech

C. Improvement by Multi-task Learning

  • For the Tsugaru input, the standard-Japanese-output model captured the core meaning and produced the most contextually accurate English translation among the compared models.Source: C. Improvement by Multi-task Learning · p. 6

VI. EFFECTIVENESS ACROSS DIFFERENT MODEL ARCHITECTURES

  • Because trainable parameter counts differ substantially across architectures, the comparison focuses on augmentation effects within each architecture, not absolute performance between them.Source: §VI · p. 6
  • BLEU gains over synthetic standard speech were +0.54 for Whisper and +0.79 for the cascade system (p < 0.001).Source: Table VII · p. 6, §VI · p. 6

VII. CONCLUSION

The paper concludes that pseudo-dialect speech combined with standard-text multi-task learning improves dialect understanding across languages and model architectures. It also identifies translation quality and realistic dialect synthesis as areas for further work.

VII. CONCLUSION

  • The proposed approach combines pseudo-dialect speech synthesized without real dialect speech with multi-task learning using corresponding standard-language text.Source: §VII · p. 6
  • Experiments showed improved dialect understanding across multiple languages and alternative model architectures.Source: §VII · p. 6
  • Future work includes evaluation protocols for low-resource dialect text translation and speech synthesis that reproduces realistic dialectal features under low-resource constraints.Source: §VII · p. 6
Loading 2610.09321v1…