PaperlayerView exact revision on arXiv

Source-linked AI summary

Scaling phoneme-based TTS augmentation for ASR: A unified pipeline and controlled study

Zhen Wang, TianRui Wu, RongQi Han, Hao Wu, Wei Liang

arXiv:2608.26697v1cs.CL

TL;DR

The paper addresses limited evidence about how synthesis scale, candidate-text content, and reference quality affect TTS-based ASR augmentation. It presents a unified multilingual phoneme-based pipeline with PFGS text ranking and evaluates it across four languages and 13 test sets. Random augmentation improves over real-only training on 11 sets, while PFGS improves over real-only training on 12 and over random selection on 9 at a nominal 60% budget.

  • Problem

    The separate effects of synthesis scale, candidate-text phoneme composition, and reference quality remain insufficiently characterized under matched multilingual protocols.

  • Method

    The paper builds a unified phoneme-based TTS-to-ASR pipeline using multilingual F5-TTS with language-ID conditioning, candidate selection, reference filtering, and matched ASR continuation.

  • Results

    Across four languages and 13 test sets, random augmentation improved over real-only continuation on 11 sets, while PFGS improved over real-only training on 12 sets and over random selection on 9.

  • Takeaways & Limitations

    Within the evaluated multilingual and multidomain settings, synthesis scale, candidate-text content, and reference quality each affected the benefit of TTS augmentation.

  • Takeaways & Limitations

    The study’s outcomes are bounded by data coverage, candidate-text distributions, TTS quality, and acoustic or linguistic domain mismatch.

Abstract

from arXiv · show

Synthetic speech provides scalable supervision for automatic speech recognition (ASR), but its benefit depends on the selected texts, reference speech, and amount of synthesized data. We present a unified phoneme-based TTS-to-ASR augmentation pipeline built around a multilingual TTS model trained from scratch using the F5-TTS architecture with language-ID conditioning. The pipeline combines language-specific grapheme-to-phoneme conversion, reference-speech filtering, candidate-text selection, synthesis, and matched ASR continuation. We further propose phoneme-frequency-guided selection (PFGS), which ranks candidate sentences using phoneme frequencies estimated from real ASR training labels. Experiments with separate monolingual ASR systems for Arabic, French, Italian, and Portuguese span 13 test sets. Across the synthesis-scale sweep, random augmentation improves over matched real-only continuation on 11 test sets. Under a nominal 60% synthesis budget, PFGS improves over real-only training on 12 test sets and over random selection on 9. Its largest relative word error rate (WER) reduction against random selection is 19.3%. With target texts and synthesis counts fixed, reference-speech filtering reduces absolute WER by 0.29 and 0.59 points on Italian and French Common Voice, respectively. These results identify synthesis scale, candidate-text content, and reference quality as important control variables in TTS-based ASR augmentation.

1. INTRODUCTION

The paper studies when synthetic speech improves ASR augmentation, focusing on synthesis scale, candidate-text phoneme composition, and reference quality. It introduces a unified multilingual phoneme-based pipeline and evaluates these factors under matched protocols across four languages and 13 test sets.

  • Research gap: Synthetic speech usefulness depends on selected text, reference-speech acoustics, and synthesized-data amount.Prior studies establish that text construction affects augmentation, but the separate effects of these control variables remain insufficiently characterized under matched multilingual protocols.
  • Approach: The unified pipeline combines phoneme-based multilingual TTS, language-specific G2P, reference filtering, candidate-text selection, and matched ASR continuation.The TTS model is trained from scratch with the F5-TTS architecture and explicit language-ID conditioning.
  • Research questions: The study tests whether shared phoneme-based TTS provides useful supervision across Arabic, French, Italian, and Portuguese ASR systems and domains.It evaluates separate monolingual ASR systems under matched continuation protocols.
  • Research questions: The experiments vary synthesis ratios from 10% to 100% and compare PFGS with random and low-frequency candidate selection at a nominal 60% budget.PFGS ranks candidates using phoneme-frequency distributions estimated from real ASR training labels.
  • Study design: The controlled study separately examines synthesis scale, candidate-text selection, and reference-speech filtering across 13 test sets.This design isolates the three factors under matched ASR continuation.

2. METHOD

The method builds synthetic ASR supervision by selecting and filtering inputs before combining generated speech–transcript pairs with real ASR data. Its central design uses shared multilingual phoneme-based TTS, quality-controlled references, and phoneme-frequency-guided text ranking.

  • Overall pipeline: The five-stage pipeline covers data preparation, phoneme-based TTS training, reference quality control, synthetic-corpus control, and ASR evaluation.Figure 1 summarizes the complete procedure.
  • Overall pipeline: Candidate texts are cleaned, checked for test overlap, ranked with real-data phoneme statistics, and synthesized into ASR pairs.PFGS selects the top N_syn targets after candidate processing.
  • Multilingual TTS: The shared TTS model is trained from scratch on naturally recorded Arabic, French, Italian, and Portuguese speech using F5-TTS architecture and language-ID conditioning.The corpus preserves speaker, channel, and background variation while excluding samples with unreliable alignment, abnormal duration, or poor text quality.
  • Multilingual TTS: Phoneme representations provide a common pronunciation interface across writing systems, while language-specific normalization and G2P prepare input text.Arabic, French, and Italian use eSpeak; Portuguese uses gruut.
  • Reference control: Reference prompts are filtered by duration, word count, speaking rate, and transcript-recording consistency before conditioning synthesis.Within each language, experimental settings share prompt-pool construction, sampling rules, inference parameters, and the TTS checkpoint.
  • Candidate selection: PFGS computes candidate scores from phoneme frequencies in real ASR labels and favors sentences composed of well-represented valid phonemes.It continuously weights every valid phoneme, unlike the low-frequency control.
  • ASR augmentation: Synthetic speech–transcript pairs are combined with real ASR training data and use the same ASR objective as real pairs.The experiments cover four target languages with language-specific training protocols.

3. EXPERIMENTAL SETUP

The experiments evaluate multilingual TTS augmentation for four languages using separate ASR systems, matched continuation protocols, and controlled studies of synthesis scale, text selection, phoneme exposure, and reference-speech filtering.

  • Data and evaluation: TTS generation and ASR augmentation cover Arabic, French, Italian, Portuguese, with evaluation across public Common Voice, FLEURS, MLS, SADA, and MASC test sets.Candidate texts come from additional speech corpora excluded from the corresponding real ASR training sets.
  • ASR systems: Each language uses an independently trained WeNet-based hybrid CTC/attention Conformer, with augmented and real-only branches continuing from the same language-specific checkpoint.The branches share architecture, optimization settings, continuation interval, and decoding configuration.
  • Evaluation metric: WER measures substitutions, deletions, and insertions relative to reference words, with lower WER indicating better recognition.Primary results use attention rescoring with beam size 10.
  • Synthesis scale: The synthesis-scale experiment samples candidate texts randomly and compares ASR performance as synthetic utterances are added relative to matched real-only continuation.r = 0% denotes matched real-only continuation; for r > 0, target texts are sampled randomly from each language’s candidate set.
  • Candidate selection: The selection experiment uses a nominal 60% utterance budget to compare real-only, random, low-frequency, and PFGS conditions.Random selection is uniform, while the low-frequency control ranks candidates using occurrences of the least frequent valid phonemes.
  • Phoneme exposure: French selection results are diagnostically analyzed with phoneme n-gram exposure computed from mixed-training transcripts and test transcript sets.The analysis considers bigrams and trigrams and reports differences in exposure and WER between PFGS and random selection.
  • Reference quality: Reference-speech filtering is ablated on French and Italian Common Voice using identical PFGS target-text lists and synthesis counts.The filtered and unfiltered conditions differ only in reference-speech filtering.

4. RESULTS AND ANALYSIS

Across the controlled experiments, synthetic-data scale, phoneme-aware text selection, and reference-speech filtering each affected ASR performance, though gains varied by language and test domain.

  • 4.1. Effect of Synthesis Scale under Random Text Selection: 11 of 13 test sets improved over matched real-only continuation at one or more random augmentation scales.The best ratio was 100% on eight sets, 60% on three, and real-only training on two.
  • 4.1. Effect of Synthesis Scale under Random Text Selection: At 100% synthesis, WER decreased by 9.03, 7.36, and 6.32 absolute points on Portuguese Common Voice, FLEURS, and MLS, respectively.Arabic Common Voice and Italian MLS did not improve at any augmentation ratio.
  • 4.2. Comparison of PFGS and Control Strategies: Under a nominal 60% budget, PFGS outperformed random selection on nine of 13 test sets, with relative WER reductions of 0.6–19.3%.PFGS also outperformed real-only training on 12 sets and low-frequency selection in eight of ten available comparisons.
  • 4.2. Comparison of PFGS and Control Strategies: French PFGS reduced phoneme entropy from 5.2220 to 5.0891 bits and increased the five-most-frequent-phoneme share from 31.90% to 35.46% versus Random-60.It increased bigram and trigram exposure on all three French test sets while reducing WER.
  • 4.2. Comparison of PFGS and Control Strategies: PFGS improved more evaluated conditions overall, but gains were not uniform across languages and test sets.Variation may reflect real-data coverage, candidate-text distributions, TTS quality, and acoustic or linguistic domain mismatch.
  • 4.3. Ablation of Reference-Speech Filtering: With target texts and synthesis counts fixed, filtering reduced WER by 0.29 points for Italian and 0.59 points for French Common Voice.The corresponding WER changes were 11.45 to 11.16 for Italian and 11.70 to 11.11 for French.

5. CONCLUSION

The paper presents a unified multilingual phoneme-based TTS-to-ASR augmentation pipeline and evaluates how synthesis scale, candidate-text content, and reference quality affect its benefits.

  • The pipeline combines language-ID conditioning, language-specific G2P, candidate-text selection, and reference-speech filtering around a multilingual F5-TTS model.
  • Across four languages and 13 test sets, random augmentation improved over real-only continuation on 11 sets at one or more scales.
  • At a nominal 60% budget, PFGS improved over real-only training on 12 sets and over random selection on nine, with a largest relative WER reduction of 19.3%.
  • Within the evaluated multilingual and multidomain settings, synthesis scale, candidate-text content, and reference quality each affected TTS augmentation benefit.
Loading 2608.26697v1…