PaperlayerView exact revision on arXiv

Source-linked AI summary

Building and Evaluating Fixed-Voice Thai TTS from Synthetic Speech

Kunat Pipatanakul, Potsawee Manakul, Warit Sirichotedumrong, Sittipong Sripaisarnmongkol, Pakorn Nathong, Phatrasek Jirabovonvisut

arXiv:2609.03502v1cs.CLcs.AI

TL;DR

Low-resource TTS deployment usually trades costly reference-conditioned voice cloning against compact fixed-voice systems requiring speaker-specific data. This paper uses a large voice-cloning model as a synthetic data source for a compact Thai student, studying pipeline and evaluation choices. The resulting model supports on-device Thai TTS, while teacher sampling reveals remaining coverage limitations.

  • Problem

    Low-resource TTS lacks a well-supported route combining compact fixed-voice deployment with a short reference instead of a speaker-specific corpus.

  • Method

    The study generates synthetic speech from a short voice reference, applies text preparation, quality filtering, rejection sampling, and frontend choices, then trains a compact fixed-voice student.

  • Results

    68.2% Challenge-Set Keyword Accuracy and 91.4% pause precision are reported for the 82M-parameter Wayu-Paxa-TTS-Edge model.

  • Takeaways & Limitations

    The model supports on-device Thai TTS without reference audio, while best-of-K sampling indicates stronger teacher targets remain possible.

  • Takeaways & Limitations

    Pause and keyword metrics remain imperfect because they rely on heuristics including Thai rule-based tokenization and phoneme conversion.

Abstract

from arXiv · show

In low-resource settings, deploying TTS typically requires choosing between a large voice-cloning model with costly inference or a compact fixed-voice system that requires a speaker-specific corpus. We study a third route: using a large voice-cloning model as a programmable data source to turn a short voice reference (e.g., 15 seconds) into a compact fixed-voice student trained entirely on synthetic speech. This setting makes pipeline design consequential: teacher errors become training targets, while filtering failed generations can reduce coverage of difficult texts. Thai further introduces challenges from ambiguous word boundaries, lexical tone, names and loanwords, numeric verbalization, and Thai-English code-switching. We study how text preparation, synthetic generation, quality filtering, rejection sampling, and frontend choices affect the resulting student, and where teacher limitations remain. We evaluate CER, Challenge-Set Keyword Accuracy, Prosody Pause Accuracy, speaker similarity, and speaking rate. The resulting 82M-parameter model, Wayu-Paxa-TTS-Edge, enables on-device Thai TTS without reference audio. It achieves 68.2% Challenge-Set Keyword Accuracy (85.5% of Gemini 3.1) and 91.4% pause precision, outperforming its OmniVoice teacher (89.9%) and reaching 94.8% of Gemini 3.1. It also achieves the lowest pause-placement error and intra-word pause rates among the three systems, and 3.7% and 1.1% CER on Thai and English, respectively. We open-source the model and evaluation framework for Thai TTS development.

1 Introduction

The paper proposes distilling a short voice reference into a compact fixed-voice Thai TTS student using synthetic speech, addressing Thai-specific pronunciation and prosody challenges. It evaluates pipeline choices, complementary metrics, and remaining teacher limitations.

  • Motivation and approach: A large voice-cloning teacher generates synthetic training speech from a short reference, avoiding the need for a speaker-specific corpus.The approach trains a compact fixed-voice student from teacher-generated speech.
  • Thai challenges: Thai TTS evaluation must address ambiguous boundaries, lexical tone, names, loanwords, numbers, and Thai–English code-switching beyond sentence-level CER.These cases can produce critical pronunciation or pause errors despite correct transcripts or low CER.
  • Evaluation and pipeline: The pipeline studies text preparation, synthetic generation, quality filtering, rejection sampling, and frontend choices across correctness, pronunciation, pauses, speaker similarity, and speaking rate.The evaluation framework separates sentence-level CER, targeted pronunciation, pause placement, speaker identity, and rate.
  • Teacher limitations: Best-of-K teacher sampling reaches 87.9% exact accuracy at K = 118, compared with the 72.8% teacher baseline.The unresolved items are concentrated on expressions underrepresented in the teacher’s training corpus.

2 From a Zero-Shot Teacher to a Fixed-Voice Student

The paper builds a fixed-voice Thai TTS student by converting prepared text into quality-controlled synthetic speech from a zero-shot teacher, then training an 82M-parameter Kokoro model. Its pipeline combines text preparation, rejection sampling, pronunciation and prosody filtering, and a Thai–English phoneme frontend.

  • Pipeline: The three-stage pipeline prepares Thai text, samples and filters teacher waveforms, then trains a Kokoro student on the resulting text–audio pairs.The teacher renders prepared text using frozen voice references, and quality filtering determines which utterances enter the synthetic corpus.
  • Text preparation: Thai text comes from broad-coverage instruction data and a keyword-synthesis pipeline targeting difficult expressions, with sentence-level chunking before teacher inference.The released model additionally includes English text from LibriTTS.
  • Text preparation: Ambiguous digits and embedded English are rewritten into pronunciation-oriented Thai or Tinglish because the multilingual teacher can misinterpret numeric and English forms.Preliminary failures included digits spoken in Chinese and English rendered with an unintended accent.
  • Quality filtering: Candidates are filtered for exact hard-token phoneme correctness, permitted pause positions, speaker-specific speaking rate, and abnormal hard-token duration.Hard tokens are out-of-vocabulary items or rare TLTK dictionary headwords; lexical tone is included in exact phoneme matching.
  • Quality filtering: 23.0% of teacher candidates are rejected, with pause-placement failures at 11.3% and pronunciation failures at 10.4% dominating the filtering outcomes.Remaining failures are re-rendered up to four times, and the best take is retained when none passes to preserve text coverage.
  • Student and frontend: The student uses an 82M-parameter Kokoro/StyleTTS2 backbone with a script-routed Thai–English frontend and requires no reference audio at inference time.TLTK handles Thai phonemization, while English can remain Latin and route through Misaki; Thai low tone adds a vocabulary entry and learned embedding.

3 Beyond CER: Evaluating Fixed-Voice Thai TTS

The paper evaluates Thai TTS beyond sentence-level CER by separately measuring targeted pronunciation and pause placement. Its challenge sets and scoring procedures account for Thai word-boundary ambiguity and pauses that transcripts can conceal.

  • Why CER Is Not Enough: CER can miss mispronounced names, code-switched expressions, or other local errors because Thai whitespace does not consistently mark word boundaries.The evaluation therefore treats targeted pronunciation as distinct from sentence-level intelligibility.
  • Pronunciation Correctness: CER measures sentence-level intelligibility, whereas Challenge-Set Keyword Accuracy isolates correctness on important local expressions.The Challenge Set contains 1,531 sentences, each with one target expression and accepted forms.
  • Pronunciation Correctness: The Challenge Set uses exact normalized-transcript matching against an expected form or authorized alternate, with results reported overall and by category.The benchmark covers five categories and is constructed from curated Thai–English terms, multiple datasets, place names, and Thai names.
  • Prosody Pause Accuracy: Prosody Pause evaluation marks acceptable rather than required pause positions, combining author-written spaces and punctuation with contextual boundaries from a text-only mask.The evaluation set contains 210 long Thai sentences, and its allowed positions form the union of these sources.
  • Prosody Pause Accuracy: Pause placement is evaluated with pause precision, pause-placement error rate, and intra-word pause rate, alongside pauses per clip.Pause precision is the primary measure because reducing pauses can mechanically improve pause-placement error rate.
  • Scoring Method: ASR transcripts can remain correct when speech contains an intra-word pause, so word positions are mapped using CTC character timestamps.In the example, OmniVoice inserts a 438 ms intra-word pause while both systems transcribe the word correctly.

3.3 Speaker Similarity and Speaking Rate

The evaluation complements pronunciation and pause metrics with speaker similarity and speaking rate. Comparisons show that sentence-level CER, targeted keyword accuracy, and pause behavior reveal distinct differences between Gemini 3.1 and OmniVoice.

  • Speaker Similarity and Speaking Rate: Speaker similarity is the mean cosine similarity between test-utterance embeddings and a target voice centroid.Embeddings are extracted with ECAPA, and the centroid uses up to 40 teacher-generated training utterances per speaker.
  • Speaker Similarity and Speaking Rate: Speaking rate is measured in tokens per voiced second as the percentage difference between each student and its own teacher reference.The measure is computed on 210 long Challenge-Set sentences.
  • Why CER Is Not Enough: 4.6% vs. 3.3% mean CER and 72.8% vs. 79.8% Challenge-Set Keyword Accuracy distinguish OmniVoice from Gemini 3.1 Flash TTS.Gemini outperforms OmniVoice on both measures, while keyword accuracy exposes difficult-expression errors that sentence-level CER averages over.
  • Prosody Comparison: Gemini 3.1 achieves higher pause precision and lower pause-placement and intra-word pause errors than OmniVoice while producing 4.44 vs. 1.79 pauses per clip.Pause frequency matters because pause-placement error rate counts whether a clip contains at least one misplaced pause.
  • Why CER Is Not Enough: Together, the results show that CER alone is insufficient because targeted pronunciation and phrasing errors may leave transcripts unchanged.Challenge-Set Keyword Accuracy and Prosody Pause Accuracy provide complementary diagnostics.

4 Building the Final Model

The final model is built by cumulatively refining synthetic-corpus construction, student initialization, and frontend processing. Resampling, pretrained initialization, and frontend changes improve different aspects of performance while exposing trade-offs between quality, coverage, and lexical realization.

  • Recipe Ablation: The cumulative recipe adds components under a matched teacher-sampling budget, retaining valid examples through filtering or resampling.The final configuration is referred to as the Thai-only model before bilingual expansion.
  • Recipe Ablation: 69.6% vs. 67.5% Keyword Accuracy, 3.5% vs. 4.0% mean CER, and 88.4% vs. 81.3% pause precision follow pause filtering at approximately 17.6 hours.Speaking-rate deviation also decreases from 12.4% to 8.2%.
  • Recipe Ablation: At fixed corpus size, the bilingual frontend preserves phrase-boundary behavior while allowing Latin spans to remain in their original form.Keyword Accuracy stays close to the pause-filtered configuration, although mean CER increases from 3.5% to 4.1%.
  • Recipe Ablation: Scaling filtered data from 17.63 to 40.58 hours slightly improves CER and reduces pause errors but lowers Keyword Accuracy and pause precision.The authors suggest repeated filtering may remove difficult examples from the training corpus.
  • Recipe Ablation: Resampling rejected candidates improves Keyword Accuracy by 2.0 points, reduces mean CER by 0.5 points, improves every pause measure, and leaves speaker similarity unchanged.The final recipe achieves the best CER and pause placement among tested configurations while retaining near-best Keyword Accuracy.
  • Student Initialization: Pretrained initialization consistently outperforms training from scratch across reported metrics, with the largest keyword gain occurring for code-switching.The code-switch gain is +43.4 points, compared with +6.1 points for names.
  • Frontend Changes: Frontend-only changes recover 1.1 Challenge-Set accuracy points with fixed acoustic-model weights, with expanded G2P handling yielding more gain than LLM verbalization.The interventions affect 7.3% and 12.3% of inputs, respectively.
  • Bilingual Model: The 82M-parameter bilingual Wayu-Paxa-TTS-Edge preserves most teacher behavior, improves Thai CER, and has the lowest reported pause-placement and intra-word pause rates among compared systems.Its main degradation is lower Challenge-Set Keyword Accuracy than OmniVoice and Gemini.

5 Teacher Analysis and the Limits of Distillation

Teacher analysis separates recoverable sampling failures from limitations imposed by the teacher’s learned distribution. Repeated sampling substantially improves exact realizations, but rare or unseen keywords remain harder to cover.

  • Recoverable Teacher Headroom: Oracle best-of-K selection raises exact accuracy from 72.8% at K = 1 to 87.9% at K = 118.Repeated sampling recovers many correct realizations that a single sample does not obtain.
  • Teacher Distribution Limits: At K = 118, coverage is 66% for unseen keywords but 98% for keywords observed 100–10k times.Keyword frequency correlates negatively with sampling difficulty at ρ = −0.437.
  • Teacher Distribution Limits: Rejection sampling expands the usable teacher distribution, but the teacher’s existing support limits how far that expansion can go.This distinction separates better selection of available realizations from missing support for difficult expressions.

6 Discussion, Limitations, and Conclusion

The paper finds that compact TTS models can remain competitive with larger proprietary systems, while evaluation metrics and remaining teacher-model gaps limit the conclusion. It also identifies heuristic dependence as an important evaluation limitation and calls for further research.

  • Smaller TTS models can remain competitive with large proprietary models despite substantially more compact architectures.
  • Some evaluation metrics may favor larger models, and a performance gap with open-source teacher models remains.
  • Pause and keyword accuracy provide useful engineering signals for corpus construction and model evaluation but remain imperfect because they rely on Thai tokenization and phoneme-conversion heuristics.
  • The authors hope the work encourages further research on Thai speech technology and languages facing similar challenges.

Ethics Statement

Synthetic voice construction can reduce costs for resource-constrained languages but also creates risks of unauthorized voice imitation. The paper therefore emphasizes provenance, consent or licensing, intended-use documentation, and safeguards.

  • Synthetic voice construction can reduce the cost of building speech systems for resource-constrained languages but can also enable unauthorized voice imitation.
  • Voice references used for training and evaluation should have clear provenance and appropriate consent or licensing.
  • Released artifacts should document intended uses, known pronunciation and accent limitations, and safeguards against impersonation.

A.1 Challenge-Set Construction

The Challenge-Set construction retains rare or orthographically irregular Thai targets and creates filtered sentences around them. Construction metadata and category-specific details accompany the resulting benchmark.

  • Rare or orthographically irregular Thai targets are retained when they are rare in the Thai National Corpus or exhibit specified spelling irregularities.These include silent-letter marks, written clusters, irregular final consonants, and linking-syllable junctures; compounds of common components are excluded unless they exhibit such irregularities.
  • After target selection, tokenizer fragments and non-words are removed, while corpus sentences are filtered by length, Thai-character share, and digit share.A sentence is authored when no suitable corpus sentence is available.
  • The remaining construction details are summarized by category in Table 12.

A.2 Prosody Pause Metric

The prosody pause metric combines segmentation, forced alignment, and code-switched or verbalized-span handling to classify pauses against allowed text positions. Its measurements are useful primarily for same-instrument comparisons because component errors and vocabulary gaps constrain accuracy.

  • Scope and residual error: The scorer combines Thai word segmentation, CTC forced alignment, and handling of code-switched and verbalized spans, so its accuracy is bounded by all three components.Segmentation determines what counts as word-internal, while character alignment contributes timing jitter.
  • Scope and residual error: The detector lacks vocabulary for digits and Latin spans, so it cannot place their spoken Thai forms and its rates are most reliable for comparisons using the same instrument.Reducing these component dependencies is left to future work.
  • Implementation: Pause-scorer constants are calibrated once on known-good and known-bad phrasing and held fixed across systems.
  • Silence detection: Internal silence detection excludes clip-boundary regions and context-appropriate consonant closures using allowances derived from approximately 7,200 within-word junctures.A glottal onset receives no allowance, and the dBFS floor is absolute rather than relative to clip loudness.
  • Artifact rejection: Detected silences are separately filtered when repeated words, impossible attribution, or unverbalized digit and Latin spans make them unattributable to phrasing.Without these removals, reported intra-word pause rates are upper bounds on the two reference systems.
  • Alignment: Alignment resamples audio to 16 kHz and snaps silence midpoints to nearby TLTK word and syllable boundaries to avoid phonotactically impossible junctures.
Loading 2609.03502v1…