PaperlayerView exact revision on arXiv

Source-linked AI summary

Complex-Text Robustness Evaluation and Failure Diagnosis for Low-Resource Multilingual Text-to-Speech

Tianlun Zuo, Ziyu Zhang, Tingzhi Mao, Zhonghua Fu, Lei Xie

arXiv:2609.11545v1cs.CLcs.SD

TL;DR

Low-resource multilingual TTS systems are difficult to evaluate for robustness on complex inputs because conventional tests emphasize regular sentences and provide limited failure diagnosis. This paper proposes a three-dimensional framework with multilingual stress tests, automatic metrics, and a training-free Text Risk Score. Complex inputs reveal model- and language-specific failures, while TRS correlates positively with content and duration errors as a low-cost pre-synthesis risk indicator.

  • Problem

    Existing low-resource multilingual TTS evaluations provide limited insight into failures on complex inputs such as numbers, dates, named entities, long sentences, code-switched expressions, and punctuation-related structures.

  • Method

    The paper evaluates content consistency, language consistency, and generation stability across four languages and complex-text categories, using automatic diagnostics and training-free TRS.

  • Results

    Complex inputs expose systematic failure patterns that ordinary short-sentence evaluation does not fully reveal, while TRS correlates positively with content and duration errors.

  • Takeaways & Limitations

    Complex-text robustness should supplement language-coverage evaluation, and TRS can support low-cost pre-synthesis risk analysis.

Abstract

from arXiv · show

Low-resource multilingual text-to-speech (TTS) systems have expanded language coverage, but their robustness under complex text inputs remains insufficiently diagnosed. Existing evaluations mainly focus on naturalness, speaker similarity, and content consistency using regular test sentences, while providing limited insight into how multilingual TTS systems fail when handling challenging inputs such as numbers, dates, named entities, long sentences, code-switched expressions, and punctuation-related structures. This paper proposes a complex-text robustness diagnosis framework for low-resource multilingual TTS. We evaluate robustness from three dimensions: content consistency, language consistency, and generation stability. A multilingual robustness testing scheme is designed for Thai, Vietnamese, Swahili, and Indonesian, covering ordinary sentences and multiple types of complex text inputs. We further introduce automatic diagnostic metrics, including character error rate, language identification accuracy, and duration abnormal rate. To support input-level risk analysis before speech generation, we propose a lightweight Text Risk Score (TRS), which estimates synthesis risk from interpretable text features without manual annotation or model training. Experiments on three representative multilingual TTS systems, including OmniVoice, VoxCPM2, and MMS-TTS, show that complex text inputs expose systematic failure patterns that are not fully reflected by ordinary short-sentence evaluation. Different systems exhibit distinct vulnerabilities in number normalization, named entity handling, long-text generation, and code-switched input processing. Furthermore, TRS shows a positive correlation with content errors and duration abnormalities, demonstrating its usefulness as a low-cost pre-synthesis indicator for complex-text risk diagnosis in low-resource multilingual TTS.

I. INTRODUCTION

Existing multilingual TTS evaluations provide limited insight into failures on complex text, especially in low-resource languages. This paper introduces a three-dimensional diagnosis framework, a multilingual test, and TRS to assess and screen such risks.

  • Motivation: Broader language coverage alone does not guarantee reliable synthesis under complex real-world inputs.The paper therefore treats complex-text robustness as an additional deployment-oriented evaluation dimension.
  • Multilingual robustness test: The test covers Thai, Vietnamese, Swahili, and Indonesian across ordinary sentences and complex categories including numbers, dates, named entities, long sentences, and punctuation-related expressions.Three representative multilingual TTS systems are evaluated with CER, LID-Acc, and DAR.
  • Problem formulation and diagnostic framework: The framework evaluates content consistency, language consistency, and generation stability for complex-text robustness.It shifts evaluation from whether a system synthesizes a language to how and where it fails on complex real-world inputs.
  • Multilingual robustness test: Complex inputs expose model- and language-specific failure patterns that ordinary sentence evaluation rarely reveals.Observed errors vary across models, languages, and text categories.
  • Text Risk Score: TRS is a training-free, interpretable score that estimates input-level synthesis risk from textual features before generation.It is positively correlated with content and duration errors but has limited predictive power for language identification errors.

A. Multilingual and Low-Resource Text-to-Speech

The paper positions complex-text robustness as a diagnosis problem beyond conventional TTS quality assessment. Its framework links controlled complex inputs to automatic failure metrics and text-level risk analysis.

  • Background: Complex-text robustness is closely tied to text-front-end processing, including normalization, pronunciation resources, grapheme-to-phoneme models, and language-specific preprocessing.These resources are often unavailable, expensive to build, or inconsistent across low-resource languages.
  • Motivation: Conventional TTS evaluation commonly emphasizes naturalness, speaker similarity, or average quality on regular sentences rather than deployment risks from heterogeneous inputs.The proposed perspective instead examines content preservation, target-language maintenance, and stable audio generation.
  • Framework: The framework tests numbers, dates, named entities, long sentences, code-switched expressions, and punctuation-related constructions because they challenge normalization, pronunciation, language control, and long-form generation.The same inputs are synthesized by OmniVoice, VoxCPM2, and MMS-TTS under identical conditions.
  • Framework: CER, LID-Acc, and DAR provide complementary diagnostics for content consistency, language consistency, and generation stability.Failure attribution combines these metrics with text categories, TRS, and manual sampling analysis.
  • Implications: The framework supports low-cost pre-deployment evaluation, model selection, and text-front-end optimization.It examines whether ordinary evaluation is adequate, how failures vary across languages and categories, and whether text complexity can serve as a prior risk indicator.

B. Multilingual Complex Text robustness Set

The robustness set covers four linguistically diverse low-resource languages using a shared taxonomy and comparable construction rules. Its six categories target practical sources of normalization, pronunciation, prosody, and generation risk.

  • Language selection: The evaluation uses Thai, Vietnamese, Swahili, and Indonesian to represent varied writing systems, linguistic characteristics, and regional conditions.The design maintains a controllable experimental scale while enabling cross-language analysis.
  • Dataset construction: All languages share the same category taxonomy, sample-count targets, length distributions, and construction rules to support comparability.The authors provide category definitions, construction rules, statistical summaries, and representative examples for replication.
  • Test categories: The six categories include ordinary short sentences, numbers and dates, named entities, long or compound sentences, code-switched or mixed-script expressions, and punctuation-related constructions.Ordinary sentences provide a reference, while the other categories target specific complex-text risks.
  • Test categories: Numbers and dates test verbalization of dates, times, prices, phone numbers, percentages, and measurement units, while named entities test pronunciation and out-of-vocabulary handling.Templates are translated or localized for each target language, and entity texts include local and international entities.
  • Dataset construction: The construction principles prioritize realistic TTS inputs, cross-language comparability, and a clear diagnostic target for each category.For example, numbers and dates target normalization risk, whereas long sentences target content preservation.

3) Diagnostic Metrics:

The diagnostic metrics quantify content consistency, language consistency, and generation stability from synthesized speech. Their definitions account for multilingual tokenization and language-specific duration variation.

  • Content consistency: CER measures content consistency by comparing normalized multilingual ASR transcriptions with reference text using character-level edit distance.CER is preferred over WER because word boundaries can be ambiguous or unstable in languages such as Thai.
  • Language consistency: LID-Acc measures whether successfully generated speech remains in the target language.A unified speech language identification model compares predicted labels with target-language labels to diagnose possible language confusion.
  • Generation stability: DAR measures abnormal duration patterns associated with omissions, repetitions, truncations, or over-length generation.It uses the ratio between generated-audio duration and normalized evaluation-reference length.
  • Generation stability: Normalized reference length prevents number and date samples from appearing duration-abnormal solely because written forms are shorter than their spoken realizations.The design is especially relevant to prices, phone numbers, times, and measurement units.
  • Generation stability: Duration abnormality is identified separately for each model-language pair using the 1.5×IQR rule on duration-to-reference-length ratios.This accounts for variation in speaking rate and duration distributions across languages and systems.

C. Text Risk Score

The Text Risk Score estimates pre-synthesis risk from interpretable text features, using normalized feature aggregation to relate input complexity to TTS failures. It supports risk grouping and correlation analysis while serving as an interpretable measure rather than an exact sample-level error predictor.

  • TRS is a training-free, low-cost input-level indicator computed from interpretable text features without manual annotations, additional training, or generated speech.It is intended to identify texts more likely to trigger content errors, language confusion, or generation instability.
  • TRS combines text length, digit ratio, punctuation ratio, mixed-script ratio, named-entity presence, and a predefined category prior into one risk score.Continuous features are normalized within each language before combination, while the category prior incorporates the robustness-test design.
  • The score encodes risks associated with long-context generation, text normalization, pause and prosody control, mixed-language input, named entities, and predefined text categories.The design assumes that complex-text failures relate to observable input properties rather than being entirely random.
  • TRS is not designed to precisely predict the synthesis error of each individual sample, but to measure text complexity for analyzing failure patterns.This scope avoids implying exact sample-level prediction or universal performance determination from the category prior.
  • Experiments rank texts within each language into low-, medium-, and high-risk tertiles, then compare CER, LID-Acc, and DAR across groups.Spearman correlations between TRS and CER, DAR, and LID error rate quantify monotonic relationships between risk and diagnostic outcomes.

A. Experimental Setup

The experiments compare OmniVoice, VoxCPM2, and MMS-TTS on the same multilingual complex-text robustness setting. The design spans four low-resource languages, controlled inputs and inference protocols, and distinct multilingual TTS modeling routes.

  • Experimental Setup: The test contains 960 input texts across Thai, Vietnamese, Swahili, and Indonesian, producing 2880 synthesis tasks for three systems.Each language contributes 240 texts, with 40 samples in each of six text categories.
  • Experimental Setup: Each model uses the same input texts, synthesizes each once, and receives no additional training, finetuning, speaker adaptation, voice cloning, or task-specific prompt engineering.Generated audio is directly used for automatic diagnosis without repeated sampling or manual selection.
  • Evaluated TTS Systems: OmniVoice represents a massive multilingual zero-shot, speech-foundation-model-style route, while VoxCPM2 represents tokenizer-free multilingual generation.VoxCPM2 is described as a 2B-parameter model supporting 30 languages.
  • Evaluated TTS Systems: MMS-TTS serves as a broad-coverage multilingual neural TTS baseline rather than a language-model-based TTS system.It uses language-specific checkpoints for the evaluated languages.
  • Evaluated TTS Systems: The study evaluates OmniVoice, VoxCPM2, and MMS-TTS under the same languages, text categories, inference protocols, and automatic diagnostic metrics.This enables controlled comparison of their failure patterns under complex input conditions.

C. Evaluation Tools and Preprocessing

The evaluation pipeline standardizes generated audio and separates content, language, and stability diagnostics. It uses robust reference matching and consistent preprocessing to support comparisons across systems, languages, and text categories.

  • Audio outputs are resampled to 16 kHz, converted to mono, and screened for missing, invalid, or extremely short files.
  • CER uses forced-target Whisper decoding for content evaluation and language-agnostic decoding for independent LID-Acc measurement.
  • Number/date samples use format-robust reference matching, while normalization is applied consistently before edit-distance computation.
  • DAR uses normalized reference length and reports the proportion of duration-abnormal samples among valid audio outputs.
  • CER, LID-Acc, and DAR diagnose content consistency, language consistency, and generation stability, respectively.

B. Effect of Complex Text Categories

Complex text categories expose distinct robustness risks that ordinary sentences obscure. Numbers and dates, mixed-script inputs, and model-specific behaviors reveal that content and duration failures vary substantially across systems.

  • Numbers and dates are the hardest category, reaching 23.40% CER despite format-robust reference matching.
  • Mixed-script expressions reach 17.77% CER, while long sentences reach only 2.06%, showing that risky tokens matter more than length alone.
  • Numbers and dates have the highest DAR at 17.92%, while punctuation-related inputs also show elevated duration abnormality.
  • Model-specific Failure Patterns: OmniVoice records 20.57% number/date CER and relatively high DAR on ordinary and numerical inputs.
  • Model-specific Failure Patterns: VoxCPM2 reaches 32.09% CER on mixed-script expressions, whereas its generally lower DAR indicates predominantly content-related failures.
  • Model-specific Failure Patterns: MMS-TTS shows the highest CER on several complex categories and reaches 23.75% DAR for number/date inputs.
  • Model-specific Failure Patterns: Category-level diagnosis is necessary because models can perform well on ordinary sentences while failing on specific high-risk categories.

D. Language-specific Robustness Analysis

Robustness varies across languages and reflects both linguistic properties and evaluation-tool reliability. Vietnamese combines perfect language identification with substantial content and duration errors, while Indonesian shows the reverse trade-off.

  • Vietnamese has 15.10% CER and 100.00% LID-Acc, but its 9.17% DAR shows language consistency does not guarantee content or stability.
  • Thai has the highest CER at 14.72%, associated with non-Latin writing, limited whitespace segmentation, and difficult ASR-based evaluation.
  • Indonesian achieves the lowest CER at 5.36% but only 93.33% LID-Acc, indicating a different robustness profile.
  • Language-level conclusions should be interpreted alongside category-level results and representative failure cases because tool reliability also affects measured robustness.

E. Effectiveness of Text Risk Score

TRS is evaluated as a coarse pre-synthesis risk indicator by comparing risk groups and correlating scores with content, duration, and language-consistency failures. It tracks content errors most consistently, while its relationships with duration abnormalities are weaker and language-identification errors show no clear monotonic trend.

  • The high-risk TRS group reaches 16.89% CER, compared with 8.80% for low-risk and 9.50% for medium-risk texts.Results are averaged over all models and languages.
  • The high-risk group has a DAR of 11.43%, but the TRS-DAR relationship is weaker and model-dependent.Duration abnormality can also be affected by repeated or over-length outputs.
  • LID-Err shows no clear monotonic trend with TRS, suggesting that language-identification errors are not mainly determined by text-level complexity.Other influences may include acoustic realization, language similarity, utterance length, or limitations of the LID classifier.
  • TRS shows a positive correlation with CER across all three systems, with an average TRS-CER correlation of 0.38.Higher-risk texts tend to produce more severe content errors, indicating a moderate association between text-level risk and content-preservation difficulty.
  • TRS is intended for coarse-grained pre-synthesis screening rather than replacing ASR-based evaluation, LID analysis, or manual inspection.Its diagnostic scope is strongest for text-induced content risk and limited for language-identification errors.

APPENDIX: REPRESENTATIVE DIAGNOSTIC CASES

Representative cases clarify why category-level CER can reflect both genuine synthesis failures and artifacts from reference formatting or ASR transcription. They show distinct failure mechanisms for number/date inputs, code-switched words, named entities, and long sentences.

  • The diagnostic cases are used to clarify the sources of high CER rather than replace quantitative metrics.The appendix covers samples across models, languages, and text categories.
  • CER can be overestimated for number/date inputs when ASR transcribes spoken numbers back into written digits.This motivates format-robust reference matching in the main evaluation protocol.
  • Number/date inputs also produce genuine failures, including lost numerical information and extreme repeated outputs that can push CER above 100%.Such cases may reflect generation instability or ASR transcription behavior rather than ordinary pronunciation errors.
  • Code-switched words may be confused with acoustically similar English expressions, while named entities may be replaced by unrelated words.Examples include “newsletter” becoming “next time,” “Elon Musk” becoming “Trump,” and “Starlink” becoming “Stalin.”
Loading 2609.11545v1…