Source-linked AI summary
T5Gemma-TTS Technical Report
Chihiro Arata, Kiyoshi Kurihara
TL;DR
Decoder-only codec language models can lose text-conditioning strength as audio sequences grow, motivating a persistent conditioning mechanism for long-utterance zero-shot TTS. T5Gemma-TTS uses a pretrained multilingual encoder-decoder with PM-RoPE across decoder cross-attention, achieving strong Japanese and cross-lingual speaker-similarity results while exposing important evaluation boundaries.
Problem
Decoder-only architectures prepend text to the growing audio sequence, causing text conditioning to weaken during long utterances.
Method
T5Gemma-TTS routes bidirectional T5Gemma text representations through all 26 decoder cross-attention layers and applies PM-RoPE while processing subwords directly.
Results
0.677 Japanese SIM versus XTTS v2’s 0.622 is statistically significant, while Korean SIM reaches 0.747 despite Korean being absent from training.
Takeaways & Limitations
The results support combining pretrained multilingual encoder-decoder modeling with PM-RoPE for zero-shot voice cloning and duration-controlled speech generation.
Takeaways & Limitations
LibriSpeech results are upper-bound estimates because LibriHeavy is a superset, and disabling PM-RoPE at inference is not a controlled training ablation.
Abstract
from arXiv · showhide
Autoregressive neural codec language models have shown strong zero-shot voice cloning ability, but decoder-only architectures treat input text as a prefix that competes with the growing audio sequence for positional capacity, weakening text conditioning over long utterances. We present T5Gemma-TTS, an encoder-decoder codec language model that maintains persistent text conditioning by routing bidirectional text representations through cross-attention at every decoder layer. Built on the T5Gemma pretrained encoder-decoder backbone (2B encoder + 2B decoder; 4B parameters), it inherits rich linguistic knowledge without phoneme conversion and processes text directly at the subword level. To improve duration control, we introduce Progress-Monitoring Rotary Position Embedding (PM-RoPE) in all 26 cross-attention layers, injecting normalized progress signals that help the decoder track target speech length. Trained on 170,000 hours of multilingual speech in English, Chinese, and Japanese, T5Gemma-TTS achieves a statistically significant speaker-similarity gain on Japanese over XTTSv2 (0.677 vs. 0.622; non-overlapping 95% confidence intervals) and the highest numerical Korean speaker similarity (0.747) despite Korean not being included in training, although this margin over XTTSv2 (0.741) is not statistically conclusive. It also attains the lowest numerical Japanese character error rate among five baselines (0.126), though this ranking should be interpreted cautiously because of partial confidence-interval overlap with Kokoro. English results on LibriSpeech should be viewed as an upper-bound estimate because LibriHeavy is a superset of LibriSpeech. Using the same checkpoint, disabling PM-RoPE at inference causes near-complete synthesis failure: CER degrades from 0.129 to 0.982 and duration accuracy drops from 79% to 46%. Code and weights are available at https://github.com/Aratako/T5Gemma-TTS.
1 Introduction
T5Gemma-TTS addresses weakening text conditioning in long-utterance decoder-only TTS with an encoder-decoder design and integrates PM-RoPE for duration control. Its multilingual evaluation shows strong Japanese and Korean speaker similarity, while some rankings and English results require caution.
- Decoder-only TTS weakens text influence during long utterances because the growing audio sequence dilutes the fixed text prefix.
- T5Gemma-TTS uses a pretrained 2B+2B parameter T5Gemma encoder-decoder backbone, enabling multilingual subword processing without phoneme conversion.
- PM-RoPE is integrated into all 26 cross-attention layers to preserve duration-control properties in a multilingual pretrained backbone.
- 0.677 Japanese SIM versus XTTS v2’s 0.622 is statistically significant, with non-overlapping 95% confidence intervals.
- 0.747 Korean SIM is numerically highest despite Korean being absent from training, but its margin over XTTS v2 is not statistically conclusive.
- 0.126 is the lowest Japanese CER, although confidence intervals partially overlap with Kokoro, and LibriSpeech results are upper-bound estimates because LibriHeavy is a superset.
2 Related Work
Related work establishes encoder-decoder codec language models and PM-RoPE as alternatives to decoder-only and non-autoregressive TTS designs. T5Gemma-TTS combines PM-RoPE with pretrained multilingual subword modeling to extend duration control and cross-lingual evaluation without language-specific phonemizers.
- Decoder-only codec language models prepend text to audio, whereas T5Gemma-TTS injects encoder representations through cross-attention throughout generation.
- PM-RoPE tracks normalized generation progress in cross-attention, supporting reliable duration control and extrapolation beyond training speech lengths.
- T5Gemma-TTS adopts PM-RoPE without modification but replaces VoiceStar’s phoneme input with T5Gemma subwords.
- Subword input avoids language-specific phonemizers and preserves pretrained multilingual embeddings, though its effect on PM-RoPE duration control remains unabated.
- Non-autoregressive systems can offer naturalness and speed but typically provide limited explicit duration control compared with PM-RoPE-based generation.
- The paper combines PM-RoPE with a large multilingual pretrained encoder-decoder backbone to expand language coverage while retaining duration-control properties.
- Training spans English, Mandarin Chinese, and Japanese, a narrower language scope than systems such as XTTS v2 but one suited to cross-lingual generalization analysis.
3 Method
T5Gemma-TTS is an autoregressive sequence-to-sequence codec model that maps text and optional reference speech to audio tokens. Its encoder-decoder design supplies persistent text conditioning, while PM-RoPE injects generation progress to support duration control.
- T5Gemma-TTS maps text and optional reference speech waveform to discrete audio tokens with an autoregressive sequence-to-sequence architecture.
- PM-RoPE replaces standard cross-attention in all 26 decoder layers by encoding relative generation progress between decoder positions and encoder positions.The resulting attention scores align decoder progress with encoder positions, enabling stage-dependent text attention.
- At inference, the target token count is estimated as ˆS = ⌊ˆD × 50⌋, and decoder progress IDs are normalized against that target length.The target duration is estimated from reference duration and reference and target text counts, with language-specific processing and fallback rules.
- The phoneme-count duration estimator is a convenience heuristic, so estimation errors may degrade duration-related quality when language or speaking style mismatches its assumptions.
- The model is trained on approximately 170,000 hours of English, Chinese, and Japanese speech using next-token cross-entropy over audio tokens.Training data includes approximately 100,000 English, 50,000 Chinese, and 20,000 Japanese hours.
4 Experiments
The experiments evaluate zero-shot synthesis across six languages using matched reference-prompt conditions and multiple similarity, intelligibility, and naturalness measures. Results are compared with four systems, while several dataset and metric caveats constrain interpretation.
- Experimental setup: Evaluation uses 100 randomly sampled utterances per test set across six languages, with each ground-truth recording serving as the reference prompt.Reference prompts last 3–15 seconds, and all systems are evaluated under identical conditions.
- Limitations: LibriSpeech results are upper-bound estimates because LibriHeavy strictly contains LibriSpeech test-clean, allowing possible training-test overlap.The language-level in-training/out-of-training distinction is based on training languages, not specific test-corpus inclusion.
- Metrics: CER is used for Chinese, Japanese, and Korean, while WER is used for English; Korean CER provides a more granular signal for its syllabic script.
- Metrics: SIM is computed as cosine similarity between ECAPA-TDNN speaker embeddings from reference and generated waveforms, with higher values indicating greater similarity.
- Limitations: UTMOS22 scores for non-English languages are approximate naturalness proxies because the predictor was trained and validated primarily on English speech.
- Limitations: ASR-based intelligibility metrics can miss mispronunciations when the recognizer transcribes plausible alternative readings, although this limitation applies equally across systems.
- Baselines: T5Gemma-TTS is compared against F5-TTS, XTTS v2, CosyVoice 2, and Kokoro across diverse architectures.Kokoro is included for intelligibility and naturalness comparison but does not perform strict zero-shot voice cloning from arbitrary reference audio.
- Limitations: The PM-RoPE comparison switches a flag on one trained checkpoint rather than comparing separately trained models, so it does not isolate the training-time contribution.A controlled ablation would require training separate models with and without PM-RoPE.
5 Results
Across six languages, T5Gemma-TTS combines strong speaker similarity with competitive intelligibility, especially on Japanese and Korean, while revealing metric and evaluation caveats. PM-RoPE is associated with substantially better synthesis quality and duration control, though English results are upper-bound estimates and some comparisons are inconclusive.
- 0.677 ± 0.016 Japanese SIM significantly exceeds XTTS v2’s 0.622 ± 0.016, with non-overlapping 95% confidence intervals.
- 0.126 Japanese CER is numerically lowest, but partial confidence-interval overlap with Kokoro warrants cautious interpretation.
- 0.722 ± 0.017 Chinese SIM ranks second among zero-shot systems, while F5-TTS reaches 0.864 SIM alongside Chinese CER of 0.155.
- 1.138 Japanese CER indicates near-complete F5-TTS intelligibility failure, because errors exceed the number of reference characters.
- English LibriSpeech results are upper-bound estimates because LibriHeavy overlaps the training and test data, while normalized cross-language radar comparisons are qualitative.
- 0.747 ± 0.029 Korean SIM is numerically highest despite Korean being unseen in training, but overlaps XTTS v2’s 0.741 ± 0.010.
- Disabling PM-RoPE changes Japanese CER from 0.129 to 0.982 and duration accuracy from 0.79 to 0.46, with repetitive or incoherent audio.
6 Conclusion
T5Gemma-TTS combines a pretrained encoder-decoder backbone with PM-RoPE across all 26 decoder layers for multilingual zero-shot speech synthesis. It reports significant Japanese speaker-similarity gains, strong Korean similarity outside training, and severe degradation when PM-RoPE is disabled, while retaining European-language and speed limitations.
- 0.677 ± 0.016 Japanese SIM versus 0.622 ± 0.016 for XTTS v2, with non-overlapping 95% confidence intervals.
- 0.747 ± 0.029 Korean SIM was numerically highest among five competitive systems despite Korean being outside the training distribution.
- 0.126 ± 0.018 Japanese CER was numerically lowest, though its confidence interval partially overlaps Kokoro’s 0.139 ± 0.016.
- Disabling PM-RoPE changed CER from 0.129 to 0.982 and SIM from 0.666 to 0.109, causing near-complete synthesis failure.
- Higher WER on unseen French and German and an RTF of approximately 0.8–2.0 limit performance relative to specialized systems and faster baselines.
Broader Impact
The paper identifies misuse risks from zero-shot voice cloning and documents licensing responsibilities for its training data and backbone. It releases the system for academic research while rejecting rights-violating applications and encouraging safeguards.
- Zero-shot voice cloning could enable speaker impersonation, audio deepfakes, fraud, disinformation, and non-consensual voice synthesis.
- The authors recommend watermarking, speaker-consent verification, and deployment restrictions before production integration.
- Emilia permits noncommercial academic research use, while LibriHeavy derives from public-domain LibriVox recordings.
- Users are responsible for complying with upstream licenses, including the Gemma Terms of Use governing the T5Gemma backbone.
- The release targets academic research and does not endorse surveillance, harassment, or applications violating rights or applicable laws.