Long-Prompt Zero-Shot TTS: Benchmarking Prosody Reproduction Beyond Voice Cloning

1. INTRODUCTION

• Long-Prompt Zero-Shot TTS uses minutes to tens of minutes of prompt speech without target-speaker parameter updates, asking how added prompt material affects speaker-specific prosody.The motivation contrasts this with speaker adaptation, which requires extra training and configuration management.Source: §1 · p. 1
• LP-TTS-Bench evaluates speaker-specific prosody reproduction against matched ground-truth utterances, rather than prosody diversity, across 37 Chinese–English speakers.Source: §1 · p. 1
• LPTTS improves several prosodic dimensions, with gains increasing up to approximately 10 minutes of prompt speech and saturating at longer durations.The introduction identifies LPTTS as a reference system for the task and benchmark.Source: §1 · p. 1
• LPTTS combines a hierarchical Long-Prompt Encoder with Qwen3TTS, compressing prompts into global and target-text-conditioned local tokens and training with contrastive ranking and on-policy distillation.Source: §1 · p. 1
• The paper’s contributions are the new task, a multidimensional benchmark protocol, and LPTTS as an initial reference system, with analyses of prompt duration and text-speech prosody relevance.The authors state that LP-TTS-Bench, evaluation scripts, and audio samples will be released upon acceptance.Source: §1 · p. 1

2.1. Task Definition and Data

• The task synthesizes target text from a speaker’s prompt corpus at inference time, without target-speaker parameter updates, to study how prompt amount affects prosody reproduction.Source: §2.1 · p. 1
• The benchmark contains 17 Chinese and 20 English speakers, 63.95 prompt hours, and 2,728 held-out test utterances paired with ground-truth recordings.Each speaker contributes 57–80 held-out test utterances longer than five seconds.Source: §2.1 · pp. 1–2

2.2. Evaluation Protocol

• Evaluation compares each synthesis with the matched recording, using WER/CER for intelligibility, SIM-O for speaker similarity, and SRD for speaking-rate distance.SRD uses text-unit count and speech duration including internal pauses but excluding leading and trailing silence.Source: §2.2 · p. 2
• The protocol measures pause placement and duration with PD, prominence with PCC, intonation with IC, and clause-level voiced-pitch range with PRD.PD penalizes missing, misplaced, and mismatched-duration pauses; IC compares aligned, normalized semitone contours and their first-order differences.Source: §2.2 · p. 2
• MPM-SIM and AutoPCP provide model-based prosody measures; alignment-based metrics use a shared subset with utterance-level WER/CER below 0.1 across systems.Source: §2.2 · p. 2
• Table 1 reports WER/CER in percent; its arrows mark preferred directions, checkmarks indicate target-speaker fine-tuning, and formatting marks within-language best and second-best results.Source: Table 1 · p. 3

3. LPTTS SYSTEM

• The system overview combines the hierarchical Long-Prompt Encoder with gated integration into the Qwen3TTS backbone.The encoder is built on Qwen3TTS.Source: §3 · p. 3

3.1. Long-Prompt Encoder Design

• The encoder turns prompts of variable duration into 32 conditioning vectors: 8 global tokens and 24 target-text-retrieved local tokens.It processes prompts in chunks of up to 30 seconds, using semantic keys from the first RVQ codebook for local retrieval.Source: §3.1 · p. 3
• LPTTS uses Qwen3TTS’s speaker encoder for timbre and injects the prosody tokens into the language model through gated cross-attention.The modules use hidden states as queries and the 32 conditioning vectors as keys and values, at layers 16, 20, 24, and 26.Source: §3.1 · p. 3

3.2. Training Objectives

• Training combines speech-generation cross-entropy with a same-speaker ranking objective that contrasts prompts while holding target text and speaker embedding fixed.The target utterance and its surrounding segment are excluded from the positive prompt to prevent direct acoustic or textual overlap.Source: §3.2 · p. 3
• On-policy distillation trains the encoder and injection modules on student-generated trajectories while keeping the TTS backbone frozen.A Qwen3TTS teacher fine-tuned on an expressive speaker supervises the student using next-token Jensen–Shannon divergence.Source: §3.2 · p. 3

4.1. Experimental Setup

• The benchmark speakers are disjoint from encoder-training data and the OPD speaker, and test utterances are excluded from prompts; comparisons use shared test texts and matched ground-truth recordings.The baselines include Qwen3TTS-Base with a 10-second prompt, Qwen3TTS-SFT, and LPTTS variants with 10-minute prompts; P-MOS uses blind ratings from three listeners.Source: §4.1 · pp. 3–4

4.2. Main Results

• LPTTS-Base has broadly similar explicit prosodic performance to Qwen3TTS-Base despite its longer prompt; PD and model-based prosody metrics improve, while SRD, PCC, and IC change little or inconsistently.The authors suggest encoder training processes and integrates long-prompt representations, but useful prosodic extraction remains limited.Source: §4.2 · p. 4
• Qwen3TTS-SFT improves every prosody-related metric over Qwen3TTS-Base, while WER/CER rises from 0.73 to 1.78 in Chinese and from 1.32 to 1.93 in English.The authors note that speaker adaptation also requires separate training for each speaker.Source: §4.2 · p. 4
• On six of eight prosody metrics in each language, LPTTS scores better than Qwen3TTS-SFT, with lower WER/CER and comparable speaker similarity, without target-speaker fine-tuning.The results position LPTTS as a single-model alternative to per-speaker adaptation.Source: §4.2 · p. 4

4.3. Effect of Prompt Length

• As prompts increase from 30 seconds to 30 minutes, speech-rate, prominence, intonation, and MPM-SIM metrics generally peak around 10 minutes, while SIM-O remains approximately stable.The gains are non-monotonic and tend to saturate beyond 10 minutes; the figure shows Chinese results, with English trends described as similar.Source: §4.3 · p. 4, Figure 3 · p. 4

4.4. Text–Prosody Relationship Analysis

• Textually similar prompts have lower pitch, energy, and timing/pause distances than random prompts, with an overall Closer Rate of 62.6%.All feature-group differences have 95% confidence intervals excluding zero, with p < 0.001 for all comparisons.Source: §4.4 · p. 4, Table 2 · p. 4

• The paper formulates Long-Prompt Zero-Shot TTS and introduces LP-TTS-Bench, a bilingual 37-speaker benchmark for multidimensional evaluation of speaker-specific prosody reproduction.Source: §5 · p. 4
• With LPTTS, longer prompts, particularly up to 10 minutes, improve most prosody metrics over short-prompt conditioning, while further increases bring diminishing or non-monotonic gains.Source: §5 · p. 4
• The findings support evaluating speaker-specific prosody beyond conventional intelligibility and speaker-similarity measures, with LP-TTS-Bench serving as a testbed for future long-context prosody modeling.Source: §5 · p. 4

• Participants provided informed consent, no personally identifiable information was collected, and the study followed applicable institutional guidelines.Source: §6 · p. 5