PaperlayerView exact revision on arXiv

Source-linked AI summary

What Makes a Good Audio Prompt for Zero-Shot TTS? Learning Prompt Selection from Synthesis Outcomes

Arnon Turetzky, Yossi Adi

arXiv:2610.40372v1cs.SD

TL;DR

In practical zero-shot TTS, users must choose a short voice prompt from a longer recording, and predicted-quality scores may not identify which prompt yields clear speech. The paper learns an audio-only prompt selector from synthesis outcomes. It improves held-out intelligibility over random and quality-based choices, approaching selection through test-sentence synthesis.

  • Problem

    Users must choose a short prompt from a speaker’s longer recording, while it remains unclear whether quality scores rank candidate prompts by the speech a TTS model produces.

    Source: §1 · p. 1

  • Method

    A small network learns from a TTS model’s synthesis outcomes to rank a speaker’s candidate prompts from their audio alone, without synthesizing at selection time.

    Source: §1 · p. 2

  • Results

    The audio selector lowers held-out WER by one third to two fifths versus a random prompt, exceeding quality-based selection and approaching a dozen test-sentence syntheses per candidate.

    Source: §1 · p. 2

  • Takeaways & Limitations

    For zero-shot TTS users, choosing a prompt by the speech the model produces rather than by how clean it sounds is worthwhile, and a small selector makes this cheap.

    Source: §9 · p. 8

    Evaluation covers four TTS models on English YouTube recordings and noisy LibriSpeech; other languages and recording conditions remain open, and each model’s selector requires a one-time labeling run of hundreds of thousands of syntheses.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Zero-shot text-to-speech (TTS) models clone a target voice from a short reference clip, commonly referred to as the prompt. In practice, the prompt has to be cut from a longer recording of the speaker, so which part to use must be chosen. To understand what makes a good prompt, we analyze hundreds of thousands of outputs synthesized by four TTS models from candidate prompts of YouTube recordings. The choice of prompt substantially changes intelligibility, naturalness and speaker similarity. Standard predicted-quality scores capture mainly the naturalness part but miss most of the intelligibility gain. We find that the models largely agree on which prompt yields the highest measured speaker similarity, but much less on which prompt yields intelligible speech. We propose a small network that learns from the outputs a TTS model produced to rank a recording's candidate prompts from their audio alone. Without synthesizing anything at selection time, it picks prompts that lower the word error rate on held-out recordings by a third (Qwen3-TTS) to two fifths (F5-TTS) relative to a randomly chosen prompt, more than predicted quality achieves and nearly as much as synthesizing a dozen test sentences from every candidate first.

1 Introduction

In practice, users must choose a short prompt from a speaker’s longer recording, but it is unclear whether predicted audio quality identifies the prompt that yields the clearest speech. The paper measures prompt effects across TTS models and learns to select prompts from audio alone.

1 Introduction

  • The choice of prompt affects intelligibility, naturalness, and speaker similarity, while predicted-quality selection mainly captures naturalness and misses most intelligibility gains.The study synthesizes outputs from candidate prompts in over 1,500 YouTube recordings using four TTS models.Source: §1 · pp. 1–2
  • A small network ranks candidate prompts from their audio alone, learning from the quality of speech synthesized from other prompts and selecting without synthesis at selection time.On held-out recordings, it lowers WER by one third to two fifths relative to a random prompt, exceeding predicted-quality selection and approaching a dozen test-sentence syntheses per candidate.Source: §1 · p. 2
  • TTS models largely agree on prompts yielding the highest measured speaker similarity but agree much less on prompts yielding intelligible speech.The paper frames prompt selection as choosing the candidate that leads to the best speech for a speaker.Source: §1 · p. 2

2 Related Work

Prior prompt-selection work has focused on matching style or emotion, screening for quality, or selecting from labeled databases. This paper instead asks which prompt cut from one recording makes a fixed TTS model produce good speech for unseen sentences.

2 Related Work

  • Prior methods retrieve prompts by text or style match, screen for expressiveness or quality, select using annotated labels, or filter in-the-wild prompts with model scores.Some work enhances noisy prompts; another approach synthesizes multiple outputs and keeps the best, multiplying the cost of every sentence.Source: §2 · p. 2
  • The paper selects once among prompts cut from the same recording, using downstream synthesis outcomes to learn which prompt to use for sentences not known in advance.This connects prompt selection to outcome-based example selection in in-context learning, while targeting speech synthesis rather than text or speech-recognition examples.Source: §2 · p. 2
  • Non-intrusive audio predictors are used to filter TTS training data and evaluate systems, but the paper examines where they fall short as prompt-selection rules.The authors note that a predictor can flatter a rule when it also judges the selected output.Source: §2 · p. 2

3 Setup

The task is to select one candidate prompt per recording before synthesis, then judge that prompt by the average quality of the speech it produces. The study uses disjoint recording splits and evaluates multiple zero-shot TTS models and speech outcomes.

3 Setup

  • A selector chooses one prompt before synthesis and without seeing the future target sentences; its gain is measured against the mean outcome of a randomly chosen candidate.The chosen prompt is then used for any sentence, and its outcome averages the outputs synthesized from it.Source: §3 · p. 2
  • Test results for Qwen3-TTS 0.6B cover 400 held-out recordings and all 19 candidates per recording, with improvements defined relative to a random candidate.The table caption identifies Audiobox as the held-out judge and marks comparisons with the UTMOS rule using interval-based †/‡ indicators.Source: Table 1 · p. 3
  • The study cuts 19 prompts of 3, 5, or 10 seconds from a 30-second stretch of each English YouTube recording and uses separate target sentences and anchor utterances.Candidate prompts compete within a recording, and splits prevent recording overlap between training, development, and test sets.Source: §3 · p. 3
  • Four open zero-shot TTS models are evaluated using WER and failure rate for intelligibility, three metrics for naturalness, and embedding cosine for speaker similarity.Each prompt is used for five target sentences with three sampling seeds; the prompt outcome is the mean over 15 outputs.Source: §3 · p. 3
  • Design choices and the main comparison were fixed using training cross-validation before test scoring, and intervals are 95% bootstrap intervals over recordings.Same-voice recordings are kept together in cross-validation and resampling.Source: §3 · p. 3

4 How Much Does the Prompt Matter, and Does Predicted Quality Find Good Prompts?

Selecting among prompts can improve speech outcomes, and part of the difference is a stable property of the prompt. Predicted-quality rules mainly recover naturalness, while their apparent gains can be inflated when the same predictor also judges them.

4 How Much Does the Prompt Matter, and Does Predicted Quality Find Good Prompts?

  • Probe selection chooses using 12 outputs for four sentences and evaluates on the fifth, making it a practical reference rather than a ceiling.Because it judges each prompt from only 12 outputs, it can miss the best candidate.Source: §4 · p. 4
  • Probe selection lowers Qwen3-TTS WER by about a third and F5-TTS WER by half relative to random prompts, while improving naturalness and speaker similarity.The share of output variation attributable to the prompt is larger for similarity than intelligibility; Chatterbox is less sensitive in intelligibility, and longer prompts leave less room for selection.Source: §4 · p. 4, Table 2 · p. 4
  • UTMOS and DNSMOS selection improve naturalness but recover only part of probe selection’s intelligibility gain and leave speaker similarity unchanged.UTMOS selects a 3-second prompt for 80% of recordings, although about half of candidates are that short.Source: §4 · p. 4
  • For Qwen3-TTS, UTMOS selection captures 41% of probe selection’s UTMOS gain but only 18% of its Audiobox gain, and captures none of the F5-TTS Audiobox gain.The other models show the same pattern, so evaluating a quality rule with its own predictor can overstate its apparent benefit.Source: §4 · p. 4

5 Can Synthesis Outcomes Teach Which Prompt Is Better?

The audio selector learns to rank prompts from their audio using a TTS model’s synthesis outcomes, without synthesizing at selection time. On held-out recordings, it improves intelligibility over simple quality rules and approaches the gains of extensive probing, while naturalness trade-offs depend on the judge.

5 Can Synthesis Outcomes Teach Which Prompt Is Better?

  • The audio selector encodes each prompt with frozen Whisper-base and scores it against the recording’s other candidates; it predicts several outcomes and selects by WER rank.A descriptor ranker first learns weights for 21 prompt descriptors; both rankers use synthesis outcomes as supervision.Source: §5 · p. 5
  • Qwen3-TTS 0.6B WER fell 1.36 pp (32% relative to a random prompt), matching probe selection with 12 syntheses per candidate while the audio selector synthesizes nothing.The selector’s gain was 0.54 pp more than the UTMOS rule.Source: §5 · p. 5, Figure 3 · p. 6
  • UTMOS favors its own rule, but DNSMOS and held-out Audiobox show only a few hundredths of a MOS point of gap, and listeners did not prefer the rule’s outputs on naturalness.Because UTMOS both selects and judges the rule’s prompts, its predicted naturalness gap may overstate perceived cost.Source: §5 · pp. 5–6
  • With the UTMOS head weighted at 0.25, the selector retains a 1.33 pp WER gain and stays 0.51 pp ahead of the UTMOS rule, matching it on DNSMOS and Audiobox.Only UTMOS itself still prefers the rule; the weight was chosen on development recordings before testing.Source: Figure 2 · p. 5, §5 · p. 6

6 Where Does the Gain Come From?

The selector’s advantage is not explained by prompt length, failed outputs, or reliance on Whisper’s judgments. Ablations indicate that supervision from synthesis outcomes matters more than architectural choices or reproducing predicted quality.

6 Where Does the Gain Come From?

  • The selector’s advantage survives length balancing, excluding failed outputs, wav2vec 2.0 judging, and independent YouTube-caption transcripts; it does not reduce pitch variability more than the quality rule.Its margin over the UTMOS rule was even larger under wav2vec 2.0 WER (0.71 pp), and remained nearly unchanged for Qwen3-TTS with YouTube captions.Source: §6 · p. 6
  • Architecture changes—including pointwise loss, one rather than five networks, and candidate attention—remain within the roughly 0.1 pp spread between seeds; Whisper-base matches Whisper-large-v3.WavLM also performs as well as Whisper-base, while the general-audio CLAP encoder gains little.Source: §6 · p. 6, Table 3 · p. 6
  • Training on synthesis outcomes is the key signal: replacing it with each candidate’s own predicted quality yields about half as much gain.This comparison used the same model and the same candidates.Source: §6 · pp. 6–7

7 Do TTS Models Agree on What Makes a Good Prompt?

The selector matches or beats the strongest simple baseline for three of four TTS models, while prompt preferences vary across models. Models agree most on speaker similarity; intelligibility depends more on how each model uses the prompt.

7 Do TTS Models Agree on What Makes a Good Prompt?

  • For three of four models, the selector matches or beats the strongest simple baseline; on F5-TTS it lowers WER 43% relative to a random prompt, 1.45 pp more than the DNSMOS rule.For Chatterbox, where there is little room for selection, the selector and strongest baseline are level.Source: §7 · p. 7
  • Across all four models, the candidate judged highest in speaker similarity is largely the same (corrected correlations 0.68–0.88), whereas which candidates produce intelligible speech is largely model-specific.Source: §7 · p. 7
  • Qwen3-TTS 0.6B’s selector transfers 80% of probe-selection gain to Qwen3-TTS 1.7B and 51% to F5-TTS; an F5-TTS-trained selector transfers 77–85% to the Qwen models.Transfer follows cross-model agreement only loosely (rank correlation 0.67 across nine model pairs).Source: §7 · pp. 7–8
  • For transcript-reading models, trailing silence and Whisper transcript confidence relate to intelligibility, while neither matters for Chatterbox; no single descriptor correlates above about 0.17 for any model.Typicality of the prompt’s voice matters for Chatterbox and F5-TTS; predicted quality correlates only a little for any model.Source: Table 4 · p. 7, §7 · p. 8

8 What If Prompts Differ in Noise?

When prompts differ in noise, predicted quality favors clean prompts and improves naturalness and similarity, while the audio selector better captures intelligibility.

8 What If Prompts Differ in Noise?

  • The quality rule selects the clean prompt 79% of the time and gains the most naturalness and similarity.With noise variation, predicted quality is useful for naturalness and similarity, but not for finding intelligible prompts.Source: §8 · p. 8
  • The audio selector lowers WER by 0.42 pp more than the quality rule and fails less often, despite choosing the clean prompt only 31% of the time.Adding the UTMOS head moves the selector toward the quality rule on naturalness while retaining its intelligibility.Source: §8 · p. 8
  • At this low error rate, probe selection using a dozen outputs per candidate gains less WER than the audio selector.The audio selector was trained on Qwen3-TTS 0.6B YouTube outcomes and evaluated on Qwen3-TTS 1.7B outputs from noisy LibriSpeech candidates.Source: §8 · p. 8

9 Conclusion

The paper studies prompt choice by the speech a model produces and finds that prompt selection can improve intelligibility without synthesizing outputs at selection time. It argues that prompt variability matters for users and benchmarks, while model sensitivity to prompts may be reducible.

9 Conclusion

  • A small audio-only network comes close to selecting prompts by synthesizing a dozen test sentences from every candidate, while prompt outcomes vary by TTS model.Models agree more on prompts yielding the highest measured speaker similarity than on those yielding intelligible speech.Source: §9 · p. 8
  • For users, choosing a prompt by what the model produces rather than by cleanliness is practical with a small selector.The selector operates at inference time for a fixed model; the same outcomes could also guide training.Source: §9 · p. 8
  • Pairing each benchmark sentence with one fixed prompt partly mixes model quality with prompt luck; reporting output variation across prompts would separate them.Source: §9 · p. 8
  • Different prompt sensitivity across models suggests that developers may be able to reduce it.The paper also identifies fewer outputs per prompt, sharing outcomes across models, and using the target sentence as possible extensions.Source: §9 · pp. 8–9

Limitations

The evaluation covers a limited set of models, languages, recordings, and prompt lengths, and several aspects of prompt quality remain unresolved.

Limitations

  • The study covers four TTS models from three families, English YouTube recordings, and LibriSpeech with added noise; other languages and recording conditions remain open.Source: Limitations · p. 9
  • The selector is trained on one model’s outcomes, requiring a one-time labeling run of a few hundred thousand syntheses per model.How far fewer labels or a selector trained for another model can replace this is only partly explored.Source: Limitations · p. 9
  • Outcomes are noisy, and the 400-recording test set leaves the selector’s margin over UTMOS variable across training seeds.The listening test has seven listeners and one output per pair, too few to resolve the selector’s clarity advantage.Source: Limitations · p. 9
  • Prosody beyond pitch variability and tempo—including emphasis, expressiveness, and context-appropriate stress—was not evaluated.Source: Limitations · p. 9
  • The study fixes a 30-second recording budget and 3–10-second prompts; longer prompts and concatenated prompts are unstudied.Source: Limitations · p. 9

Ethical Considerations

The paper notes voice-cloning misuse risks and describes safeguards for research data and outputs. Its appendix specifies recording preparation, synthesis and scoring, selector training, and compute requirements.

Ethical Considerations

  • Voice cloning can enable impersonation and fraud; the study distributes no voice models, speaker audio, or synthesized speech and keeps synthesis internal.The recordings’ Creative Commons license permits reuse but does not constitute speakers’ consent to voice cloning.Source: Ethical Considerations · p. 9

A Data, Models and Training

  • YODAS recordings are split by shard, filtered for a consistent main speaker, and limited to English; each recording uses a 30-second stretch dominated by that speaker.Prompts are 3–20-second utterances, and target sentences and anchors are taken at least two seconds away from the selected stretch.Source: A Data, Models and Training · p. 11, Table 6 · p. 12
  • The study uses four open zero-shot TTS models from three families with default inference settings, and defines automatic measures for WER, quality, failure, and speaker similarity.The listed settings include prompt audio with or without transcripts, depending on the model.Source: A Data, Models and Training · p. 11
  • The audio selector encodes prompts with a frozen Whisper-base encoder and ranks candidates using heads trained for WER, failure, quality, and similarity outcomes.Selection uses the WER-rank head; training applies ListNet across each recording’s candidates.Source: A Data, Models and Training · pp. 11–12
  • The descriptor ranker uses 21 recording-normalized descriptors, while voice-matched recordings are excluded and cross-validation keeps same-voice clusters together.Source: A Data, Models and Training · p. 12
  • Labeling takes about 1–2.5 seconds of GPU time per output, or roughly 100–270 GPU hours per model for the training sets.Source: A Data, Models and Training · p. 12

B Robustness and Ablations

Across models, selector transfer varies, while tests of prompt robustness, quality rules, prosody, and listening add boundaries on how selection outcomes should be interpreted. The listening comparison found no clear average preference, and its one-output-per-pair design requires many pairs to distinguish small effects.

B Robustness and Ablations

  • Transcript checks found selector picks about as reliable as UTMOS picks, and the selector stayed ahead of the rule when candidates were restricted to consensus prompts.Transcript disagreement is the WER between Whisper and wav2vec 2.0 prompt transcripts; consensus prompts have matching Whisper transcripts and YouTube captions.Source: B Robustness and Ablations · p. 12
  • For Qwen3-TTS 0.6B, selections slightly lowered output pitch variability, while the audio selector matched a random prompt on pitch-variability error and modestly improved tempo error.The selector changed pitch variability by −0.06 st and tempo error by −0.013; the measures capture global prosody, not emphasis or context-appropriate stress.Source: B Robustness and Ablations · p. 12

C Quality Rules

  • Quality-rule results compare DNSMOS and UTMOS gains with probe-selection gains on 400 training recordings using 3-second candidates.The descriptor analysis reports within-recording correlations with prompt intelligibility across 400 test recordings.Source: C Quality Rules · p. 12, Table 7 · p. 13, Table 11 · p. 14

D Other TTS Models

  • 1.49 pp WER gain transferred from a Qwen3-TTS 0.6B-trained selector to Qwen3-TTS 1.7B; transfer to Chatterbox and F5-TTS was 0.14 and 1.53 pp.Transfer varied by training model: Chatterbox-trained selectors gained 0.88, 0.77, and 1.77 pp on Qwen3-TTS 0.6B, 1.7B, and F5-TTS, respectively; F5-TTS-trained selectors gained 1.04, 1.57, and 0.23 pp on Qwen3-TTS 0.6B, 1.7B, and Chatterbox.Source: Table 12 · p. 14, D Other TTS Models · p. 15
  • WER agreement between Qwen3-TTS models was 0.35, between Qwen and F5-TTS was 0.22–0.23, and with Chatterbox was at most 0.03.Reliability ranged from 0.29 for Chatterbox WER to 0.97 for F5-TTS similarity; WER reliability was 0.52 and 0.59 for Qwen3-TTS 0.6B and 1.7B, 0.29 for Chatterbox, and 0.64 for F5-TTS.Source: D Other TTS Models · pp. 13, 15

E Prompts Differing in Noise

  • The noise experiment varied prompt duration and noise level while candidates competed within each speaker, but the supplied passage reports its design rather than its outcome.It used 3-, 5-, and 10-second prompts with white or DEMAND noise at 20, 10, or 5 dB SNR.Source: E Prompts Differing in Noise · p. 15

F Listening Test

  • The audio selector and UTMOS rule had near-zero mean listener preference: +0.02 for clarity and +0.08 for naturalness.Across 23 pairs with different word errors, 62% of ratings that preferred a side favored the side with fewer Whisper errors, but the clarity interval was [−0.29, 0.81].Source: F Listening Test · p. 15
  • A one-output-per-pair listening test would need about 305 pairs for a 95% interval to exclude zero under the assumed Whisper-aligned ratings.Real listeners may require more because they tie more often and disagree with one another.Source: F Listening Test · p. 15
Loading 2610.40372v1…