Source-linked AI summary
Beyond Speech Captions: Speech-Rewarded Style Planning for Conversational Text-to-Speech
Shiao Zhu, Lianbo Liu, Sizhen Lyu, Yuzhe Wang, Sheng Li, Takahiro Shinozaki
TL;DR
Natural-language style descriptions offer an interpretable interface for conversational TTS, but captions omit acoustic details and descriptive alignment need not predict effective synthesizer control. SRSP trains a text-based style planner using target-speech likelihood from a frozen TTS model; on an English ISCSLP 2026 CoT-TTS subset, it outperforms baselines on acoustic metrics and gains on LLM-based expressive-speech evaluation.
Problem
Speech captions omit acoustic details, and speech–text alignment only weakly predicts downstream acoustic similarity among candidate instructions for the same utterance.
Source: §1 · p. 1Method
SRSP uses GRPO to train a text-based style planner with teacher-forced target-speech likelihood from a frozen downstream TTS model, without target style captions.
Source: §1 · p. 1Results
On an English subset of the ISCSLP 2026 CoT-TTS corpus, SRSP achieves the best performance on five downstream acoustic metrics in both partitions and gains in contextual appropriateness and reference consistency.
Source: §1 · p. 1, §4.2 · p. 4Takeaways & Limitations
The results support optimizing natural-language instructions for downstream synthesis behavior rather than relying solely on descriptive alignment.
Source: §5 · p. 4Expressive-speech evaluation relies on an LLM judge and has not been validated by human listening tests.
Source: §5 · p. 4
Abstract
from arXiv · showhide
Natural-language style descriptions provide an interpretable interface between large language models (LLMs) and controllable text-to-speech (TTS). However, using descriptions as pseudo-labels compresses target acoustics into text, and descriptive fidelity need not imply effective control of a particular synthesizer. We empirically show that speech-text alignment only weakly predicts downstream acoustic similarity among candidate instructions for the same utterance. We therefore propose Speech-Rewarded Style Planning (SRSP), which trains a text-based style planner through a frozen downstream TTS model. Given dialogue history and response text, the planner generates candidate instructions and is optimized with group-relative policy optimization (GRPO), using the teacher-forced likelihood of target speech tokens as the reward. On an English subset of the ISCSLP 2026 CoT-TTS corpus, SRSP achieves higher speech-style and emotion similarity to target speech and lower mel-cepstral distortion than the Base LLM and target-audio-informed captioning baselines. LLM-based expressive speech evaluation further shows gains over all baselines in contextual appropriateness and reference consistency.
1. INTRODUCTION
Conversational delivery depends on context, but speech captions omit acoustic details and descriptive accuracy does not necessarily translate into effective control of a particular synthesizer. The paper proposes training a style planner with feedback from the downstream TTS model and reports gains over baselines.
1. INTRODUCTION
- Caption supervision omits acoustic details, and an accurate description need not induce the closest acoustic realization from a particular synthesizer.The paper distinguishes predicting a target description from producing an instruction that controls the synthesizer effectively.Source: §1 · p. 1
- Speech–text alignment only weakly predicts downstream acoustic similarity among candidate style instructions for the same utterance.The authors therefore describe alignment as a weak proxy for selecting effective instructions.Source: §1 · p. 1
- SRSP trains a text-based style planner with teacher-forced target-speech likelihood from a frozen downstream TTS model, using GRPO without target style captions.Candidate instructions receive rewards based on target-speech likelihood, optimizing for downstream synthesis behavior rather than agreement with speech-derived descriptions.Source: §1 · p. 1
- On an English subset of the ISCSLP 2026 CoT-TTS corpus, SRSP outperforms all baselines on five acoustic metrics across both partitions and gains in contextual appropriateness and reference consistency.The expressive-speech evaluation is LLM-based.Source: §1 · p. 1
2. DESCRIPTIVE ALIGNMENT AS A PROXY FOR DOWNSTREAM CONTROL
Candidate-level analyses compare speech–text alignment scores with downstream acoustic metrics and find that alignment is a weak proxy for selecting instructions that synthesize speech close to the target. Selection by downstream acoustic metrics outperforms selection by speech–text alignment.
2. DESCRIPTIVE ALIGNMENT AS A PROXY FOR DOWNSTREAM CONTROL
- The analysis samples eight Base LLM instructions per utterance from the 5,500-utterance development set and synthesizes each response under each instruction.Instructions are generated from dialogue history and response text using the training decoding parameters.Source: §2 · p. 1
- T2S measures instruction-to-target-speech style alignment, while S2S, emotion similarity, and MCD-DTW measure downstream acoustic similarity.T2S uses CLAPScore; S2S is speech–speech cosine similarity, emotion similarity uses emotion2vec probability vectors, and MCD-DTW is mel-cepstral distortion after dynamic time warping.Source: §2 · pp. 1–2
- Within-utterance Spearman correlations compare each T2S score with downstream acoustic metrics across candidate instructions, averaging correlations across utterances.MCD is negated so that higher values consistently indicate better target matching.Source: §2 · p. 2
- ParaSpeechCLAP T2S scores are nearly uncorrelated with S2S, emotion similarity, and −MCD; CLSP correlations are somewhat stronger but remain weak.For each downstream acoustic metric, its correlation with T2S remains lower than its correlations with the other acoustic metrics.Source: §2 · p. 2, Figure 1 · p. 2
- Acoustic-metric selection outperforms all three T2S-based criteria across all five downstream acoustic metrics, while T2S selection provides only small improvements over random.The comparison uses the combined development partitions.Source: §2 · p. 2, Table 1 · p. 2
3. SPEECH-REWARDED STYLE PLANNING
SRSP generates natural-language style instructions from dialogue history and response text, using target speech only to score candidates during training. GRPO standardizes group rewards and updates the planner with a clipped, KL-regularized objective.
3.1. Problem formulation
- The style policy generates an instruction from dialogue history and response text; target speech is used only to construct the training reward.Source: §3.1 · p. 2
3.2. Speech-token likelihood as reward
- For each candidate, the frozen TTS model computes mean teacher-forced cross-entropy over target speech tokens, and the reward subtracts that loss and auxiliary penalties.The penalties discourage overly short, generic, and repetitive instructions.Source: §3.2 · p. 2
3.3. Group-relative policy optimization
- GRPO standardizes rewards across sampled instructions, then optimizes the planner with a clipped objective and a KL penalty to the reference policy.The group standardization uses the reward mean, standard deviation, and a small numerical-stability constant.Source: §3.3 · pp. 2–3
- Paired Wilcoxon tests found significant gains in 38 of 40 downstream acoustic comparisons; the exceptions were test-out emotion similarity versus Raw TTS and Qwen3-Omni.Source: Table 2 · p. 3
- The planner produces one English sentence describing speaking style, with zero to three shared in-context examples during training and two sampled examples at inference.Source: §3.3 · p. 3
4. EXPERIMENTS
Experiments compare SRSP with raw, Base LLM, and target-audio-informed captioning baselines across held-out partitions and LLM-judged expressive-speech settings; SRSP leads the reported acoustic metrics and judge preferences.
4.1. Experimental setup
- The English corpus subset is partitioned into development and test sets that hold out either complete movie scenes or entire movies.The retained pool contains 299.75 hours; training contains 277.60 hours after excluding overlap with held-out utterances.Source: §4.1 · p. 3
4.2. System-level results
- SRSP achieves the best performance on all five downstream acoustic metrics in both partitions, despite privileged captioning baselines having access to target audio.The unadapted Base LLM has lower S2S scores and higher MCD-DTW than Raw TTS; speech-rewarded post-training reverses this degradation.Source: §4.2 · p. 4
- AF-Next has the highest T2S scores under all three speech–text encoders but ranks last on all five downstream acoustic metrics in both partitions.This system-level mismatch mirrors the candidate-level finding that alignment only weakly predicts downstream acoustic similarity.Source: §4.2 · p. 4
4.3. LLM-based expressive speech evaluation
- SRSP is preferred over all four baselines in both evaluation settings and partitions, with p < .001 in two-sided exact binomial tests.In context-only comparisons against Base LLM, win rates are 54.19%/53.67%; ground-truth speech remains preferred over all synthesized systems.Source: §4.3 · p. 4, Table 3 · p. 4
5. CONCLUSION
The conclusion argues that descriptive alignment is a weak proxy for downstream synthesis control and presents SRSP as a way to optimize instructions for a frozen TTS model. It reports gains across acoustic and expressive-speech evaluations, while noting that human validation and transfer to other TTS backends remain open.
5. CONCLUSION
- Speech–text alignment only weakly predicts downstream acoustic similarity among candidate instructions for the same utterance.The conclusion recommends optimizing natural-language instructions for downstream synthesis behavior rather than relying solely on descriptive alignment.Source: §5 · p. 4
- SRSP trains a conversational style planner using teacher-forced target-speech likelihood from a frozen TTS model.The reward computation and synthesis both use CosyVoice3.Source: §5 · p. 4
- SRSP outperforms all baselines on five downstream acoustic metrics across both evaluation partitions, with further gains in contextual appropriateness and reference consistency.These findings come from the English subset of the ISCSLP 2026 CoT-TTS corpus.Source: §5 · p. 4
- The expressive-speech evaluation has not been validated by human listening tests, and transfer beyond CosyVoice3 remains for future work.Both reward computation and synthesis use CosyVoice3.Source: §5 · p. 4