Source-linked AI summary
Post-Training Zero-Shot TTS for Fine-Grained Emotion and Duration Control via Natural Language
Lianru Gao, Yujie Guo, Yong Qin
TL;DR
Existing TTS controls often apply style at the utterance level, although applications need emotion and pacing to change within an utterance. This paper post-trains pretrained TTS with natural-language segment instructions, and reports improved fine-grained control while maintaining intelligibility and speaker identity.
Problem
Practical speech applications need distinct emotion and timing targets across utterance segments, while typical TTS controls are applied at the utterance level and joint natural-language control remains underexplored.
Source: §1 · p. 1Method
The framework uses supervised fine-tuning to establish instruction-conditioned generation, then GRPO to refine segment-level emotion and duration control with content and speaker-preservation objectives.
Source: §1 · p. 1Results
Staged post-training improves fine-grained controllability and outperforms most evaluated baselines on major control metrics, with counterfactual, instruction-variation, and human evaluations indicating robust control and more natural transitions.
Source: §5 · p. 5Takeaways & Limitations
Abstract
from arXiv · showhide
Audiobook narration, conversational agents, and audiovisual dubbing require speech that conveys changing emotions and adapts its pacing within a single utterance. But most existing TTS systems typically rely on utterance-level style conditioning, making such fine-grained control difficult to achieve. In light of this, and inspired by the success of post-training in large language models, we propose a unified post-training framework that equips pretrained text-to-speech models with natural-language control over segment-level emotion and duration. Supervised fine-tuning establishes instruction-conditioned speech generation, while reinforcement learning with group relative policy optimization refines control accuracy using emotion and duration rewards alongside content and speaker preservation objectives. By reusing the pretrained architecture, our approach avoids additional inference-time control modules. Experiments demonstrate significantly improved fine-grained controllability while maintaining speech intelligibility and speaker identity, highlighting post-training as a practical approach to extending existing speech synthesis models.
1. INTRODUCTION
Practical speech applications need emotion and pacing to vary across segments, but existing TTS controls are generally utterance-level or rely on specialized signals. The paper proposes post-training pretrained TTS models for natural-language segment-level emotion and duration control.
1. INTRODUCTION
- Audiobook narration, conversation, and dubbing need distinct emotion and timing targets within an utterance, while typical TTS controls operate at the utterance level.This makes satisfying multiple segment-specific requirements difficult.Source: §1 · p. 1
- Fine-grained prior approaches use structured signals or specialized mechanisms, leaving joint segment-level emotion and duration control through natural-language instructions underexplored.Earlier post-training work mainly addressed content fidelity, emotion, or general instruction following individually.Source: §1 · p. 1
- The framework uses supervised fine-tuning to establish instruction-following, then GRPO to optimize segment-level emotion and duration rewards alongside content and speaker preservation.It reuses the pretrained generation architecture without additional inference-time control modules.Source: §1 · p. 1
- Instruction-equivalence distillation trains a student on paraphrased instructions against a frozen teacher using canonical instructions, promoting consistent behavior across equivalent formulations.The stated goal is better generalization beyond instruction forms observed during training.Source: §1 · pp. 1–2
- The evaluation covers emotion-only, duration-only, and joint control, with segment- and utterance-level metrics plus counterfactual and instruction-variation tests.The contribution also includes human evaluation of the framework's control behavior.Source: §1 · p. 2
2. METHOD
The method fine-tunes the speech language model of CosyVoice2 while freezing its tokenizer, acoustic generator, and vocoder. It builds instruction control through staged emotion and duration training, joint refinement, and instruction-equivalence distillation.
2. METHOD
- The approach fine-tunes CosyVoice2's speech language model for segment-level emotion, duration, or relative-rate instructions while freezing its tokenizer, acoustic generator, and vocoder.The generated utterance is intended to satisfy segment requests while preserving content and reference-speaker identity.Source: §2 · p. 2
2.1. Training Data Construction
- The training data comprises 56.6 hours of manually screened emotion speech, 168 hours of duration-only speech, and 30 hours of joint emotion–duration speech.Duration supervision is constructed from transcribed LibriTTS-R recordings, while joint emotion–duration supervision applies the procedure to constructed emotional speech, retaining emotion labels.Source: §2.1 · p. 2
2.2. Emotion-Control Post-Training
- Emotion post-training uses teacher-forced token prediction for supervised initialization, then GRPO with emotion confidence, segment-accuracy, and strict utterance-accuracy rewards.Content and speaker-preservation rewards are also included.Source: §2.2 · pp. 2–3
2.3. Duration Acquisition and Joint Refinement
- Duration SFT adds token-length and boundary supervision, duration-focused GRPO scores timing accuracy, and joint refinement mixes emotion-only, duration-only, and joint examples.Emotion and duration rewards apply when their corresponding controls are present; content and speaker rewards apply across examples.Source: §2.3 · p. 3
2.4. Instruction-Equivalence Distillation
- Instruction-equivalence distillation pairs a frozen teacher using canonical instructions with a student using intent-preserving paraphrases, then applies a short GRPO stage to refine control accuracy.The distillation objective promotes consistent generation across equivalent instruction formulations.Source: §2.4 · p. 3
3. EXPERIMENTS
The experiments use disjoint standard, counterfactual, and paraphrased-instruction evaluations to test segment-level control, with objective metrics and human listening tests assessing control and speech quality.
3.1. Evaluation Setup
- The 534-example test set covers emotion-only, duration-only, and joint control, while separate counterfactual and paired-paraphrase sets test control under neutral content and wording variation.Evaluation speakers are unseen during post-training; the counterfactual sets are also disjoint at the full-text and segment levels.Source: §3.1 · p. 3
3.2. Objective Metrics
- Emotion is evaluated with segment, strict utterance, and dual-classifier accuracy; duration metrics assess exact-target tolerance, rate-range compliance, and fast/normal/slow direction.ASR error and speaker-embedding similarity measure content and speaker preservation, while boundary JR-F0 measures local pitch discontinuities near segment boundaries.Source: §3.2 · p. 3
3.3. Human Evaluation
- Two human listening tests use three raters per sample to examine emotion labels on classifier-disagreement segments and compare transition and utterance naturalness against concatenated-speech baselines.The naturalness test compares the single-pass joint-control model with CosyVoice3 and IndexTTS2 concatenations.Source: §3.3 · p. 4
4. RESULTS
Staged post-training improves emotion and duration control, and the final models outperform most evaluated baselines on major control metrics. Counterfactual, instruction-variation, and human evaluations also report robust control behavior and more natural transitions.
Control performance
- Joint-RL reaches MER/E2V utterance accuracies of 54.1%/24.1% and duration A15/U15 of 54.6%/32.0%, with 4.1% ASR error and 0.795 speaker similarity.The reported results surpass natural-language baselines and structured-interface systems on their respective control dimensions.Source: §4 · p. 4
Ablation over post-training stages
- The IED+GRPO stage achieves 3.7% ASR error and MER scores of 79.6%/58.2%, plus the best DA at 63.4%, while standard-format duration accuracy softens.Across stages, speaker similarity remains 0.780–0.806 and JR-F0 0.123–0.136; duration-SFT restores content fidelity, and duration-RL improves timing before joint refinement recovers emotion performance.Source: §4 · pp. 4–5
Counterfactual evaluation and instruction robustness
- Under unseen instruction formats, ASR error falls from 19.8% to 3.6%, with improved emotion segment accuracy and A15; standard-format duration performance declines, indicating a wording-robustness versus timing-precision trade-off.The paired instruction-variation set preserves texts, references, and targets, so it measures wording robustness rather than generalization to new text.Source: §4 · p. 5
Human evaluation
- Joint-RL earns boundary MOS 4.64 versus 3.34 for concatenated CosyVoice3 and 3.99 for concatenated IndexTTS2, supporting smoother transitions.On selected classifier-disagreement segments, MERaLiON agrees with human labels more often than emotion2vec+, without establishing overall classifier superiority.Source: Table 5 · p. 4, §4 · p. 5
5. CONCLUSION
The paper presents natural-language segment-level emotion and duration control through post-training, reports improved controllability and natural transitions, and identifies word-level emphasis as future work.
5. CONCLUSION
- The framework uses staged SFT and GRPO for natural-language segment-level emotion and duration control without additional inference-time modules.The paper reports that the method outperforms most evaluated baselines on major control metrics.Source: §5 · p. 5
- Counterfactual, instruction-variation, and human evaluations further demonstrate robust control behavior and more natural speech transitions.Source: §5 · p. 5
- Future work will explore word-level emphasis.Source: §5 · p. 5
6. COMPLIANCE WITH ETHICAL STANDARDS
The authors report that the listening study followed the Declaration of Helsinki, used voluntary informed participation, and collected no personally identifiable or sensitive information.
6. COMPLIANCE WITH ETHICAL STANDARDS
- Participants were informed of the research purpose and voluntarily agreed to participate; no personally identifiable or sensitive information was collected.Source: §6 · p. 5
- The study was conducted in accordance with the principles of the Declaration of Helsinki.Source: §6 · p. 5
- The authors state that their institution has no formal ethics-review procedure applicable to minimal-risk non-biomedical listening studies of this nature.Source: §6 · p. 5