Source-linked AI summary
Sequential Trajectories and Simultaneous Blending: Multi-Emotion Modeling for Instruction-Following TTS
Yan Zhou, Yun Hong, Yang Feng
TL;DR
Multi-emotion TTS lacks adequate control and evaluation for ordered emotion trajectories and simultaneous emotion blending. HybridEmo combines supervised initialization with task-matched GRPO rewards, improving trajectory correctness and blending intensity, including a rise from 3.47 to 3.71 over CosyVoice 3.
Problem
Multi-emotion TTS remains underexplored because existing systems largely model single utterance-level emotion and lack structure-aware evaluation for trajectories and blending.
Method
HybridEmo combines supervised multi-emotion initialization with GRPO using routed, task-specific rewards alongside shared ASR feedback.
Results
3.47 to 3.71: HybridEmo significantly improves blending intensity over CosyVoice 3 while also improving trajectory correctness without noticeable speaker-similarity degradation.
Takeaways & Limitations
Human evaluators prefer HybridEmo to CosyVoice 3 and EmoVoice-0.5B, while preferences are nearly balanced against Qwen3-TTS.
Takeaways & Limitations
The Qwen3-TTS comparison is not strictly matched because its tested interface uses textual instructions and a designated voice rather than jointly accepting speech prompts and text instructions.
Abstract
from arXiv · showhide
Natural-language instructions enable flexible control of synthesized speech, yet emotional TTS systems primarily model a single utterance-level affect, leaving multi-emotion control underexplored. We study two complementary multi-emotion TTS tasks: emotion trajectory, which spans several ordered affective stages, and emotion blending, in which multiple emotions coexist throughout an utterance. These tasks expose a supervision mismatch: supervised fine-tuning (SFT) does not explicitly evaluate emotion features, while single-emotion rewards provide neither structure-aware feedback for trajectory completion nor pair-aware feedback for blending. We introduce HybridEmo, a post-training framework that initializes both tasks with SFT and then aligns the speech-token policy through Group Relative Policy Optimization using a sample-aware hybrid reward. For trajectory samples, segment-aligned consistency combines average and weakest-stage evidence to preserve the correctness and completeness of prescribed stages. For blending samples, a GMM-based reward combines frame-level support from the union of target-emotion anchors in an offline emotion space with an utterance-level weaker-target margin. Both branches share an ASR reward and are routed within a unified policy. On MultiEmo-Test, HybridEmo significantly improves trajectory correctness and blending intensity, without a noticeable degradation in speaker similarity. Human evaluation prefers HybridEmo to CosyVoice 3 and EmoVoice-0.5B, with nearly balanced preferences against Qwen3-TTS.
Introduction
The paper frames multi-emotion TTS as sequential emotion trajectories and simultaneous blending, arguing that conventional SFT cannot explicitly assess these structured patterns. HybridEmo addresses this gap with SFT initialization followed by sample-aware GRPO rewards, improving both tasks while preserving speaker similarity.
- Problem formulation: Multi-emotion speech comprises sequential evolution and simultaneous blending, requiring relationships among emotions beyond a single utterance-level label.The paper introduces these as the two target control forms for TTS and evaluates their precise emotional patterns.
- Limitations of SFT: Token-level SFT fits target speech-token sequences but does not explicitly assess prescribed sequential or concurrent emotional patterns.Learning stable speech generation and structured realization from large-scale multi-emotion triplets is also costly.
- HybridEmo: HybridEmo initializes multi-emotion generation with SFT, then aligns the speech-token policy using GRPO and a sample-aware hybrid reward.The framework routes task-matched emotion rewards alongside shared content feedback within a unified policy.
- Task-matched rewards: Trajectory rewards combine mean and weakest-stage evidence, while blending rewards combine frame-level target-pair compatibility with an utterance-level weaker-target margin.These mechanisms assess stage completion for trajectories and encourage coexistence without single-target dominance for blending.
- Evaluation: HybridEmo significantly improves trajectory correctness and blending-oriented perceptual scores without noticeable speaker-similarity degradation, and human evaluation favors it over CosyVoice 3 and EmoVoice-0.5B.Human preferences are near-balanced between HybridEmo and Qwen3-TTS.
Background
The background traces speech-token language modeling, natural-language style prompting, and reinforcement learning as complementary foundations for controllable and perceptually aligned TTS. Prior work progressively expands generation interfaces, control expressiveness, and sequence-level optimization.
- Speech-token TTS: Discrete speech tokens connect waveforms with language-model generation, supporting zero-shot TTS and staged semantic-to-acoustic modeling.VALL-E models neural-codec codes autoregressively, while SPEAR-TTS factorizes semantic and acoustic generation to use audio-only data.
- Instruction-based control: Natural-language prompting replaces fixed style labels with expressive textual control over style, vocal factors, and disentangled acoustic representations.PromptTTS, PromptTTS 2, and InstructTTS progressively address textual style conditioning, underspecified vocal factors, and separation of style, speaker, and content.
- Preference optimization: Reinforcement learning complements token-level supervision with sequence-level preference or reward signals targeting perceptual and task-specific speech quality.DPO avoids explicit reward modeling, Emo-DPO sharpens emotional distinctions through constructed preferences, and GRPO is introduced as a related optimization approach.
Methodology
HybridEmo first learns multi-emotion speech-token generation with SFT, then uses GRPO with a shared ASR reward and task-specific emotion feedback. Trajectory rewards combine average and weakest-stage evidence, while blending rewards use frozen emotion-space GMM anchors and weaker-target regularization.
- Conditional generation: The model preserves linguistic content and reference timbre while realizing the emotional pattern specified by the natural-language instruction.The timbre reference is used only by the acoustic generator for waveform decoding.
- Training framework: HybridEmo initializes trajectory and blending generation with SFT, then treats the speech-token LLM as a policy and aligns it with GRPO using decoded rollouts.SFT learns from real multi-emotion speech; GRPO updates the policy under a KL constraint without target waveforms.
- Hybrid reward: The unified reward shares ASR-based intelligibility feedback and routes either trajectory consistency or blending GMM feedback according to sample type.The inapplicable emotion component is masked, and invalid or degenerate token sequences receive zero reward.
- Trajectory consistency: Trajectory consistency segments generated speech by text-length proportions and combines mean and weakest-stage emotion scores to prevent a failed stage from being hidden.The timestamp-free segmentation assumes text length roughly tracks speaking duration.
- Blending GMM reward: Blending compatibility is scored using frozen frame-level GMM anchors in an offline projected emotion space, with a weaker-target margin discouraging dominance by one emotion.The margin becomes inactive once the weaker target reaches the minimum desired contribution, while running standardization accommodates score-scale variation.
Experiments
Experiments evaluate HybridEmo on newly constructed multi-emotion reinforcement-learning and test sets, using task-specific emotion metrics alongside speech quality, intelligibility, and speaker-similarity measures. Comparisons include CosyVoice 3, EmoVoice, and Qwen3-TTS, with the latter treated as an unmatched external capability reference.
- Datasets: MultiEmo-RL contains 12,000 trajectory and 2,400 blending text–instruction pairs spanning 12 unordered emotion pairs without target waveforms.Trajectory examples include emotion-tagged transcripts solely for constructing the alignment reward.
- Datasets: MultiEmo-Test contains 720 conditions: 200 each for 1-, 2-, and 3-stage trajectories, plus 120 blending examples covering the same 12 emotion pairs.Every condition uses a reference voice from the English Seed-TTS evaluation set, shared with systems supporting timbre conditioning.
- Training setup: HybridEmo is initialized from the 0.5B CosyVoice 3 checkpoint, SFT-trained for 5 epochs, then optimized for 1 GRPO epoch on MultiEmo-RL.SFT uses Adam with learning rate 2×10−6 and effective global batch size 64; GRPO uses n = 8 rollouts per input, learning rate 10−6, and batch size 64.
- Baselines: HybridEmo is compared with 0.5B CosyVoice 3, EmoVoice at 0.5B and 1.5B, and Qwen3-TTS-12Hz-1.7B-VoiceDesign on shared target texts and emotion instructions where supported.Qwen3-TTS cannot jointly accept speech prompts and text instructions, so it uses a designated voice and is treated as an external capability reference.
- Evaluation: Evaluation uses 1–5 task-specific rubrics for trajectory correctness and naturalness and blending intensity and naturalness, supplemented by WER, UTMOS, and speaker similarity.Trajectory correctness averages across 1–3 stages, naturalness across 2–3 stages; speaker similarity is unavailable for Qwen3-TTS because it uses no reference speech.
Results and Analysis
HybridEmo improves automatic and human-evaluated multi-emotion TTS quality while preserving intelligibility and speaker similarity. Ablations support SFT initialization, joint hybrid reinforcement learning, and the weaker-target margin, while tokenizer reconstruction retains emotion information and GMM anchors support discriminative blending scores.
- Automatic evaluation: Trajectory macro correctness rises from 3.24 to 3.33 over CosyVoice 3, with gains at 1E, 2E, and 3E.Macro naturalness also increases from 2.40 to 2.50; WER is 1.87% and UTMOS is 3.21.
- Automatic evaluation: Blending Intensity increases from 3.47 to 3.71 and Naturalness from 3.06 to 3.24 over CosyVoice 3.HybridEmo exceeds both EmoVoice variants, matches Qwen3-TTS in intensity, and trails it by 0.05 in naturalness; WER is 0.37%.
- Human evaluation: Human preferences favor HybridEmo over CosyVoice 3 and EmoVoice-0.5B by 32 percentage points in win-minus-loss rates, while comparison with Qwen3-TTS is balanced.Against Qwen3-TTS, HybridEmo receives 35% wins, 29% ties, and 36% losses, with differing timbre conditions affecting the comparison.
- Ablation analysis: SFT alone provides limited gains, Direct Hybrid-GRPO raises some emotion scores but increases WER, and task-specialized variants show task-dependent strengths.Trajectory-GRPO slightly exceeds joint HybridEmo on trajectory metrics, while joint HybridEmo achieves the highest blending intensity and naturalness.
- Ablation analysis: Removing the weaker-target margin lowers Intensity from 3.71 to 3.66 and Naturalness from 3.24 to 3.05, reducing their mean from 3.48 to 3.36.The results suggest that limiting single-anchor dominance improves both aspects of blending quality.
- Representation and scoring analysis: Speech-token reconstruction achieves a paired emotion2vec+ large cosine similarity of 0.9441, indicating that emotion representations remain highly similar after reconstruction.GMM anchor distributions are distinct but partially overlapping, with full-covariance components capturing intra-emotion modes and anisotropic variation for continuous mixture-density scoring.
Conclusion
The paper formulates multi-emotion control as emotion trajectories and blending, and proposes HybridEmo with supervised initialization and sample-aware hybrid-reward GRPO. On MultiEmo-Test, HybridEmo improves trajectory correctness and blending perceptual scores over CosyVoice 3 while matching Qwen3-TTS.
- Method: HybridEmo combines supervised multi-emotion initialization with sample-aware hybrid-reward GRPO, routing task-matched trajectory and blending rewards alongside shared ASR feedback.The framework uses a unified policy with rewards matched to each multi-emotion task.
- Results: On MultiEmo-Test, HybridEmo significantly improves emotion correctness over CosyVoice 3 at every trajectory length and both blending-oriented perceptual scores.The reported gains cover both trajectory and blending evaluations.
- Results: On MultiEmo-Test, HybridEmo matches Qwen3-TTS.The conclusion reports matching performance against this baseline.