Source-linked AI summary
Emo-BPO: Emotion Bidirectional Preference Optimization for Diffusion-based Emotional TTS
Jiacheng Shi, Hongfei Du, Xinyuan Song, Y. Alicia Hong, Yanfu Zhang, Ye Gao
TL;DR
Diffusion-based emotional TTS preference methods reinforce target emotions but do not explicitly model separation from competing modes under classifier-free guidance. Emo-BPO learns aligned and contrastive score functions from reordered same-text pairs; experiments report improvements in emotional alignment, prosodic control, and perceptual quality while maintaining competitive intelligibility.
Problem
Existing diffusion-based preference alignment reinforces target emotions but does not explicitly model divergence from competing emotional modes under classifier-free guidance.
Source: §1 · p. 1Method
Emo-BPO jointly learns emotion-aligned and emotion-contrastive score functions from reordered same-text emotional pairs and combines them through contrastive guidance.
Source: §1 · p. 1, §3.2.1 · p. 3, §3.2.2 · p. 3Results
Experiments report improvements in emotional alignment, prosodic control, and perceptual quality while maintaining competitive intelligibility.
Source: §5 · p. 4Takeaways & Limitations
The results indicate that jointly modeling emotional attraction and separation leads to controllable and expressive emotional speech generation.
Source: §5 · p. 4
Abstract
from arXiv · showhide
Diffusion-based TTS has demonstrated strong capability in emotional speech generation. However, existing emotion alignment methods primarily reinforce target emotional expressions while overlooking modeling of competing emotional modes, limiting fine-grained controllability. This issue becomes more pronounced under classifier-free guidance (CFG), which relies on contrasting conditional trajectories to shape generation. We propose Emotion Bidirectional Preference Optimization (Emo-BPO), a simple effective framework that jointly learns emotion-aligned and emotion-contrastive score functions from reordered same-text emotional pairs. Our method requires neither auxiliary reward models nor additional annotations, and can be seamlessly integrated into existing diffusion-based TTS systems to improve emotional controllability and expressiveness.
1. Introduction
Emotional TTS must coordinate several expressive dimensions, yet existing preference-alignment approaches do not explicitly model separation from competing emotional modes. Emo-BPO addresses this gap by learning aligned and contrastive score functions from reordered same-text emotional pairs.
1. Introduction
- Emotional TTS must coordinate stress, intonation, rhythm, and their interactions to generate speech that is both intelligible and affectively expressive.Fine-grained emotional control matters for conversational agents [1] [2], expressive narration [3], and human-machine interaction.Source: §1 · p. 1
- Earlier systems such as Tacotron [7], FastSpeech [8], and VITS use categorical embeddings or conditioning vectors that limit fine-grained expressiveness.The passage also notes that LLM-based TTS and diffusion modeling seek improved controllability and generalization.Source: §1 · p. 1
- Preference-based methods include paired-sample direct optimization, reward-based reinforcement learning [18], and stepwise diffusion preferences [19].The cited approaches reframe emotional alignment through human judgments, reward functions, or stepwise preference signals.Source: §1 · p. 1
- Existing preference methods reinforce preferred outputs but do not explicitly model divergence from non-preferred emotional modes under classifier-free guidance.The passage says this asymmetry may weaken inter-emotion discrimination.Source: §1 · p. 1
- Emo-BPO jointly learns emotion-aligned and emotion-contrastive score functions from reordered same-text pairs, without auxiliary reward models or additional annotations.Its branches are combined through contrastive guidance compatible with classifier-free guidance.Source: §1 · p. 1
2. Related Work
Prior emotional TTS work uses explicit emotion conditioning and preference-driven optimization, but preference methods do not model separation from competing emotional modes. Emo-BPO pairs aligned and contrastive diffusion branches to address that gap.
2. Related Work
- Emotion-controllable TTS uses emotion embeddings, VAD-based vectors, natural-language prompts, or instruction-conditioned LLM synthesis.The cited examples include EmoSpeech [8], EmoSphere++ [21], EmoVoice [22], CosyVoice [10], and CosyVoice2 [11].Source: §2 · p. 1
- Emo-BPO learns complementary emotion-aligned and emotion-contrastive score functions from reordered same-text pairs without auxiliary reward models.The overview depicts training on inverted emotional pairs and combining both branches through contrastive guidance at inference.Source: §2 · p. 2, Figure 1 · p. 2
- Prior preference-driven methods reinforce target emotions without modeling separation from competing emotional modes.Examples include Emo-DPO, EMORL [18], and EASPO [19].Source: §2 · p. 2
3. Proposed Method
Emo-BPO adapts classifier-free guidance to emotional TTS by jointly learning aligned and contrastive emotion scores from reordered same-text pairs. Inference combines the branches with scheduled guidance to strengthen target emotion while suppressing competing modes.
3.1. Preliminaries: Classifier-Free Guidance
- Classifier-free guidance contrasts conditional and reference scores, with guidance strength controlling the contrastive correction and motivating emotion-specific reference conditions.The log-ratio view motivates replacing the usual unconditional reference with an emotion-contrastive condition.Source: §3.1 · p. 2
3.2. Emotion Bidirectional Preference Optimization
- Emo-BPO jointly learns an emotion-aligned objective that increases likelihood under the desired emotion and an emotion-contrastive objective that suppresses alternatives.The attractive aligned and repulsive contrastive directions are realized through preference optimization during training and contrastive guidance at inference.Source: §3.2 · p. 2
3.2.1. Train with Emo-BPO
- Reordering same-text samples with different emotion labels supplies aligned and complementary contrastive training directions without additional annotations or auxiliary objectives.Each ordered pair defines one direction; reversing its order supplies the complementary direction.Source: §3.2.1 · p. 2
- Emo-BPO achieves the best performance in emotion similarity, prosody similarity, UTMOS, and mean emotion recognition accuracy while maintaining competitive intelligibility.Results are averaged over three independent runs.Source: Table 1 · p. 3
- The subjective evaluation reports that Emo-BPO outperforms representative baselines in MOS, Emo MOS, MOS EC, and mean human-recognized emotion accuracy.Source: Table 2 · p. 3
- Separate aligned and contrastive parameterizations preserve clearer inter-emotion gradient estimates than optimizing both directions with one parameterization.The two branches share a diffusion architecture but have distinct optimization objectives.Source: §3.2.1 · p. 3
3.2.2. Inference with Emo-BPO
- Inference subtracts the contrastive score from the amplified aligned score, while late-step guidance preserves coarse acoustic structure early and strengthens fine-grained emotional cues later.The guidance schedule is ω(t) = 1 − t/T.Source: §3.2.2 · p. 3
4. Experiments
Experiments on two English emotional-speech corpora assess objective and listener-rated speech quality, emotional alignment, and intelligibility. Emo-BPO shows stronger emotional and prosodic performance, while ablations highlight the roles of its contrastive branch and late guidance scheduling.
4.1. Datasets and Experimental Setup
- Training uses ESD and EmoVoiceDB, with preference pairs formed from identical text by treating target-emotion samples as preferred and other-emotion samples as dispreferred.ESD contains 17,500 English utterances across five emotions; EmoVoiceDB contains approximately 22,100 emotion-labeled utterances.Source: §4.1 · p. 3
- The Grad-TTS setup updates only the diffusion decoder and uses weaker early guidance that increases later to refine emotional prosody.The text encoder and duration predictor are frozen; synthesis uses a pretrained HiFi-GAN vocoder.Source: §4.1 · p. 3
4.2. Evaluation Metrics
- Listeners prefer Emo-BPO over EmoSpeech and CosyVoice2, and it achieves the best reported emotion similarity, prosody similarity, UTMOS, and mean emotion-recognition accuracy.CosyVoice2 has the lowest WER, while Emo-BPO maintains competitive intelligibility.Source: Figure 2 · p. 4, §4.2 · p. 4
4.3. Ablation Study.
- Late scheduling achieves the highest converged F0V, higher prosody similarity, and lowest WER; early scheduling degrades prosody and causes a transient WER spike.The comparison includes constant, early, and late schedules; strong early guidance is associated with instability during high-noise denoising.Source: Figure 3 · p. 4, §4.3 · p. 4
4.2. Evaluation Metrics
- Evaluation combines WER, AutoPCP prosody similarity, emotion similarity and recognition accuracy, UTMOS, and listener ratings and preference tests.Subjective evaluations involve 30 participants and use emotion-balanced samples.Source: §4.2 · p. 4
4.3. Ablation Study.
- Removing the contrastive branch reduces emotion similarity, while removing scheduling degrades prosody similarity and increases WER.The ablation attributes weaker inter-emotion discrimination to branch removal and unstable denoising guidance to scheduling removal.Source: §4.3 · p. 4
5. Conclusion
Emo-BPO jointly models desired emotions and competing emotional tendencies using reordered same-text pairs and progressive guidance. Experiments report improved emotional alignment, prosodic control, and perceptual quality with competitive intelligibility.
5. Conclusion
- Emo-BPO learns complementary emotion-aligned and emotion-contrastive score functions from reordered same-text pairs and combines them with progressive guidance at inference.The framework targets a limitation of preference-driven methods that reinforce target emotions without explicitly separating competing emotional tendencies under CFG.Source: §5 · p. 4
- Experiments report improvements in emotional alignment, prosodic control, and perceptual quality while maintaining competitive intelligibility.The conclusion attributes controllable and expressive emotional speech generation to jointly modeling emotional attraction and separation.Source: §5 · p. 4
6. Generative AI Use Disclosure
Generative AI tools were used for manuscript language editing, not for research content generation; the authors reviewed and approved the final manuscript and accept responsibility for it.
6. Generative AI Use Disclosure
- Generative AI tools were used only for grammar correction, polishing, and readability improvement.The disclosure excludes research ideas, methodology, results, analyses, conclusions, and significant portions of the manuscript from AI generation.Source: §6 · p. 5
- The authors state that AI tools did not generate research ideas, methods, experimental results, analyses, or conclusions.Source: §6 · p. 5
- All authors reviewed and approved the final manuscript and take full responsibility for its content.Source: §6 · p. 5