PaperlayerView exact revision on arXiv

Source-linked AI summary

EmoTra-TTS: Smooth Intra-Utterance Emotion Transitions for Speech Synthesis

Tianchi Liu, Zeyang Song, Tianrui Wang, Zhipeng Li, Chenglin Xu, Yiwen Guo

arXiv:2608.23791v1eess.AScs.AI

TL;DR

Emotional TTS often lacks explicit control over continuous emotion transitions within an utterance, despite affect unfolding dynamically over time. EmoTra-TTS addresses this with synthetic frame-aligned transition data, dual-stage VAD conditioning, and bounded direction–magnitude injection, achieving stronger transition evaluations with minimal overhead. Its experiments remain limited in speaker and language diversity and do not demonstrate zero-shot generalization to unseen speakers.

  • Problem

    Existing emotional TTS provides limited explicit control of continuous intra-utterance emotion trajectories, while natural transition data is scarce.

  • Method

    EmoTra-TTS combines multi-pass flow blending, dual-stage VAD conditioning, and direction–magnitude decoupled injection for transition synthesis and control.

  • Results

    30%–87% relative improvement in emotion transition smoothness and 64.4%–79.5% pairwise preference win rates were achieved, with +0.43% parameters and no latency overhead.

  • Takeaways & Limitations

    The system supports modeling emotion as a temporal process rather than a static attribute while retaining bounded conditioning influence.

  • Takeaways & Limitations

    Experiments use EmoVoice-DB with limited speaker and language coverage, and zero-shot generalization to arbitrary unseen speakers is not demonstrated.

Abstract

from arXiv · show

Psychological research on emotion dynamics has established that human affect is a continuous, evolving process: emotions rise, decay, and transition within seconds. Current emotional text-to-speech (TTS) systems, however, condition on a single discrete label or static embedding per utterance, fundamentally misaligning with the temporal nature of affect. While recent LLM-based TTS systems may implicitly vary prosody through text understanding, such variation is neither explicitly controllable nor precise enough for targeted intra-utterance transitions. We address three challenges: (1) a multi-pass flow blending pipeline synthesizes frame-aligned transition audio, circumventing the scarcity of natural intra-utterance transitions; (2) dual-stage Valence-Arousal-Dominance (VAD) conditioning guides prosodic planning in the LLM and acoustic realization in the flow decoder via frame-level VAD embeddings; (3) direction-magnitude decoupled injection structurally separates emotion direction from injection magnitude, preventing content degradation. EmoTra-TTS adds only +0.43% parameters with no latency overhead, achieves 30%-87% relative improvement on emotion transition quality, corroborated by 64.4%-79.5% overall win rates in pairwise preference tests against four SOTA baselines and two commercial systems.

1 Introduction

EmoTra-TTS targets the mismatch between continuous intra-utterance emotion dynamics and predominantly static emotional TTS control. It combines synthetic transition data, dual-stage VAD conditioning, and direction–magnitude decoupling to support smoother, controllable emotion transitions.

  • Motivation: Human affect continuously rises, decays, and transitions, motivating emotion control that varies across both emotion space and time.
  • Motivation: Existing emotional TTS mainly uses one categorical label per utterance, while newer utterance- and word-level controls still miss continuous within-utterance dynamics.
  • Approach: Synthetic emotion transition data uses multi-pass flow blending to generate frame-aligned transition audio despite scarce natural transition recordings.
  • Approach: Dual-stage VAD conditioning guides prosodic planning in the LLM and acoustic realization through frame-level modulation in the flow decoder.
  • Approach: Direction–magnitude decoupled injection separates learned emotion direction from bounded injection strength to mitigate the content-expressiveness trade-off.

2 Related Work

Related work frames EmoTra-TTS within research on continuous emotion dynamics, fine-grained emotional TTS control, synthetic speech data, LLM-based flow-decoder TTS, and conditioning of frozen models. The paper distinguishes user-defined temporal VAD trajectories from discrete or reference-driven control and emphasizes bounded conditioning influence.

  • Emotion Dynamics and Emotional TTS: Affective states have temporal microstructure, motivating VAD trajectories rather than isolated VAD points as conditioning signals.
  • Emotion Dynamics and Emotional TTS: Existing emotional TTS spans discrete embeddings, prosody transfer, continuous VAD control, activation steering, reinforcement learning, and finer-grained reference-based or segment-level modeling.
  • Emotion Dynamics and Emotional TTS: User-defined continuous VAD trajectories remain underexplored, while benchmarks report that commercial systems struggle with genuine intra-utterance emotion changes.
  • Synthetic Transition Data: Mel-space flow blending extends synthetic-data approaches to within-utterance transition synthesis that existing emotional speech corpora cannot provide.
  • LLM-Based TTS: CosyVoice2 uses supervised semantic tokens with a causal conditional flow-matching decoder, providing EmoTra-TTS's LLM-based TTS backbone.
  • Conditioning Injection in Frozen Models: Conditioning methods for frozen models commonly control influence strength through mechanisms such as FiLM, zero-initialization, LoRA, and DoRA.

3 Method

EmoTra-TTS models intra-utterance emotion as a smooth transition between VAD states rather than a static label. It combines synthetic frame-aligned training data with temporal VAD conditioning in the LLM and flow decoder, plus bounded emotion injection.

  • Problem formulation: EmoTra-TTS represents an utterance as a smooth transition from an initial VAD state to a resultant VAD state.The VAD space supports interpolation and connects emotion dimensions to acoustic correlates.
  • Synthetic emotion transition data: Synthetic transition data uses content-invariant flow decoding and mel-spectrogram blending to create temporally aligned emotion pairs.The same speech-token sequence is decoded under different emotion conditions before blending.
  • Synthetic emotion transition data: The multi-pass pipeline supports arbitrary emotion pairs, controlled transition timing and shape, and scalable generation of thousands of samples per hour.Whisper-based CER/WER filtering removes samples with compromised content near transition boundaries.
  • Dual-stage VAD conditioning: Temporal VAD tokens condition the LLM with endpoint and interpolated states, while a projection network maps VAD vectors into the LLM embedding space.Intermediate tokens provide explicit transition trajectories instead of relying only on endpoint conditioning.
  • Dual-stage VAD conditioning: Frame-level VAD embeddings modulate the flow decoder through the speaker pathway, reusing frozen Stage 1 modules for cross-stage perceptual consistency.Piecewise linear interpolation matches inference-time conditioning to the temporal structure of synthetic training data.
  • Direction–magnitude decoupled injection: Direction–magnitude decoupled injection uses LayerNorm and a fixed scale to learn emotion direction while keeping injection magnitude bounded.This addresses content degradation caused by unconstrained emotion-vector norms exceeding the speaker embedding norm.

4 Experimental Setup

The experiments use CosyVoice2-0.5B with filtered EmoVoice-DB speech and approximately 100K synthetic transition utterances. Evaluation combines blind listening metrics, speaker similarity, WER, pairwise preferences, and prosodic measures.

  • Data and model: CosyVoice2-0.5B is the base model, and EmoVoice-DB supplies five speakers across seven emotion categories.The corpus is filtered using a pretrained wav2vec2-based VAD predictor and literature-informed acceptance ranges.
  • Data and model: Approximately 100K synthetic utterances cover all emotion-pair combinations, using τCER = 0.10 and δvad = 0.35 for quality and transition filtering.The data are split 90/10 for training and validation.
  • Training: Stage 1 trains VAD projection and reconstruction modules, while Stage 2 freezes the decoder and trains only the injection MLP and LayerNorm affine parameters.Stage 2 uses approximately 280K trainable parameters and fixed emotion scale ϵ=0.07.
  • Evaluation: A blind listening test with 21 professional raters reports MOS-Qua, MOS-Emo, MOS-Tra, speaker similarity, and WER.MOS-Tra specifically measures emotion-transition smoothness.
  • Evaluation: Two pairwise preference tests compare against open-source baselines and commercial systems, reporting win rates, 95% confidence intervals, and inter-rater agreement.The tests use partially overlapping rater pools and different pair counts.

5 Results and Analysis

EmoTra-TTS improves intra-utterance emotion transition quality through dual-stage conditioning and direction–magnitude-controlled injection, while preserving competitive speech quality. It outperforms open-source and commercial alternatives in transition and overall preference evaluations, with negligible inference overhead.

  • Injection architectures: Dir–Mag achieves the best emotion performance, with MOS-Tra 3.63 and MOS-Emo 3.52, while maintaining MOS-Qua 3.54, SIM 0.701, and WER 1.77%.Linear preserves speech quality better but achieves weaker emotion performance, illustrating the content–expressiveness trade-off.
  • Open-source comparisons: Neutral baselines reach MOS-Qua up to 4.02 but produce minimal emotion rendering and transition quality.MOS-Qua primarily reflects speech quality, so high neutral-condition scores do not indicate emotional correctness.
  • Open-source comparisons: MOSS-TTS reaches the highest baseline MOS-Emo of 3.89 but the lowest MOS-Tra of 1.94, showing that stronger expressiveness can expose boundary discontinuities.Prompt-based concatenation improves emotion rendering by cloning segments from matching references, but creates perceptual discontinuities at boundaries.
  • Open-source comparisons: All listed systems achieve MOS-Tra ≤2.80, indicating that controllable smooth intra-utterance emotion transition remains unresolved.Word-level control in WeSCon yields MOS-Tra 2.39 because discrete assignments lack smooth transitions between segments.
  • Dual-stage SFT: Flow SFT raises MOS-Emo from 2.70 to 3.52 and MOS-Tra from 2.58 to 3.63 after LLM-only SFT, indicating complementary planning and acoustic-realization contributions.LLM-only SFT first improves MOS-Emo from 2.53 to 2.70 and MOS-Tra from 2.32 to 2.58 while preserving speech quality.
  • Fixed-scale ablation: At ϵ = 0.07, r ≈1.0 yields MOS-Tra 3.63; ϵ = 0.10 produces catastrophic degradation as emotion injection overwhelms content and speaker identity.The deployed checkpoint measures an injection norm of approximately 0.667 versus a speaker norm of approximately 0.622, while larger unconstrained norms collapse content.
  • Pairwise preference: EmoTra-TTS is preferred over four baselines with 67.5%–79.5% win rates, and every 95% CI lower bound exceeds 60%.Inter-rater agreement ranges from 63.5% to 71.6%.
  • Commercial comparisons: EmoTra-TTS is preferred over both commercial systems overall at 72.0% and 64.4%, with emotion and transition preference margins of 74.5% and 70.5%.Every 95% CI lower bound exceeds the 50% chance level.

6 Conclusion

EmoTra-TTS models emotion as a temporal process within utterances through synthetic transition data, dual-stage VAD conditioning, and constrained injection. It achieves strong perceptual improvements with minimal added parameters and no latency overhead.

  • EmoTra-TTS combines multi-pass flow blending, dual-stage VAD conditioning, and direction–magnitude decoupled injection to model intra-utterance emotion trajectories.The design targets synthetic transition data, complementary prosodic and acoustic control, and the content–expressiveness trade-off.
  • 30%–87% relative improvement in emotion transition smoothness is reported over multiple SOTA TTS baselines and commercial systems.

Limitations

The evaluation is limited by speaker and language coverage, inconsistent control for subtle VAD differences, piecewise linear emotion trajectories, and unresolved conditioning-tolerance characterization.

  • EmoVoice-DB provides limited speaker diversity and language coverage, and zero-shot generalization to arbitrary unseen speakers is not demonstrated.Experiments train a separate model per voice.
  • Emotion control stability can be inconsistent for subtle VAD differences.The authors suggest more diverse training data may improve this issue.
  • Piecewise linear VAD interpolation provides only a first-order approximation of natural emotion dynamics.More complex nonlinear trajectories are identified as a possible direction for realistic affective evolution.
  • Formal characterization of conditioning tolerance boundaries in frozen generative models remains an open research problem.

Ethical Statement

The study uses synthesized speech and synthetic transition data without private or personally identifiable data, while restricting release to non-commercial academic research and acknowledging misuse risks.

  • All evaluators were paid according to local labor regulations and institutional guidelines, with no personally identifiable data collected.The listening task involved subjective quality ratings of AI-synthesized speech.
  • Training and transition data come from the publicly available, entirely AI-generated EmoVoice-DB corpus.The paper states that no private, user-uploaded, or personally identifiable data are used.
  • Code and model pipelines are intended for release under an open-source license restricted to non-commercial academic research.
  • The authors identify deceptive or manipulative audio generation as a potential misuse risk and encourage human oversight.
  • AI language models generated test sentences and assisted with language polishing, while the authors conducted the experimental and scientific work.

A Training Details

The appendix reports synthetic-data, LLM-training, flow-training, evaluation-interface, and listening-test details, including fixed interpolation, optimization, filtering, and assessment settings.

  • Synthetic data generation: Synthetic transition data use crossfade steepness κ = 12, segment ratios 60%/15%/25%, capped crossfade duration, and a Whisper large-v3 CER threshold of 0.10.
  • Stage 1 training (LLM): Stage 1 LLM training uses Adam with learning rate 1.0 × 10−5, gradient clipping at 5.0, and three intermediate VAD tokens.
  • Stage 2 training (Flow): Stage 2 flow training freezes decoder parameters and uses gradient accumulation of 2 with fixed emotion scale ϵ = 0.07.
  • Evaluation: The blind-test interface collects MOS-Qua, MOS-Emo, and MOS-Tra ratings on 1–5 Likert scales with hidden, randomly shuffled system identities.
  • Stage 2 training (Flow): The reported flow projection maps 1024 dimensions through 256 and ReLU to 80 mel dimensions, with the last layer zero-initialized.
  • Evaluation cost: The two commercial paired-comparison systems incurred a total evaluation cost of USD 11.
  • Listening evaluation: Blind listening evaluations used a professional paid team with quality-control procedures and 21 raters, including researchers and non-technical listeners.

B.1 MOS-based Evaluation Protocol

The MOS-based evaluation uses a blinded, randomly sampled protocol combining expert and general-listener ratings to assess speech quality, emotion rendering, and transition smoothness. Its test set contains independently generated, event-driven emotion-transition sentences with balanced emotion-pair coverage and VAD annotations.

  • Evaluation Design: 10 evaluators rated approximately 250 utterances per system, combining 3 researchers and 7 non-technical listeners.Each system contained 200 utterances, and each evaluator rated about 25 randomly sampled utterances per system.
  • Test Set: The test set was generated with event-driven narratives, balanced emotion-pair coverage, and VAD annotations, independently of the EmoVoice-DB training set.The generation prompt targeted at least 25 of 30 possible emotion pairs, with no pair appearing more than 15 times among 200 samples.
  • Evaluation Design: All samples were randomly shuffled and system identities were hidden, while the evaluation achieved an approximately 94% completion rate.The protocol prioritized careful ratings over raw throughput.
  • Evaluation Metrics: Three 1–5 Likert metrics separately measured speech naturalness, emotion-rendering adequacy, and intra-utterance transition smoothness.MOS-Emo evaluates both initial and resultant emotions, whereas MOS-Tra assesses gradualness rather than abruptness.
  • Commercial Comparison: The commercial comparison used 12 raters and collected dimension-specific preferences for emotion accuracy, transition smoothness, and overall preference.ElevenLabs v3 was manually configured with discrete emotion tags and its “Stability” control set to “Creative.”

D Objective Smoothness Evaluation

The paper introduces Prosodic Jerk Ratio, a signal-level f0 metric for detecting abrupt pitch changes. EmoTra-TTS achieves the lowest ratio and is smoother than both baselines and its synthetic training data.

  • Metric: Prosodic Jerk Ratio (JR-F0) measures the fraction of voiced frames containing abrupt second-order f0 discontinuities.The metric is computed directly from Praat-derived f0 contours using an adaptive per-utterance threshold τ.
  • Metric: Lower JR-F0 indicates smoother prosodic evolution.The metric complements perceptual evaluation with a fully signal-level measure.
  • Results: EmoTra-TTS attains the lowest JR-F0 among the proposed system and baselines, with all differences statistically significant at p < 0.001.The comparison covers Categories (d) and (e) of Table 2.
  • Results: 0.025 vs. 0.030: EmoTra-TTS is smoother than its own synthetic training data.The synthetic corpus retains residual discontinuities from four-segment blending, whereas frame-level interpolated VAD conditioning yields globally continuous prosody.
  • Interpretation: The injection-architecture study links content failures to excessive emotion-conditioning magnitude and supports structurally decoupling emotion direction from magnitude.Manual scaling, norm measurements, and LayerNorm results identify magnitude control as central to preserving content fidelity.

E.5 MLP Compensation (FiLM)

The FiLM and norm-clamping experiments show that soft or discontinuous magnitude constraints can undermine auxiliary conditioning. LayerNorm with fixed scaling preserves content while retaining strong emotion and transition quality.

  • FiLM Compensation: FiLM achieves MOS-Qua 1.96, WER 53.35%, and SIM 0.400 because joint optimization inflates its output magnitude 3–5×.The MLP compensates for FiLM’s learnable scale and shift despite identity initialization and a frozen decoder.
  • FiLM Compensation: Soft constraints are insufficient because the flow loss does not penalize magnitude growth, allowing the MLP and FiLM parameters to co-adapt around the constraint.This explains why learnable scale control does not reliably preserve the decoder’s operating range.
  • Norm Clamp: Hard norm clamping improves content to MOS-Qua 3.29 and WER 3.89%, but MOS-Emo remains 3.30 because clamping suppresses gradients beyond the threshold.The discontinuity between clamped and unclamped regimes causes oscillation near the boundary.
  • LayerNorm: LayerNorm with fixed ϵ=0.07 achieves MOS-Qua 3.54, MOS-Emo 3.52, and MOS-Tra 3.63 while keeping the decoder frozen.Its fixed magnitude prevents inflation while allowing the MLP to learn emotion directions.
  • Crossfade Selection: The selected sigmoid blending curve uses κ=12 to balance gradual transitions against emotion purity.Linear and cosine blending sacrifice contrast or quality, whereas κ≥100 collapses toward a hard switch.
Loading 2608.23791v1…