EmoTra-TTS: Smooth Intra-Utterance Emotion Transitions for Speech Synthesis

Motivation

• Human affect continuously rises, decays, and transitions, motivating emotion control that varies across both emotion space and time.Source: §1 · p. 1
• Existing emotional TTS mainly uses one categorical label per utterance, while newer utterance- and word-level controls still miss continuous within-utterance dynamics.Source: §1 · p. 1

Approach

• Synthetic emotion transition data uses multi-pass flow blending to generate frame-aligned transition audio despite scarce natural transition recordings.Source: §1 · p. 2
• Dual-stage VAD conditioning guides prosodic planning in the LLM and acoustic realization through frame-level modulation in the flow decoder.Source: §1 · pp. 1–2
• Direction–magnitude decoupled injection separates learned emotion direction from bounded injection strength to mitigate the content-expressiveness trade-off.Source: §1 · p. 2

Emotion Dynamics and Emotional TTS

• Affective states have temporal microstructure, motivating VAD trajectories rather than isolated VAD points as conditioning signals.Source: §2.1 · p. 2
• Existing emotional TTS spans discrete embeddings, prosody transfer, continuous VAD control, activation steering, reinforcement learning, and finer-grained reference-based or segment-level modeling.Source: §2.1 · p. 2
• User-defined continuous VAD trajectories remain underexplored, while benchmarks report that commercial systems struggle with genuine intra-utterance emotion changes.Source: §2.1 · p. 2

Synthetic Transition Data

• Mel-space flow blending extends synthetic-data approaches to within-utterance transition synthesis that existing emotional speech corpora cannot provide.Source: §2.2 · p. 2

LLM-Based TTS

• CosyVoice2 uses supervised semantic tokens with a causal conditional flow-matching decoder, providing EmoTra-TTS's LLM-based TTS backbone.Source: §2.3 · p. 2

Conditioning Injection in Frozen Models

• Conditioning methods for frozen models commonly control influence strength through mechanisms such as FiLM, zero-initialization, LoRA, and DoRA.Source: §2.4 · pp. 2–3

Problem formulation

• EmoTra-TTS represents an utterance as a smooth transition from an initial VAD state to a resultant VAD state.The VAD space supports interpolation and connects emotion dimensions to acoustic correlates.Source: §3.1 · p. 3

Synthetic emotion transition data

• Synthetic transition data uses content-invariant flow decoding and mel-spectrogram blending to create temporally aligned emotion pairs.The same speech-token sequence is decoded under different emotion conditions before blending.Source: §3.2.1 · p. 3, §3.2.2 · p. 3
• The multi-pass pipeline supports arbitrary emotion pairs, controlled transition timing and shape, and scalable generation of thousands of samples per hour.Whisper-based CER/WER filtering removes samples with compromised content near transition boundaries.Source: §3.2.3 · p. 4, §3.2.2 · p. 4

Dual-stage VAD conditioning

• Temporal VAD tokens condition the LLM with endpoint and interpolated states, while a projection network maps VAD vectors into the LLM embedding space.Intermediate tokens provide explicit transition trajectories instead of relying only on endpoint conditioning.Source: §3.3.2 · p. 4, §3.3.1 · p. 4
• Frame-level VAD embeddings modulate the flow decoder through the speaker pathway, reusing frozen Stage 1 modules for cross-stage perceptual consistency.Piecewise linear interpolation matches inference-time conditioning to the temporal structure of synthetic training data.Source: §3.4.3 · p. 5, §3.4.1 · p. 5

Direction–magnitude decoupled injection

• Direction–magnitude decoupled injection uses LayerNorm and a fixed scale to learn emotion direction while keeping injection magnitude bounded.This addresses content degradation caused by unconstrained emotion-vector norms exceeding the speaker embedding norm.Source: §3.4.4 · p. 6

Data and model

• CosyVoice2-0.5B is the base model, and EmoVoice-DB supplies five speakers across seven emotion categories.The corpus is filtered using a pretrained wav2vec2-based VAD predictor and literature-informed acceptance ranges.Source: §4 · p. 6
• Approximately 100K synthetic utterances cover all emotion-pair combinations, using τCER = 0.10 and δvad = 0.35 for quality and transition filtering.The data are split 90/10 for training and validation.Source: §4 · p. 6

Training

• Stage 1 trains VAD projection and reconstruction modules, while Stage 2 freezes the decoder and trains only the injection MLP and LayerNorm affine parameters.Stage 2 uses approximately 280K trainable parameters and fixed emotion scale ϵ=0.07.Source: §4 · p. 6

Evaluation

• A blind listening test with 21 professional raters reports MOS-Qua, MOS-Emo, MOS-Tra, speaker similarity, and WER.MOS-Tra specifically measures emotion-transition smoothness.Source: §4 · p. 6
• Two pairwise preference tests compare against open-source baselines and commercial systems, reporting win rates, 95% confidence intervals, and inter-rater agreement.The tests use partially overlapping rater pools and different pair counts.Source: §4 · p. 7

Injection architectures

• Dir–Mag achieves the best emotion performance, with MOS-Tra 3.63 and MOS-Emo 3.52, while maintaining MOS-Qua 3.54, SIM 0.701, and WER 1.77%.Linear preserves speech quality better but achieves weaker emotion performance, illustrating the content–expressiveness trade-off.Source: §5.1 · p. 7

Open-source comparisons

• Neutral baselines reach MOS-Qua up to 4.02 but produce minimal emotion rendering and transition quality.MOS-Qua primarily reflects speech quality, so high neutral-condition scores do not indicate emotional correctness.Source: §5.2 · p. 7
• MOSS-TTS reaches the highest baseline MOS-Emo of 3.89 but the lowest MOS-Tra of 1.94, showing that stronger expressiveness can expose boundary discontinuities.Prompt-based concatenation improves emotion rendering by cloning segments from matching references, but creates perceptual discontinuities at boundaries.Source: §5.2 · p. 7
• All listed systems achieve MOS-Tra ≤2.80, indicating that controllable smooth intra-utterance emotion transition remains unresolved.Word-level control in WeSCon yields MOS-Tra 2.39 because discrete assignments lack smooth transitions between segments.Source: §5.2 · p. 7

Dual-stage SFT

• Flow SFT raises MOS-Emo from 2.70 to 3.52 and MOS-Tra from 2.58 to 3.63 after LLM-only SFT, indicating complementary planning and acoustic-realization contributions.LLM-only SFT first improves MOS-Emo from 2.53 to 2.70 and MOS-Tra from 2.32 to 2.58 while preserving speech quality.Source: §5.3 · p. 8

Fixed-scale ablation

• At ϵ = 0.07, r ≈1.0 yields MOS-Tra 3.63; ϵ = 0.10 produces catastrophic degradation as emotion injection overwhelms content and speaker identity.The deployed checkpoint measures an injection norm of approximately 0.667 versus a speaker norm of approximately 0.622, while larger unconstrained norms collapse content.Source: §5.4 · pp. 8–9

Pairwise preference

• EmoTra-TTS is preferred over four baselines with 67.5%–79.5% win rates, and every 95% CI lower bound exceeds 60%.Inter-rater agreement ranges from 63.5% to 71.6%.Source: §5.5 · p. 9, Figure 3 · p. 9

Commercial comparisons

• EmoTra-TTS is preferred over both commercial systems overall at 72.0% and 64.4%, with emotion and transition preference margins of 74.5% and 70.5%.Every 95% CI lower bound exceeds the 50% chance level.Source: §5.6 · p. 9

• EmoTra-TTS combines multi-pass flow blending, dual-stage VAD conditioning, and direction–magnitude decoupled injection to model intra-utterance emotion trajectories.The design targets synthetic transition data, complementary prosodic and acoustic control, and the content–expressiveness trade-off.Source: §6 · p. 9
• 30%–87% relative improvement in emotion transition smoothness is reported over multiple SOTA TTS baselines and commercial systems.Source: §6 · p. 9

• EmoVoice-DB provides limited speaker diversity and language coverage, and zero-shot generalization to arbitrary unseen speakers is not demonstrated.Experiments train a separate model per voice.Source: Limitations · p. 10
• Emotion control stability can be inconsistent for subtle VAD differences.The authors suggest more diverse training data may improve this issue.Source: Limitations · p. 10
• Piecewise linear VAD interpolation provides only a first-order approximation of natural emotion dynamics.More complex nonlinear trajectories are identified as a possible direction for realistic affective evolution.Source: Limitations · p. 10
• Formal characterization of conditioning tolerance boundaries in frozen generative models remains an open research problem.Source: Limitations · p. 10

• All evaluators were paid according to local labor regulations and institutional guidelines, with no personally identifiable data collected.The listening task involved subjective quality ratings of AI-synthesized speech.Source: Ethical Statement · p. 10
• Training and transition data come from the publicly available, entirely AI-generated EmoVoice-DB corpus.The paper states that no private, user-uploaded, or personally identifiable data are used.Source: Ethical Statement · p. 10
• Code and model pipelines are intended for release under an open-source license restricted to non-commercial academic research.Source: Ethical Statement · p. 10
• The authors identify deceptive or manipulative audio generation as a potential misuse risk and encourage human oversight.Source: Ethical Statement · p. 10
• AI language models generated test sentences and assisted with language polishing, while the authors conducted the experimental and scientific work.Source: Ethical Statement · p. 10

Synthetic data generation

• Synthetic transition data use crossfade steepness κ = 12, segment ratios 60%/15%/25%, capped crossfade duration, and a Whisper large-v3 CER threshold of 0.10.Source: A Training Details · p. 14

Stage 1 training (LLM)

• Stage 1 LLM training uses Adam with learning rate 1.0 × 10−5, gradient clipping at 5.0, and three intermediate VAD tokens.Source: A Training Details · p. 14

Stage 2 training (Flow)

• Stage 2 flow training freezes decoder parameters and uses gradient accumulation of 2 with fixed emotion scale ϵ = 0.07.Source: A Training Details · p. 14

Evaluation

• The blind-test interface collects MOS-Qua, MOS-Emo, and MOS-Tra ratings on 1–5 Likert scales with hidden, randomly shuffled system identities.Source: Figure 5 · p. 15

Stage 2 training (Flow)

• The reported flow projection maps 1024 dimensions through 256 and ReLU to 80 mel dimensions, with the last layer zero-initialized.Source: A Training Details · p. 15

Evaluation cost

• The two commercial paired-comparison systems incurred a total evaluation cost of USD 11.Source: A Training Details · p. 15

Listening evaluation

• Blind listening evaluations used a professional paid team with quality-control procedures and 21 raters, including researchers and non-technical listeners.Source: B Evaluation Protocol · p. 15

Evaluation Design

• 10 evaluators rated approximately 250 utterances per system, combining 3 researchers and 7 non-technical listeners.Each system contained 200 utterances, and each evaluator rated about 25 randomly sampled utterances per system.Source: §B.1 · p. 15

Test Set

• The test set was generated with event-driven narratives, balanced emotion-pair coverage, and VAD annotations, independently of the EmoVoice-DB training set.The generation prompt targeted at least 25 of 30 possible emotion pairs, with no pair appearing more than 15 times among 200 samples.Source: §B.1 · p. 15, §B.5 · p. 17

Evaluation Design

• All samples were randomly shuffled and system identities were hidden, while the evaluation achieved an approximately 94% completion rate.The protocol prioritized careful ratings over raw throughput.Source: §B.1 · p. 15, Figure 6 · p. 15

Evaluation Metrics

• Three 1–5 Likert metrics separately measured speech naturalness, emotion-rendering adequacy, and intra-utterance transition smoothness.MOS-Emo evaluates both initial and resultant emotions, whereas MOS-Tra assesses gradualness rather than abruptness.Source: §B.1 · p. 15

Commercial Comparison

• The commercial comparison used 12 raters and collected dimension-specific preferences for emotion accuracy, transition smoothness, and overall preference.ElevenLabs v3 was manually configured with discrete emotion tags and its “Stability” control set to “Creative.”Source: §B.3 · pp. 16–17

Metric

• Prosodic Jerk Ratio (JR-F0) measures the fraction of voiced frames containing abrupt second-order f0 discontinuities.The metric is computed directly from Praat-derived f0 contours using an adaptive per-utterance threshold τ.Source: D Objective Smoothness Evaluation · pp. 17–18
• Lower JR-F0 indicates smoother prosodic evolution.The metric complements perceptual evaluation with a fully signal-level measure.Source: D Objective Smoothness Evaluation · pp. 17–18

Results

• EmoTra-TTS attains the lowest JR-F0 among the proposed system and baselines, with all differences statistically significant at p < 0.001.The comparison covers Categories (d) and (e) of Table 2.Source: D Objective Smoothness Evaluation · p. 18, Table 7 · p. 18
• 0.025 vs. 0.030: EmoTra-TTS is smoother than its own synthetic training data.The synthetic corpus retains residual discontinuities from four-segment blending, whereas frame-level interpolated VAD conditioning yields globally continuous prosody.Source: D Objective Smoothness Evaluation · p. 18

Interpretation

• The injection-architecture study links content failures to excessive emotion-conditioning magnitude and supports structurally decoupling emotion direction from magnitude.Manual scaling, norm measurements, and LayerNorm results identify magnitude control as central to preserving content fidelity.Source: E Detailed Failure Analysis · p. 18, §E.1 · p. 18, §E.2 · p. 18

FiLM Compensation

• FiLM achieves MOS-Qua 1.96, WER 53.35%, and SIM 0.400 because joint optimization inflates its output magnitude 3–5×.The MLP compensates for FiLM’s learnable scale and shift despite identity initialization and a frozen decoder.Source: §E.5 · p. 18
• Soft constraints are insufficient because the flow loss does not penalize magnitude growth, allowing the MLP and FiLM parameters to co-adapt around the constraint.This explains why learnable scale control does not reliably preserve the decoder’s operating range.Source: §E.5 · pp. 18–19

Norm Clamp

• Hard norm clamping improves content to MOS-Qua 3.29 and WER 3.89%, but MOS-Emo remains 3.30 because clamping suppresses gradients beyond the threshold.The discontinuity between clamped and unclamped regimes causes oscillation near the boundary.Source: §E.6 · p. 19

LayerNorm

• LayerNorm with fixed ϵ=0.07 achieves MOS-Qua 3.54, MOS-Emo 3.52, and MOS-Tra 3.63 while keeping the decoder frozen.Its fixed magnitude prevents inflation while allowing the MLP to learn emotion directions.Source: §E.7 · p. 19

Crossfade Selection

• The selected sigmoid blending curve uses κ=12 to balance gradual transitions against emotion purity.Linear and cosine blending sacrifice contrast or quality, whereas κ≥100 collapses toward a hard switch.Source: F Crossfade Curve Selection · p. 19, Table 9 · p. 19