PaperlayerView exact revision on arXiv

Source-linked AI summary

EmoRES-TTS: Residual-Enhanced Vector Steering for Emotional Speech Generation

Kuan-Po Huang, Haohe Liu, Puyuan Peng, Haibin Wu, Zhaoheng Ni, Hung-yi Lee, Jinwon Lee, Neha Chachra

arXiv:2609.38157v1cs.SDcs.CLeess.AS

TL;DR

Emotion-conditioned TTS may not reliably express a requested emotion, while conventional steering couples shared and category-specific changes under one strength. EmoRES separates and independently controls those components without retraining; on IEMOCAP it improves emotion-control measures over CoCoEmo on two TTS backbones.

  • Problem

    Native emotion conditioning does not always strongly match requested emotions, and conventional steering couples shared and category-specific components under one global strength.

    Source: §1 · p. 2

  • Method

    EmoRES decomposes emotion vectors into shared and residual components and controls them independently without updating TTS model parameters.

    Source: §1 · p. 2

  • Results

    26.13 and 12.97 percentage-point gains in rank correlation and 12.95 and 6.92-point gains in emotion hit rate over CoCoEmo on IEMOCAP, for IndexTTS-2 and CosyVoice2, respectively.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Component ablations and human evaluations support the interpretation that effective emotion control preserves the shared component while strengthening the residual.

    Source: §6 · p. 10

    In the experiments, steering vectors are rescaled to the original emotion-vector norm, so comparisons at fixed strength isolate direction rather than injected magnitude.

    Source: §4.2 · p. 5

Abstract

from arXiv · show

Emotion-conditioned text-to-speech (TTS) models may fail to express the requested emotion reliably, and improving controllability by additional training is costly in both computation and emotion-labeled speech training data. We therefore study vector steering, a training-free approach that modifies the internal representations of a frozen model. CoCoEmo, a conventional vector steering method for emotion TTS, treats each emotion vector as an indivisible direction controlled by a single global strength, limiting adherence to the requested emotion. In this work, we first discover that an emotion vector can be decomposed into a shared component that moves speech away from neutral expression and a residual component that directs generation toward the requested emotion. Building on this finding, we propose Emotion Residual-Enhanced Steering for TTS (EmoRES), a novel method that controls the two components without retraining the backbone. On IEMOCAP, EmoRES outperforms CoCoEmo across all four objective emotion metrics on the IndexTTS-2 and CosyVoice2 backbones. Rank correlation improves by 26.13 and 12.97 percentage points, corresponding to relative gains of 118.8% and 33.1%, while emotion hit rate improves by 12.95 and 6.92 points, corresponding to relative gains of 20.1% and 9.8%. Human evaluation further shows a relative improvement up to 35.0% in the rate at which listeners correctly identified the dominant requested emotion and up to a 17.3% improvement in fidelity, while listeners prefer EmoRES for naturalness in up to 63.8% of pairwise comparisons. Component ablations further demonstrate that effective control benefits from preserving the shared component while strengthening the residual of the emotion steering vectors.

1 Introduction

Emotion TTS needs to convey requested emotions while preserving speech quality, but native conditioning can be weak and conventional vector steering scales shared and emotion-specific direction together. EmoRES separates those components and improves emotion-control measures and human judgments without retraining the backbone.

1 Introduction

  • Emotion control matters in conversational agents, narration, accessibility, and dubbing, where requested expression must coexist with naturalness, intelligibility, and speaker identity.Native emotion conditioning alone does not always produce a strong match to the requested emotion.Source: §1 · p. 1
  • CoCoEmo scales each emotion vector as one unit, coupling movement away from neutral with direction toward a requested category.The shared structure across emotion vectors is left unexamined by that single-strength design.Source: §1 · p. 2
  • The shared component points from neutral activations toward the emotional centroid, while the residual points from that centroid toward the requested emotion.The authors report this as a functional analysis of shared and category-specific structure in LM-based TTS emotion vectors.Source: §1 · p. 2
  • On IEMOCAP, EmoRES raises rank correlation by 26.13 and 12.97 percentage points and emotion hit rate by 12.95 and 6.92 points on IndexTTS-2 and CosyVoice2, respectively.Human evaluation also reports higher dominant-emotion identification and naturalness preference over CoCoEmo on both backbones.Source: §1 · p. 2
  • EmoRES independently controls a shared component and an emotion-specific residual, strengthening the residual relative to the shared component without retraining or changing model parameters.The work reports consistent improvements across multiple backbones, held-out evaluation, and human judgments.Source: §1 · p. 2

2 Related Work

Prior emotional TTS offers several conditioning interfaces, while vector steering provides inference-time control without parameter updates. EmoRES builds on CoCoEmo’s steering framework but addresses the shared structure within categorical emotion directions.

Emotion-controllable text-to-speech

  • Emotion-controllable TTS uses categorical labels, continuous embeddings, natural-language instructions, or expressive reference speech, often relying on emotion-aware training, specialized modules, or reference signals.The cited examples include EmoSphere++, PromptTTS, EmoVoice, IndexTTS-2, and CosyVoice2.Source: §2 · p. 2

Vector steering for speech generation

  • Vector steering changes intermediate representations at inference without updating model parameters, and related TTS work manipulates parameter, speaker, or style embeddings to transfer emotional characteristics.EmoSteer-TTS uses difference-of-means vectors in flow-matching TTS, whereas CoCoEmo steers speech language model representations.Source: §2 · pp. 2–3
  • EmoRES adopts CoCoEmo’s extraction and inference-time steering framework but separates the common displacement from the request-dependent residual in categorical directions.Prior geometric analysis reports more separable and speaker-invariant emotion directions in speech language model representations than in flow-matching representations.Source: §2 · p. 3

Composition and representation structure

  • Existing compound-emotion steering sums categorical directions as indivisible units, while related multi-attribute steering methods use learned or optimized subspaces.EmoRES instead decomposes mean-difference vectors exactly around the emotion-activation centroid and exposes separate controls without learning a new subspace or modifying backbone weights.Source: §2 · p. 3

3 Method

The method extracts emotion directions from emotional-versus-neutral activations and injects them into a frozen TTS backbone at inference. It decomposes each direction into a shared neutral-to-centroid shift and a category-specific residual, then controls those components separately, including for mixed-emotion targets.

Steering vector extraction

  • Emotion vectors are extracted offline by contrasting mean model activations for emotional and neutral speech, then reused across utterances without gradient computation or weight updates.The vector is the average activation displacement separating speech carrying an emotion from neutral speech.Source: §3 · p. 3

Steering

  • At inference, the requested steering vector is added to activations at the extraction site, with its strength controlled by a scalar.The applied update restores each hidden position’s magnitude after addition, leaving the hidden-state norm unchanged while the backbone remains frozen.Source: §3 · p. 3

Shared and residual components

  • Each emotion vector decomposes around the unweighted centroid of emotional-category mean activations into a shared neutral-to-centroid shift and a category-specific residual.The shared vector is common to every request; the residual varies with the requested emotion.Source: §3 · p. 4
  • The shared component moves activations away from neutral and indicates emotional expression, while the residual distinguishes one requested emotion from another.The shared component does not itself specify which emotion is conveyed.Source: §3 · p. 4

EmoRES

  • EmoRES assigns independent coefficients to the shared and residual components, allowing the neutral-to-centroid shift and category-specific contrast to be adjusted separately.When both coefficients equal 1, the centroid terms cancel and the rule reduces exactly to conventional CoCoEmo steering.Source: §3 · p. 4

Mixture targets

  • For mixed-emotion targets, requested emotion proportions determine the residual while the shared shift remains constant across requested emotions.The method represents each requested proportion as a nonnegative value, with proportions summing to 1.Source: §3 · p. 4

4 Experimental Setup

The experiments test steering on two frozen zero-shot TTS backbones using a held-out cross-corpus evaluation design. Objective metrics and listener evaluations assess emotional expression alongside intelligibility, speaker preservation, fidelity, and naturalness.

4.1 Datasets

  • CREMA-D is the development set for tuning, while IEMOCAP is held out for testing and contributes no speakers or recordings to steering-vector construction.IEMOCAP tests transfer across domains.Source: §4.1 · pp. 4–5

4.2 Model steering

  • IndexTTS-2 uses emotion embeddings and CosyVoice 2 uses natural-language prompts; steering modifies selected attention outputs without updating weights or changing default conditioning.The selected layers are the backbones’ top-K emotion-separable layers identified by CoCoEmo.Source: §4.2 · p. 5

4.3.1 Objective evaluation

  • Objective evaluation measures emotion with TEP and E-SIM, speaker preservation with S-SIM, intelligibility with WER, and mixed-emotion ordering and dominance with ρ and H-Rate.The objective measures cover multiple aspects of output quality and requested emotion expression.Source: §4.3.1 · p. 5

4.3.2 Subjective evaluation

  • Human evaluation uses emotion labels to assess Dom-hit and Fidelity, and randomized pairwise judgments to compare naturalness against CoCoEmo; each sample receives at least nine annotations.Fidelity reflects agreement with the full target emotion mixture, while naturalness preference is tie-adjusted.Source: §4.3.2 · p. 5

4.4 Baselines

  • No-steer uses native emotion conditioning, CoCoEmo is recovered at λc = λr = 1, and Random steer uses a norm-matched isotropic direction to isolate direction-specific effects.Figure 1 fixes λc = 1 and varies λr; at each α, steering vectors have equal norms and differ in direction.Source: §4.4 · pp. 5–6, Figure 1 · p. 6

5 Results and Discussion

Objective and listener evaluations compare EmoRES with CoCoEmo and unsteered systems; ablations and embedding visualizations examine how shared and residual components contribute to emotion control.

5.1 Residual-Enhanced Vectors

  • On IEMOCAP, versus CoCoEmo, ρ rises 22.00%→48.13% (IndexTTS-2) and 39.13%→52.10% (CosyVoice2); H-Rate rises 64.34%→77.29% and 70.90%→77.82%.Table 1 compares steered systems on CREMA-D development and IEMOCAP; steered rows hold λc = 1 and vector norm fixed while varying residual strength.Source: Table 1 · p. 7, §5.1 · p. 7
  • On IEMOCAP, listeners’ Dom-hit rises from 54.72% to 73.89% for IndexTTS-2 and from 58.18% to 77.88% for CosyVoice2 under EmoRES versus CoCoEmo.Fidelity also rises from 60.05% to 66.32% for IndexTTS-2 and from 50.80% to 59.60% for CosyVoice2; both measures improve significantly for both backbones.Source: Table 2 · p. 8, §5.1 · p. 8
  • EmoRES improves S-SIM in every comparison and reduces WER under most settings.The embedding visualization assesses generated IndexTTS-2 utterances on the single-emotion CREMA-D subset (n = 486).Source: §5.1 · p. 8, Figure 2 · p. 9

5.3 Visualization of speech emotion embeddings

  • In Figure 2, EmoRES reaches 86.9% probe accuracy and 0.460 silhouette, forming more distinct emotion clusters than CoCoEmo, especially for sad and happy.Residual-only steering reaches 81.4% probe accuracy and 0.379 silhouette, supporting the interpretation that residuals carry category-specific contrasts.Source: Figure 2 · p. 9, §5.3 · p. 10

5.2 Shared and Residual Components

  • Shared-only steering reduces IndexTTS-2’s neutral posterior from 36.51% to 7.49% but gives the lowest ρ and H-Rate; residual-only improves those metrics but lowers TEP and E-SIM.Retaining the shared component while increasing residual strength to λr = 3 gives the best emotion-control results; all ablation conditions are norm-matched.Source: Table 3 · p. 8, §5.2 · p. 9

6 Conclusions and Future Work

EmoRES separates shared and emotion-specific residual components, with ablations and human evaluations supporting their distinct functions. Geometrically, the shared shift reaches the emotional centroid before the residual directs toward a requested emotion.

6 Conclusions and Future Work

  • EmoRES independently controls shared and residual components without fine-tuning; ablations and human evaluations support their distinct functions.Future work will test token- or segment-level control and whether the decomposition generalizes across emotions, languages, speech corpora, and TTS architectures.Source: §6 · p. 10

A Geometric illustration of EmoRES

  • The shared component moves from the neutral mean to the emotional centroid, while each categorical residual points from that centroid toward an emotion-specific mean.The illustration fixes λc = 1; a mixture residual is a convex combination of categorical residuals, tracing a line segment between their endpoints.Source: A Geometric illustration of EmoRES · p. 13
  • At λr = 1, the construction recovers CoCoEmo; increasing λr expands the mixture segment about the centroid and increases the residual’s relative contribution.The main experiments use λr = 3, and vectors are rescaled to a common reference norm before injection, preserving direction.Source: A Geometric illustration of EmoRES · p. 13

B Metrics

The section defines objective and listener-based measures, then describes emotion-vector extraction, steering-layer selection, and the native conditioning interfaces used alongside steering.

B.1 Objective metrics

  • TEP averages recognizer posterior probability over target emotions, while E-SIM measures mixture-weighted cosine similarity to emotion-specific speech anchors.E-SIM uses 100 emotion2vec+ large embeddings of ESD recordings as anchors for each emotion.Source: §B.1 · p. 14
  • ρ measures Spearman rank agreement between requested emotion proportions and steering-induced posterior increases; H-Rate checks whether the dominant requested emotion has the largest increase.H-Rate is computed on utterances with at least two target emotions, and tied dominant targets count as hits if any tied target has the largest increase.Source: §B.1 · p. 14
  • S-SIM measures speaker-embedding cosine similarity to the reference prompt, while WER compares Whisper transcripts with the prompted text.These metrics track speaker similarity and word error rate, respectively.Source: §B.1 · pp. 14–15

B.2 Subjective metrics

  • Dom-hit is the fraction of clips whose listener-assigned label matches the dominant target emotion; listener labels are compared across all corpus emotions.A clip labeled as an emotion that was not requested is a miss.Source: §B.2 · p. 15
  • Fidelity is one minus the total variation distance between listener and target emotion mixtures, while naturalness preference aggregates pairwise judgments with neutral responses scored midway.Fidelity ranges from one for coinciding distributions to zero for disjoint supports.Source: §B.2 · p. 15

C.1 Emotion vector construction

  • Emotion vectors come from matched neutral–emotional pairs in ESD, CREMA-D, and RAVDESS, with IEMOCAP held out; EmoRES decomposes CoCoEmo’s vectors without building a new library.Pairs match speaker and linguistic content, are removed together if either side fails a filter, and produce emotion-specific within-pair means; extraction uses frozen backbones without gradient computation or weight updates.Source: §C.1 · pp. 15–16

C.2 Steering layers

  • For controlled comparisons, steering uses layers 1, 6, and 8 in IndexTTS-2 and layers 14 and 17 in CosyVoice2.Across 24 CosyVoice2 layers on CREMA-D, probe accuracy correlates positively with TEP, and the two highest-accuracy layers are also the two highest-TEP admissible layers.Source: §C.2 · pp. 16–17, Figure 4 · p. 16

C.3 Input conditions

  • Steered systems retain the backbone’s native emotion interface and add vector steering; IndexTTS-2 uses emotion vectors, while CosyVoice2 uses instructions naming emotions in weight order.The IndexTTS-2 conditioning configuration is held constant across composition rules; CosyVoice2 instructions omit emotion proportions and exclude zero-weight emotions.Source: §C.3 · p. 17

C.4 Subjective evaluation

The subjective evaluation used quality-filtered listener annotations for emotion labeling and naturalness comparisons, with paired bootstrap intervals for the reported comparisons.

C.4 Subjective evaluation

  • The evaluation covered 360 IndexTTS-2 and 330 CosyVoice2 samples, each receiving at least nine independent annotations per task after annotator quality checks.The retained annotations were used for Table 2 results.Source: §C.4 · p. 17
  • Emotion-labeling intervals resampled matched CoCoEmo and EmoRES outputs together, while naturalness intervals resampled each comparison with its retained judgments.Both used sample-level bootstrap replicates and the 2.5th and 97.5th percentiles.Source: §C.4 · p. 17
  • Table 4 compares norm-matched steering conditions on CREMA-D mixed-emotion targets at α = 5, marking Gaussian-replaced components as controls.It reports mean ± one standard deviation over five runs and bolds the best unrandomized setting per column and backbone.Source: Table 4 · p. 18

D Component ablation and random-direction controls

Norm-matched random-direction controls show that emotion control depends on the measured directions of both components, with residual randomization producing particularly large losses.

D Component ablation and random-direction controls

  • Every nonzero vector is rescaled to a common reference norm independent of λc and λr, so randomized contrasts test component direction rather than total steering magnitude.Gaussian controls match the replaced component’s per-clip, per-layer norm; when both components are randomized, one Gaussian replaces the full vector at the same final norm.Source: D Component ablation and… · p. 18

D.1 Residual randomization

  • 45.87% to 0.58% for IndexTTS-2 and 51.94% to 4.12% for CosyVoice2 are the ρ drops when residual direction is randomized at λr = 3.H-Rate also falls from 82.49% to 67.57% and from 84.76% to 68.65%, respectively; the comparisons preserve residual allocation and final steering norm.Source: §D.1 · pp. 18–19

D.2 Shared component randomization

  • At λr = 3, shared-component randomization lowers E-SIM from 64.80% to 62.36% and TEP from 39.98% to 34.59% for IndexTTS-2.For CosyVoice2, E-SIM falls from 74.71% to 71.09% and TEP from 68.80% to 61.63%; ρ and H-Rate also decrease for both backbones.Source: §D.2 · p. 19

D.3 Complete randomization

  • Complete randomization yields ρ of −0.50% and H-Rate of 67.14% for IndexTTS-2, and −0.94% and 67.57% for CosyVoice2.For both backbones, E-SIM and TEP are at or below the no-steer baseline.Source: §D.3 · p. 19

E Steering with emotional speech prompts

EmoRES outperforms CoCoEmo with both neutral and target-emotion speaker prompts. Target-emotion prompts also improve the no-steer system’s emotion scores, providing a useful control signal.

E Steering with emotional speech prompts

  • A target-emotion prompt is a same-speaker recording on different content that raters heard as the requested emotion.Replacing a neutral prompt with this prompt substantially improves no-steer E-SIM and TEP for both backbones and datasets.Source: E Steering with emotional speech prompts · p. 20
  • The target-emotion reference provides a useful control signal even without steering, while EmoRES further improves E-SIM and TEP over CoCoEmo.The reported pattern holds for every backbone, dataset, and prompt type.Source: E Steering with emotional speech prompts · p. 20
  • With target-emotion prompts, EmoRES raises E-SIM and TEP over CoCoEmo on both backbones and datasets, including IEMOCAP and CREMA-D.For IndexTTS-2 on IEMOCAP, E-SIM rises from 79.36% to 84.54% and TEP from 78.02% to 89.17%; Table 5 covers the prompt and single-emotion evaluation conditions.Source: E Steering with emotional speech prompts · p. 20, Table 5 · p. 21

F Steering Without Native Emotion Conditioning

Without native emotion conditioning, steering provides the only explicit emotion-control signal, and EmoRES improves emotion metrics over CoCoEmo across both backbones and datasets.

F Steering Without Native Emotion Conditioning

  • Without native emotion embeddings or instructions, both CoCoEmo and EmoRES improve TEP and E-SIM over the no-steer systems.Table 6 evaluates mixed emotions with no native emotion conditioning and reports means and standard deviations over five replicates.Source: F Steering Without Native Emotion… · p. 22, Table 6 · p. 22
  • EmoRES improves E-SIM, TEP, ρ, and H-Rate over CoCoEmo for both backbones and datasets, including when steering is the only explicit emotion control.On IEMOCAP, IndexTTS-2 ρ rises from 23.14% to 44.76%, and CosyVoice2 ρ rises from 22.08% to 45.45%.Source: F Steering Without Native Emotion… · p. 22, Table 6 · p. 22
  • On IEMOCAP, H-Rate rises from 62.99% to 73.65% for IndexTTS-2 and from 61.98% to 73.39% for CosyVoice2 with EmoRES over CoCoEmo.The reported increases are for steering without native emotion conditioning.Source: F Steering Without Native Emotion… · p. 22

G Different speech emotion recognizers

Across three emotion recognizers, EmoRES retains its advantage over CoCoEmo on held-out IEMOCAP. The agreement holds under identical audio-generation conditions, indicating the result is not specific to one recognizer.

G Different speech emotion recognizers

  • The recognizer comparison holds the generated audio, steering coefficients, speaker prompts, and five prompt-resample replicates fixed.The audio was reused from Table 1, with only the recognizer changed.Source: G Different speech emotion recognizers · p. 23
  • Across both backbones, EmoRES beats CoCoEmo on TEP, ρ, and H-Rate in all 18 comparisons across the three recognizers.Table 7 evaluates held-out IEMOCAP under emotion2vec+ large, Qwen3-Omni, and Odyssey 2024 WavLM.Source: G Different speech emotion recognizers · p. 23
  • All three recognizers preserve the CoCoEmo-versus-EmoRES ordering despite different absolute score ranges, so the improvements are not specific to emotion2vec+ large.The generated audio and generation conditions are unchanged across recognizers.Source: G Different speech emotion recognizers · p. 23
Loading 2609.38157v1…