PaperlayerView exact revision on arXiv

Source-linked AI summary

EmoSteer-TTS: Fine-Grained and Training-Free Emotion-Controllable Text-to-Speech via Activation Steering

Tianxin Xie, Shan Yang, Chenxing Li, Dong Yu, Li Liu

arXiv:2508.03543v3cs.SDcs.AIeess.AS

TL;DR

Existing TTS emotion control is often coarse or unstable and can require extensive training data. EmoSteer-TTS steers internal activations of pretrained TTS models for fine-grained emotion control, and the authors report robust zero-shot control that outperforms state-of-the-art methods.

  • Problem

    Existing emotion-control methods are described as unstable or poorly generalizing, coarse in controllability, and difficult to interpret because TTS emotion representations have received limited exploration.

    Source: §1 · p. 2

  • Method

    EmoSteer-TTS identifies and modulates emotion-relevant internal activations in pretrained TTS models using steering vectors, without fine-tuning the models.

    Source: §1 · p. 2

  • Results

    Extensive experiments report robust zero-shot emotion control with broad applicability, outperforming state-of-the-art methods.

    Source: §1 · p. 3, §5 · p. 10

  • Takeaways & Limitations

    Analysis of emotion-steering dynamics offers insight into pretrained TTS models and practical guidance for designing the algorithm.

    Source: §1 · p. 2, §5 · p. 10

    Effective steering-vector extraction relies on high-quality emotional speech samples, and strong activation steering may introduce artifacts.

    Source: §5 · p. 10

Abstract

from arXiv · show

Text-to-speech (TTS) has shown great progress in recent years. However, most existing TTS systems offer only coarse and rigid emotion control, typically via discrete emotion labels or a carefully crafted and detailed emotional text prompt, making fine-grained emotion manipulation either inaccessible or unstable. These models also require extensive, high-quality datasets for training. To address these limitations, we propose EmoSteer-TTS, a novel training-free approach, to achieve fine-grained speech emotion control (conversion, interpolation, erasure) by activation steering. We first empirically observe that modifying a subset of the internal activations within a flow matching-based TTS model can effectively alter the emotional tone of synthesized speech. Building on this insight, we then develop a training-free and efficient algorithm, including activation extraction, emotional token searching, and inference-time steering, which can be seamlessly integrated into a wide range of pretrained models (e.g., F5-TTS, CosyVoice2, and E2-TTS). In addition, to derive effective steering vectors, we construct a curated emotional speech dataset with diverse speakers. Extensive experiments demonstrate that EmoSteer-TTS enables fine-grained, interpretable, and continuous control over speech emotion, outperforming the state-of-the-art (SOTA). To the best of our knowledge, this is the first method that achieves training-free and continuous fine-grained emotion control in TTS. Demo samples are available at https://emosteer-tts-demo.pages.dev/.

1 INTRODUCTION

Existing emotion-control methods are described as unstable or poorly generalizing, coarse, and difficult to interpret. EmoSteer-TTS uses internal representations of pretrained TTS models for training-free, fine-grained emotion control, with evaluations demonstrating effectiveness across pretrained models.

  • Existing approaches are limited by instability or poor generalization, coarse emotion control, and insufficient exploration of how TTS models represent emotion.Source: §1 · p. 2
  • EmoSteer-TTS identifies emotion-relevant activations and uses emotional–neutral activation differences to form steering vectors for six basic emotions, controlled by an adjustable strength parameter.The authors report that only a subset of tokens significantly influences synthesized emotional tone.Source: §1 · p. 2, Figure 2 · p. 3
  • The analysis of emotion-steering dynamics offers practical guidance for designing the proposed algorithm.Source: §1 · p. 2
  • Extensive objective and subjective evaluations demonstrate effective fine-grained emotion control and potential applicability across a wide range of pretrained TTS models.Source: §1 · p. 3

2 RELATED WORK

Prior emotion-controllable TTS methods use emotion labels, textual descriptions, or reference speech, while activation steering has been explored in language and image-generation models. EmoSteer-TTS brings activation steering to pretrained TTS models to control emotion.

  • Emotion-controllable TTS includes label-based and description-based methods, zero-shot approaches using reference speech, and systems that typically require large-scale datasets and training.Source: §2 · p. 3
  • Prior activation-steering work modulates internal activations in language and text-to-image models toward distributions associated with desired behaviors or styles.Source: §2 · p. 3
  • EmoSteer-TTS derives steering vectors and weights from neutral–emotional speech pairs, then modulates TTS activations during inference.Source: Figure 3 · p. 4

3 METHOD

EmoSteer-TTS extracts emotion directions from neutral and emotional reference speech, then searches for emotion-relevant tokens to build vectors applied to model activations at inference. The resulting controls support emotion conversion, interpolation, erasure, and composite emotion steering.

3.1 OVERVIEW

  • The pipeline computes neutral-versus-emotional activation differences, selects top-k emotion-relevant tokens to form steering vectors, and applies them to activations at inference.Steering uses unseen reference speech and text.Source: §3.1 · p. 4

3.2 ACTIVATION EXTRACTION

  • Layer-wise activation differences between neutral and emotional references provide steering directions, which are normalized and selected layers can be steered.The direction indicates the trajectory of emotional change, while its original magnitude reflects the extent of the transition; activation sequences are aligned to a fixed average length.Source: §3.2 · p. 4

3.3 STEERING VECTOR CONSTRUCTION

  • The method searches for emotion-relevant tokens using a speech emotion recognizer, then combines selected steering directions with token weights into a sparse weighted vector.The weighted vector lies in a feature subspace associated with emotional tone; searching across layers at shared token indices reduces computational complexity.Source: §3.3 · p. 5

3.4 FINE-GRAINED EMOTION CONTROL

  • Adjusting steering strength moves synthesized speech toward or away from a target emotion, while zero strength leaves its emotional tone unchanged.This enables emotion conversion and interpolation through inference-time activation modification.Source: §3.4 · p. 5
  • Projection-scaled subtraction removes the target emotion from reference activations while preserving other speech characteristics.The erasing strength is controlled by β.Source: §3.4 · pp. 5–6
  • Composite control can replace one emotion with another or combine multiple emotion vectors to synthesize compound and nuanced emotions.Examples include contempt, pleasant surprise, and happiness tinged with sadness.Source: §3.4 · p. 6

4 EXPERIMENT

Experiments evaluate EmoSteer-TTS across three flow-matching TTS models, conversion, interpolation, erasure, composite control, and out-of-distribution data. Results report fine-grained emotion control, with steering dynamics varying by token count, layer, and flow step.

4.1 DATASETS AND MODELS

  • Steering vectors use a curated 6,900-utterance dataset spanning six basic emotions and neutrality; evaluation covers three TTS models and in-distribution MSP-Podcast/ESD plus OOD SeedTTS/EMNS samples.The curated dataset includes diverse speakers with balanced gender distribution.Source: §4.1 · p. 6

4.2 EMOTION CONVERSION AND INTERPOLATION

  • EmoSteer-TTS achieves top E-SIM scores with F5-TTS, E2-TTS, and CosyVoice2, outperforming all baselines and matching FleSpeech; with F5-TTS, it records WER 2.79, close to CosyVoice2’s 2.53.EmoSteer-TTS+CosyVoice2 has N-MOS 3.65, close to the best, EmoVoice at 3.81.Source: §4.2 · p. 7

4.3 EMOTION ERASURE

  • Emotion erasure lowers target-emotion scores and earns fairly high EE-MOS; F0 contours illustrate the speech tone becoming calmer during erasure.The experiment uses β=2.5 and k=200.Source: §4.3 · p. 8, Figure 5 · p. 8

4.4 COMPOSITE CONTROL

  • Replacement suppresses the original emotion while enhancing the target, and two-emotion steering raises probabilities for both intended emotions.Fear-to-happiness replacement increases happiness by 0.28 and decreases fear by 0.33; fear-and-happiness steering yields probabilities of 0.22 and 0.33, respectively.Source: Figure 6 · p. 8, §4.4 · pp. 8–9

4.5 CROSS-DATASETS EVALUATION

  • On unseen SeedTTS and EMNS data, EmoSteer-TTS maintains robust performance with minimal degradation across metrics.Source: §4.5 · p. 9

4.6 ANALYSIS OF EMOTION STEERING DYNAMICS

  • In F5-TTS dynamics analyses, k=200 balances expressiveness and efficiency, spaced-layer steering boosts emotion probabilities, and steering across all flow steps yields the strongest emotion signal.Increasing k gives diminishing gains beyond 200 for most emotions; early-step steering has little effect, while middle and late steps exert more influence.Source: §4.6 · p. 9

5 CONCLUSION

EmoSteer-TTS provides training-free, fine-grained emotion control by steering internal TTS activations, supporting conversion, interpolation, and erasure without modifying or fine-tuning the pretrained model. Experiments report robust zero-shot control, broad applicability, and performance above SOTA methods, while the method relies on high-quality emotional speech samples and strong steering may introduce artifacts.

5 CONCLUSION

  • EmoSteer-TTS achieves robust, zero-shot emotion control with broad applicability, outperforming SOTA methods.The framework supports emotion conversion, interpolation, and erasure without modifying or fine-tuning the pretrained TTS model.Source: §5 · p. 10
  • Steering a subset of internal TTS activations enables flexible emotion conversion, interpolation, and erasure without fine-tuning the pretrained model.The authors also report that their analysis provides insights into emotion steering dynamics in flow matching-based TTS.Source: §5 · p. 10
  • The method relies on high-quality emotional speech samples, albeit in modest quantities, to extract effective steering vectors.Strong activation steering may introduce artifacts; future work will explore combining steering with learning-based approaches to mitigate these issues.Source: §5 · p. 10

6 REPRODUCIBILITY STATEMENT

The paper provides implementation and reproducibility details for its experiments, including steering configurations, code snippets, and dataset information. Appendix A describes hooks for conversion and interpolation, erasure, replacement, and multiple-emotion steering.

6 REPRODUCIBILITY STATEMENT

  • The paper documents its methodology, dataset curation, model configurations, experimental settings, evaluation criteria, and planned release of runnable code and data.Core algorithms are described in Section 4 and Appendix A provides code snippets for steering operations.Source: §6 · p. 11

A APPENDIX A: CODE SNIPPETS FOR FINE-GRAINED EMOTION CONTROL

A.1 EMOTION CONVERSION AND INTERPOLATION

  • Conversion and interpolation modify the first residual stream’s inputs with a weighted steering vector scaled by α.The steering vector is stored as steering_activations, and args.steering_strength controls its intensity.Source: §A.1 · p. 15

A.2 EMOTION ERASURE

  • Erasure subtracts a projection of the weighted steering vector from the first residual stream, with β controlling erasing intensity.The vector is stored in steering_activations and applied through a forward_pre_hook.Source: §A.2 · p. 16

A.3 EMOTION REPLACEMENT

  • Emotion replacement combines erasure of one emotion with steering toward another in the first residual stream.The corresponding intensities are β and α, controlled by args.erasing_strength and args.steering_strength.Source: §A.3 · p. 17

A.4 MULTIPLE EMOTION STEERING

  • Multiple-emotion steering adds two weighted steering vectors to the first residual stream, with separate intensities α1 and α2.The intensities are controlled by args.steering_strength_1 and steering_strength_2.Source: §A.4 · pp. 18–19

B APPENDIX B: DATASET CONSTRUCTION

The appendix describes a curated emotional speech dataset assembled from multiple corpora and filtered for audio quality and emotion-label confidence. It also identifies the pretrained TTS models and layer locations used for steering.

B APPENDIX B: DATASET CONSTRUCTION

  • The curated dataset combines emotional speech samples from multiple existing corpora, including MSP-Podcast, IEMOCAP, RAVDESS, CREMA-D, TESS, SAVEE, ASVP-ESD, CASIA, M3ED, ESD, and Emo-Emilia.The collection and filtering aim to retain samples with clearly distinguishable emotional tones.Source: B APPENDIX B: DATASET CONSTRUCTION · p. 20
  • Filtering removes utterances shorter than 2 seconds or longer than 20 seconds, samples with over 30% silence or SNR below 10 dB, and samples with emotion-recognition confidence below 0.6.The pipeline also manually inspects 50% of the data.Source: B APPENDIX B: DATASET CONSTRUCTION · p. 20
  • The resulting dataset covers a broad range of speakers, emotions, and speaking styles for fine-grained emotion steering and evaluation.The authors state that data and code will be released upon acceptance.Source: B APPENDIX B: DATASET CONSTRUCTION · p. 20

C.1 MODEL CONFIGURATIONS

  • Main experiments steer the first residual stream at each layer of F5-TTS, E2-TTS, and CosyVoice2.Table 2 is identified as detailing model configurations for activation steering.Source: §C.1 · p. 20, Table 2 · p. 20

C.2 HARDWARE AND SOFTWARE CONFIGURATIONS

Experiments ran on a server equipped with eight NVIDIA RTX 6000 Ada GPUs, two Intel Xeon Platinum 8375C CPUs, and 256 GB of RAM.

C.2 HARDWARE AND SOFTWARE CONFIGURATIONS

  • Experiments used 8 NVIDIA RTX 6000 Ada GPUs, 2 Intel Xeon Platinum 8375C CPUs, and 256 GB of RAM.The CPUs ran at 2.9 GHz with 32 cores each.Source: §C.2 · p. 20
  • The server ran Ubuntu 20.04.6 LTS, and model-specific software libraries and frameworks are described in the respective GitHub repositories.Code was executed in Conda environments.Source: §C.2 · pp. 20–21

D APPENDIX D: OBJECTIVE EVALUATION METRICS

Appendix D defines three listener-rated metrics for naturalness, emotional interpolation, and emotion erasure, each averaged across multiple annotators.

D.1 NATURALNESS MEAN OPINION SCORE

  • N-MOS rates perceived naturalness on a 5-point scale, regardless of emotional expressiveness or content accuracy.Each utterance is scored by multiple annotators, and the reported score is their average.Source: §D.1 · p. 21

D.2 EMOTION INTERPOLATION MEAN OPINION SCORE

  • EI-MOS assesses how smoothly and consistently emotion changes across a sequence with gradually increasing intensity.Raters focus on continuity of emotional expression rather than the naturalness or correctness of individual utterances; scores are averaged across annotators.Source: §D.2 · pp. 21–22

D.3 EMOTION ERASURE MEAN OPINION SCORE

  • EE-MOS rates how effectively a target emotion is removed, with emotionally neutral and natural speech as the desired outcome.Multiple listeners rate each utterance; higher average scores indicate more effective erasure.Source: §D.3 · p. 22

E APPENDIX E: REPRODUCED BASELINE RESULTS

Appendix E reports reproduced open-source baseline results under a controlled protocol and provides additional emotion-interpolation visualizations.

E APPENDIX E: REPRODUCED BASELINE RESULTS

  • Reproduced baselines use the same text prompts, reference speeches, and evaluation scripts as the in-distribution evaluation on MSP-Podcast and ESD.Only open-source methods are included, and the reproduced results are provided for reference because their quality cannot be guaranteed.Source: E APPENDIX E: REPRODUCED BASELINE… · p. 22
  • The reproduced results are not presented as guaranteed-quality comparisons and are intended for reference only.The section limits reporting to open-source methods because reproduced quality cannot be guaranteed.Source: E APPENDIX E: REPRODUCED BASELINE… · p. 22
  • Figure 9 visualizes interpolation from neutrality (α=0) to a target emotion (α=2) across F5-TTS, E2-TTS, and CosyVoice2.The figure includes anger, disgust, fear, happiness, sadness, and surprise.Source: Figure 9 · p. 23

F.1 EMOTION INTERPOLATION

Additional F0 visualizations illustrate emotion interpolation, with pitch trends varying by target emotion.

F.1 EMOTION INTERPOLATION

  • The subsection presents additional visualizations of F0 contours to illustrate the proposed EmoSteer-TTS system’s fine-grained and continuous emotion-interpolation capabilities.Source: §F.1 · p. 23
  • Angrier, happier, and more surprised interpolations tend to have higher pitch, while sadder tones tend to have lower pitch.The passage presents these patterns as tendencies in the F0 contours.Source: §F.1 · p. 24
  • Disgust and fear interpolation show no clear monotonic pitch trend.The authors attribute this to these emotions being more closely tied to semantic content than acoustic characteristics.Source: §F.1 · p. 24

F.2 EMOTION ERASURE

Additional F0 visualizations show that pitch changes after emotion erasure differ by emotion, while pitch alone captures only part of emotional perception.

F.2 EMOTION ERASURE

  • After erasure, angry, disgusted, happy, and surprised voices have noticeably flatter pitch contours, while fear and sadness show more varied changes.The authors relate fear variability to multiple vocal styles and sadness variability to forms ranging from soft weeping to loud crying.Source: §F.2 · p. 25
  • F0 contours capture only part of emotional perception, so the authors encourage listening to the audio samples for complex emotional nuances.Pitch alone cannot fully convey those nuances.Source: §F.2 · p. 25

G APPENDIX G: THE USE OF LLMS

Loading 2508.03543v3…