PaperlayerView exact revision on arXiv

Source-linked AI summary

Sparse Autoencoders for Interpretable Emotion Control in Text-to-Speech

Hongfei Du, Jiacheng Shi, Sidi Lu, Gang Zhou, Ye Gao

arXiv:2606.01479v1cs.CL

TL;DR

Interpretable emotional control in LLM-based TTS is limited by external conditioning and global steering. This paper uses sparse autoencoders to identify emotion-related semantic features and intervenes on selected features for bidirectional control, achieving comparable or superior performance while preserving key speech properties.

  • Problem

    Existing TTS emotion controls rely mainly on external conditioning or global activation shifts, limiting insight into the internal representations underlying emotional expression.

    Source: §1 · pp. 1–2

  • Method

    The paper trains sparse autoencoders on semantic hidden states to identify emotion-related latent features and modulates selected features without changing backbone parameters.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    Emotional variation is distributed across multiple sparse features, and steering selected features supports both emotion induction and suppression with comparable or superior performance to global steering and existing TTS baselines.

    Source: §1 · p. 2, §4.3 · p. 9, §5 · p. 9

  • Takeaways & Limitations

    Emotion expression in LLM-based TTS can be controlled through coordinated, interpretable latent contributions while largely preserving linguistic content, naturalness, and speaker identity.

    Source: §5 · p. 9

    Full analysis uses a single backbone configuration, and systematic characterization of the full audio latent space remains open.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Integrating large language models (LLMs) into text-to-speech (TTS) systems has improved speech expressiveness, yet interpretable emotional control remains challenging. Existing approaches primarily rely on external conditioning or global activation steering, offering limited insight into the internal representations underlying emotional control. In this work, we analyze emotion-related variation in the semantic hidden states of LLM-based TTS models using sparse autoencoders (SAEs) to identify sparse latent features. Our analysis shows that emotional variation is distributed across multiple sparse latent features, while intervening on a small subset enables interpretable emotion control. Building on this observation, we introduce a feature-level intervention framework for bidirectional emotion induction and suppression without modifying backbone parameters. We further show that distinct latent features are associated with specific acoustic attributes (e.g., pitch), suggesting that emotional expression arises from coordinated latent contributions rather than a single global shift. Empirically, steering these sparse latent features achieves comparable or superior emotion induction and suppression performance relative to global steering and existing TTS baselines.

1. Introduction

LLM-based TTS makes expressive speech possible, but existing emotion controls provide limited interpretability and flexibility. This work decomposes emotion-related variation into sparse semantic features and uses them for bidirectional, parameter-free control.

Motivation

  • Existing label-, reference-, and global-steering methods offer limited interpretability or flexibility because they rely on fixed conditions or dense representation shifts.Label-based methods can average nuanced variation, reference-based methods require suitable samples, and dense steering captures only a global shift.Source: §1 · pp. 1–2

Approach

  • The study trains a sparse autoencoder on semantic hidden activations collected during speech semantic-token generation to identify emotion-related latent features.The SAE is applied before acoustic synthesis, focusing analysis on internal semantic representations.Source: §1 · p. 2

Findings

  • Individual latent features correspond to distinct acoustic variations, including pitch and spectral brightness.The analysis links semantic latent features to observable acoustic attributes.Source: §1 · p. 2
  • Emotion-related expression is encoded across multiple sparse latent features rather than a single global direction.This provides a representation-level account of emotion modeling in speech synthesis.Source: §1 · p. 2

Contribution

  • The proposed feature-level framework enables emotion induction and suppression without modifying backbone parameters.Increasing selected features induces target emotion, while decreasing them suppresses it toward neutral speech with text and speaker identity fixed.Source: Figure 1 · p. 1, §1 · p. 2

2. Related Work

Prior work controls TTS emotion through external conditioning, reference transfer, or dense activation steering, while SAE research in audio has focused mainly on acoustic latent spaces. This work instead identifies and modulates sparse emotion-related features in the semantic backbone before acoustic synthesis.

Sparse Autoencoders

  • Prior SAE work on audio generative models mainly interprets acoustic latent features through properties such as pitch, amplitude, and timbre.Those studies focus on acoustic realization rather than semantic representations before synthesis.Source: §2 · p. 2

Novelty

  • This work applies SAEs to the semantic backbone of autoregressive LLM-based TTS to identify and modulate emotion-related features before acoustic generation.The intervention path decodes modified semantic representations and passes them to the CFM module and vocoder.Source: §2 · pp. 2–3, Figure 2 · p. 3

Emotion-Controllable Text-to-Speech

  • Prior emotion-controllable TTS methods use discrete labels, textual prompts, dimensional affect coordinates, or reference speech, but their controls are externally defined or opaque.Reference-based methods also require selecting suitable speech at inference time.Source: §2 · p. 2

Activation Steering and Linear Representations

  • Activation steering can control TTS emotion by modifying hidden states, but typically uses a dense mean-difference direction that creates a global shift.The paper investigates sparse latent modulation as a more feature-level alternative.Source: §2 · p. 2

3. Methodology

The method decomposes dense semantic-backbone activations into sparse latent features, identifies emotion-related features under controlled comparisons, and selectively steers them before acoustic synthesis. This enables bidirectional emotion control through sparse, interpretable directions rather than a single dense global shift.

3.1. Sparse Autoencoders for LLM-based TTS Models

  • The SAE encodes dense residual activations into sparse latent features, decodes them back into residual representations, and passes the modified states to the acoustic generator.The overcomplete representation retains only the Top-k activations, while decoder columns represent directions in residual space.Source: §3.1 · pp. 3–4
  • SAE training combines reconstruction loss with an auxiliary residual-reconstruction loss to improve latent utilization and reduce dead latents.Selected inactive features model residual error not captured by the Top-k active features.Source: §3.1 · p. 4

3.2. Sparse Latent Feature Identification and Intervention for Emotion Control

  • The inference pipeline encodes intercepted residual activations, modifies selected emotion-related latent features, decodes them, and sends the result to the CFM module and vocoder.Backbone parameters remain unchanged because intervention occurs in the semantic representation before acoustic synthesis.Source: §3.1 · p. 3, §3.2 · p. 4

3.2.1. IDENTIFYING EMOTION-RELATED LATENT FEATURES

  • Emotion-related features are identified by comparing sentence-level activation frequency between matched emotional and neutral utterances with fixed text and speaker identity.The controlled pairing reduces lexical and speaker confounds, while sentence-level occurrence emphasizes stable utterance-level emotional participation over isolated token spikes.Source: §3.2.1 · p. 4

3.2.2. SPARSE LATENT-FEATURE STEERING

  • The formulation represents emotion modulation as steering along a sparse combination of interpretable decoder directions rather than a single dense global direction.Applying the same feature set and scaling factor across token positions yields time-invariant semantic-token-level control, while position-dependent α_e permits time-varying variants.Source: §3.2.2 · p. 5
  • Steering changes selected latent activations while leaving other features unchanged, with positive α_e inducing emotion and negative α_e suppressing it.Varying α_e provides continuous emotion-strength control while keeping the selected feature set fixed.Source: §3.2.2 · p. 5

4. Experiments

Experiments show that emotion-related variation is sparse and distributed across latent features with distinct acoustic effects, enabling bidirectional steering that preserves key speech properties. Across controlled analyses and evaluations, selectivity-ranked SAE steering performs competitively with global steering and existing TTS systems.

4.1. Experimental Setup

  • The SAE is trained on 56,000 emotion-controlled generations using 4,096 latent features with 32 active features per token, and evaluated with acoustic, emotion, content, speaker, and human-listening metrics.The experimental design matches text and speaker conditions to isolate emotion-related variation.Source: §4.1 · pp. 5–6

4.2.1. SPARSE ORGANIZATION OF EMOTION SIGNALS

  • Most latent features have similar anger and neutral activation frequencies, while a sparse tail shows strong emotion selectivity; the top six anger features each reach a selectivity score of 1.Similar heavy-tailed organization appears for happiness and sadness, indicating consistency across emotions.Source: §4.2.1 · pp. 6–7, Figure 3 · p. 6

4.2.2. ACOUSTIC VALIDATION OF SELECTED LATENT FEATURES

  • Steering one selected latent feature produces localized mid- to high-frequency amplification and local energy increases while largely preserving the overall time-frequency structure.Paired analysis under matched text and speaker conditions finds +23.11 Hz mean F0 and Δ = +0.00435 RMS energy, with no significant duration change (p = 0.687).Source: §4.2.2 · p. 7, Figure 4 · p. 7
  • Spectral-centroid deviations vary systematically with steering scale: negative scales lower spectral centroid and positive scales increase it, indicating controllable spectral-brightness modulation.The analysis uses scales from −60 to +60 across 20 neutral utterances with 95% confidence intervals.Source: §4.2.2 · p. 7, Figure 5 · p. 7
  • A rank-1 happiness feature moves generated speech toward a real happiness reference in emotion-embedding space as steering increases.The comparison uses an IEMOCAP-based happiness prototype and real neutral and happiness reference levels.Source: §4.2.2 · p. 7, Figure 6 · p. 8

4.3. Emotion Steering in TTS

  • The composite steering direction combines six selectivity-ranked latent features with equal weights, outperforming arbitrary Random SAE selection in emotion consistency and improving interpretability over dense global steering.Global steering uses a target-neutral mean-difference vector, whereas SAE-Emotion maps selected SAE decoder directions back to residual space.Source: §4.3 · pp. 8–9
  • SAE-Emotion achieves the highest or second-highest Emo-SIM for induction across all evaluated emotions while maintaining competitive WER, and the highest or comparable Emo-SIM for suppression.Suppression also yields generally comparable WER and speaker similarity; Table 1 covers both directions across anger, happiness, and sadness.Source: §4.3 · pp. 8–9, Table 1 · p. 8
  • Human evaluation with 20 raters gives SAE-Emotion the highest EMOS and NMOS among compared steering methods, supporting perceived emotion accuracy and naturalness.Table 2 compares SAE-Emotion with Global Steering and Random SAE on a 0–5 scale.Source: §4.3 · p. 9, Table 2 · p. 9

5. Conclusion

The paper finds that emotional variation in LLM-based TTS is distributed across multiple sparse latent features with distinct activation and acoustic patterns. Selectivity-ranked interventions enable bidirectional emotion control without changing backbone parameters while largely preserving linguistic content, naturalness, and speaker identity.

5. Conclusion

  • Emotional variation is distributed across multiple sparse latent features with distinct activation and acoustic patterns in the semantic backbone.The study identifies a small subset of emotion-related features under controlled conditions.Source: §5 · p. 9
  • Selectivity-ranked latent-feature interventions support both emotion induction and suppression without modifying backbone parameters.The approach provides an interpretable feature-level alternative to dense global steering.Source: §5 · p. 9
  • Manipulating selected sparse features can steer emotional expression while largely preserving linguistic content, naturalness, and speaker identity.Source: §5 · p. 9

Limitations

The study’s full analysis uses one backbone configuration because SAE training is computationally and storage intensive, while cross-backbone evidence is limited to the appendix. It also analyzes representative observable features rather than systematically characterizing the full latent space.

  • Full analysis is restricted to a single backbone configuration, with additional cross-backbone analysis provided only in the appendix.The authors identify broader controlled evaluation across architectures as future work.Source: Limitations · p. 9
  • The study focuses on representative, clearly observable emotional latent features because speech expression is multidimensional and difficult to quantify.A systematic characterization of the full latent space remains open.Source: Limitations · p. 9

Impact Statement

The work targets responsible emotional steering for accessibility, affective computing, and personalized speech synthesis. Its controlled analyses use neutral prompts and fixed speaker and timbre references so differences can be attributed primarily to emotional style.

Impact Statement

  • The framework may benefit accessibility, affective computing, and personalized speech synthesis, while raising misuse risks including emotional manipulation, impersonation, and deceptive synthetic content.The authors call for safeguards concerning consent, misuse prevention, and detection of emotionally expressive synthetic speech.Source: Impact Statement · p. 9

Appendix Organization

  • The appendix documents controlled prompts, SAE analyses, emotion-control studies, baselines, evaluation protocols, and an additional cross-backbone analysis.Source: Appendix Organization · p. 13

A. Text Prompts Used for Controlled Emotion Analysis

B. SAE Training Details

The SAE is trained with a fixed sparse architecture and reconstruction-focused objectives, alongside regularization and stabilization mechanisms. A 4096-dimensional model is selected because a larger model fragments emotion-related features and reduces interpretability.

B. SAE Training Details

  • The 4096-dimensional SAE uses K = 32 active features per token and is trained for 30,000 Adam optimization steps on 16,384 tokens per update.The learning rate is 10^-4 and the latent dimensionality is fixed at 4096.Source: B. SAE Training Details · p. 14
  • Usage regularization and an auxiliary residual projection loss reduce feature collapse and dead latent features during normalized reconstruction training.The regularizer weight is 0.01 and λaux = 0.1.Source: B. SAE Training Details · p. 14
  • Decoder renormalization, orthogonal gradient projection, and EMA stabilize intervention scales and optimization while preserving decoder-column norms.The EMA decay is 0.99.Source: B. SAE Training Details · p. 14
  • The 10,240-dimensional SAE lowers reconstruction error but fragments emotion-related features across more sparse directions, so analysis focuses on the more interpretable 4096-dimensional model.Source: B. SAE Training Details · p. 14

C. SAE Evaluation

The SAE preserves semantic variance under strict sparsity, while emotion selectivity is concentrated in a small subset of features. Emotion control is continuous and emotion-dependent: some emotions use one dominant feature, whereas others require coordinated multi-feature steering.

C. SAE Evaluation

  • Mean squared reconstruction error is 0.129, indicating that sparse activations preserve most semantic variance under the Top-k constraint.Source: C. SAE Evaluation · p. 14

D. Additional Emotion Selectivity and Control Analyses

D.1. Single-Scalar Intensity Control

  • Similarity rises from approximately 0.77 at αe = −60 to 0.86 at αe = +60, providing a continuous single-coefficient control of target-emotion intensity.Source: §D.1 · p. 16, Figure 9 · p. 16

D.2. Cross-Emotion Behavior and Latent-Feature Budget

  • Happiness shows monotonic control with one feature, whereas anger and sadness benefit from larger budgets, indicating emotion-dependent distribution across sparse features.Source: §D.2 · p. 16, Figure 10 · p. 17

D.3. Latent Feature Overlap Across Emotions

  • No overlap appears among emotions’ top-6 features in this setting, although partial overlap may emerge under other sparsity thresholds or selection strategies.Source: §D.3 · p. 16

D.4. Comparison with Alternative Selection Criteria

  • Sentence-level selectivity achieves stronger emotion alignment than magnitude-based and token-level alternatives across all three target emotions.Source: Table 3 · p. 18, §D.4 · p. 18

E. Quantitative Acoustic Analysis

Controlled steering changes specific acoustic properties: pitch and vocal intensity increase, while speaking duration remains statistically unchanged.

  • Steering significantly increases mean F0 and RMS energy, but produces no statistically significant duration change.The matched-text, matched-speaker comparison indicates modulation of pitch and vocal intensity rather than speaking duration.Source: E. Quantitative Acoustic Analysis · p. 18, Table 4 · p. 18

F. Robustness under Strong Steering

Under stronger steering, sparse interventions preserve linguistic fidelity better than dense global steering, while bidirectional control produces larger pitch increases than positive-only steering. The evaluation uses IndexTTS2 and compares standard emotional TTS baselines under matched control conditions.

F. Robustness under Strong Steering

  • SAE steering substantially reduces WER and deletion errors versus dense global steering under high steering strength.This suggests sparse feature-level interventions introduce less decoding interference and preserve linguistic fidelity better.Source: F. Robustness under Strong Steering · p. 18, Table 5 · p. 18

F.1. Bidirectional Latent-Feature Intervention

  • Bidirectional latent-feature intervention produces a larger mean F0 increase than positive-only steering under paired utterances.The supplementary intervention increases target-emotion features while decreasing opposing or neutral-associated features.Source: §F.1 · p. 18, Figure 11 · p. 19, Figure 12 · p. 19
  • Steering increases mean F0 by +23.11 Hz on average under matched text and speaker identity.The paired comparison reports p = 1.07 × 10−4.Source: Figure 11 · p. 19

G. Baselines and Backbone

  • The comparison includes VALL-E-X, Spark-TTS, EmoVoice, and CosyVoice using their released implementations, checkpoints, and default inference settings.These systems use their standard emotional control interfaces rather than explicit representation-level steering.Source: G. Baselines and Backbone · p. 20, H. Baseline Evaluation Details · p. 21
  • IndexTTS2 provides the autoregressive semantic-token backbone used for SAE analysis and steering, with separate control over timbre and emotional style.Its soft instruction mechanism supports natural-language emotional guidance, while the analysis targets implicitly encoded emotional representations.Source: G. Baselines and Backbone · p. 20

H. Baseline Evaluation Details

  • Because baseline systems lack dedicated emotion suppression mechanisms, evaluation compares target-emotion generations against neutral generations using identical text and speaker conditions.Speaker similarity is measured between neutral and emotion-controlled generations to assess preservation of speaker identity.Source: H. Baseline Evaluation Details · p. 21

I. Additional Cross-Backbone Analysis

The additional LLaSA analysis tests whether emotion-alignment behavior extends beyond the main backbone. Increasing steering scale monotonically shifts generations toward the anger prototype.

  • Increasing steering scale consistently shifts LLaSA generations from the neutral region toward the anger prototype.Cosine similarity surpasses the neutral reference around scale 2 and approaches the real anger reference at larger scales.Source: §I · p. 21, Figure 13 · p. 21
Loading 2606.01479v1…