Loud and Clear: Dynamic Activation Steering for Improving Speech Intelligibility in Noisy Environments

1. INTRODUCTION

• Lombard adaptations—including greater vocal effort and clearer articulation—improve intelligibility, motivating synthesized speech that could help in noisy settings, L2 learning, and assistive communication.Articulatory changes are also relevant to robust speech recognition.Source: §1 · p. 1
• Earlier neural TTS work mainly controls global properties such as effort, pitch, and spectral characteristics; hyper-articulation and joint control of both attributes remain less explored.Prior methods include transfer learning, fine-tuning, spectral-tilt augmentation, pitch/energy modeling [15], loudness control, and latent interpolation.Source: §1 · p. 1
• Existing approaches do not provide joint, continuous within-utterance control of vocal effort and hyper-articulation, while training, fine-tuning, or fixed transformations limit inference-time control across speakers and languages.The required enhancement may vary with acoustic conditions or listener needs.Source: §1 · p. 1
• Prompt-Relative Activation Steering compares each generated token’s activation with its reference-prompt baseline and applies only the residual needed to reach the target, avoiding accumulated steering effects.The method extracts vocal-effort and hyper-articulation directions from paired speech data using Qwen3-TTS [28].Source: §1 · p. 1
• The contributions are training-free joint and continuous control across speakers and languages, a mechanism that prevents accumulation, and dynamic adjustment of steering strength during synthesis.The method modifies hidden representations without changing model parameters [22].Source: §1 · pp. 1–2

2. METHOD

• Activation steering modifies hidden representations along attribute-specific directions during inference, without changing model parameters; the method combines vocal-effort and hyper-articulation directions for generated tokens.The directions are extracted independently and applied to residual outputs of Qwen3-TTS’s autoregressive backbone.Source: §2 · p. 2

2.1. Steering Vector Extraction

• Paired samples use the same speaker and linguistic content but different speaking styles; average target-versus-reference differences at generated-token positions define each attribute’s steering direction.Vocal effort uses loud versus regular speech from EARS [29], while hyper-articulation uses enunciated versus default speech from Expresso.Source: §2.1 · p. 2

2.2. Joint Control of Vocal Effort and Articulation

• User-defined coefficients weight and combine the vocal-effort and articulation directions, enabling continuous control of their relative contributions or steering of either attribute alone.Setting one coefficient to zero gives single-attribute steering; nonzero values for both give joint control.Source: §2.2 · p. 2

2.3. Prompt-Relative Activation Steering

• Fixed steering can accumulate through causal attention and distort longer outputs; prompt-relative steering instead applies the signed residual between each token’s current projection and its prompt-specific target.Because the residual can be negative, reducing steering strength during generation can move representations back toward the prompt baseline; modified representations are normalized and steering is applied at layers 19–20.Source: §2.3 · p. 2

3.1. Lombard-Related Acoustic Changes

• As steering strength increases, spectral tilt becomes less negative, speaking rate decreases, and vowel-space dispersion increases, matching distinct and combined vocal-effort and articulation effects.Vocal-effort steering primarily affects spectral tilt, while articulation steering more strongly reduces speaking rate; both directions also increase VSA.Source: §3.1 · p. 3, Table 1 · p. 3
• Across unseen speakers and German, Spanish, and Japanese, spectral tilt and speaking rate shift consistently, while VSA is more speaker- and language-dependent and speaker similarity remains largely preserved.The results suggest precise articulatory control may benefit from language-specific calibration.Source: §3.1 · p. 3, Table 2 · p. 3

3.2. Speech Intelligibility In Noise

• Under restaurant babble, WER consistently decreases with steering, with a larger effect at higher noise levels, especially for German, Spanish, and Japanese.Tests used 1, 5, and 10 dB SNR, with noise scaled to maintain a fixed SNR across conditions.Source: §3.2 · p. 3, Figure 1 · p. 4

3.3. Human Evaluation

• +0.958 ± 0.48 CMOS with a 95% confidence interval indicates a preference for steered speech among seven participants evaluating noisy utterance pairs.The study used 15 baseline-versus-steered pairs at 5 dB SNR and randomized, blinded presentation.Source: §3.3 · p. 4

3.4. Dynamic Steering Evaluation

• During streaming, dynamic steering follows the unsteered baseline before intervention, shifts during the active window, and reverts afterward, while streaming hooks add negligible computational overhead.The speech example also shows increased amplitude and higher-frequency energy during steering, followed by a return to baseline.Source: §3.4 · p. 4, Table 4 · p. 4, Table 3 · p. 4

• The method jointly controls vocal effort and hyper-articulation without fine-tuning, with experiments across seen and unseen speakers and multiple languages.Source: §4 · p. 4
• Prompt-Relative Activation Steering regulates activations relative to the reference voice to prevent cumulative effects and enable stable, dynamic within-utterance control.Source: §4 · p. 4
• Experiments show systematic Lombard-related acoustic changes, preserved speaker similarity, and improved ASR-based intelligibility under noise.Future work will address more diverse acoustic environments and listener types and improve hyper-articulation control across languages.Source: §4 · p. 4