Source-linked AI summary
Steering Speech-Language Models: Training-Free Task Specialization via Contrastive Activation Addition
Séverin Baroudi, Yanis Labrak, Pierfrancesco Melucci, Sergio Burdisso, Petr Motlicek, Hervé Bredin, Mirco Ravanelli, Ricard Marxer
TL;DR
SpeechLLM steering has seen limited training-free use, leaving open whether it can specialize models across speech tasks. This paper derives task directions from a small set of examples and injects them at inference, alone or with prompts. The authors report gains over prompting alone on nearly all evaluated ASR, ER, and AC benchmarks, along with transfer to out-of-domain data.
Problem
Training-free steering in SpeechLLMs had been limited to narrow settings, leaving open whether task directions can specialize one model across speech tasks and generalize across datasets.
Source: §1 · p. 1Method
The paper uses training-free CAA to extract task-specific directions from a small set of examples and injects them into SpeechLLM representations at inference.
Source: §1 · p. 1, §2.1 · p. 2Results
The authors report improvements over prompting alone on nearly all ASR, ER, and AC benchmarks across two SpeechLLMs, with reduced prompt sensitivity and transfer to out-of-domain datasets.
Source: §5 · p. 4Takeaways & Limitations
The paper demonstrates that a script-normalization direction can be combined with transcription steering to enforce a target non-Latin script.
Source: §1 · p. 2The extraction assumes that pairing positive and negative prompts with the same audio pool makes acoustic, speaker, and lexical contributions cancel on average.
Source: §2.1 · p. 2
Abstract
from arXiv · showhide
Activation steering has proven effective for controlling the behavior of Large Language Models (LLMs) at inference time, but its application to SpeechLLMs remains new, and training-free steering approaches for such models are still largely unexplored. We propose a training-free Contrastive Activation Addition (CAA) protocol that derives steering vectors for common speech tasks (e.g. transcription) in SpeechLLMs from a small number of labeled utterances. We showcase that adding these vectors in the representation space, at inference time, enforces better the targeted speech task. We further show that, when combined with prompting, these vectors yield to consistent improvement over prompting alone on most evaluated tasks such as Automatic Speech Recognition (ASR) or Emotion Recognition (ER), and transfer to out-of-domain data. We additionally demonstrate the usefulness of script-normalization directions to enforce the target script of a specific language.
1. INTRODUCTION
Activation steering changes a model’s internal representations at inference time, but training-free steering for SpeechLLMs has so far covered only narrow settings. This paper proposes using task-specific directions to specialize speech models, combine steering with prompts, transfer across datasets, and control transcription script.
1. INTRODUCTION
- Activation steering alters LLM hidden states during generation without retraining, using vectors associated with interpretable attributes.Prior work has extended steering vectors beyond language to image and audio models.Source: §1 · p. 1
- Training-free CAA in SpeechLLMs had been limited to accented transcription and chain-of-thought reasoning, leaving specialization across a range of speech tasks open.The paper asks whether speech tasks are linearly encoded and whether their directions can complement prompting and generalize across datasets.Source: §1 · p. 1
- The proposed training-free CAA procedure extracts a task-specific steering vector from a small set of speech or audio examples.The paper injects the direction at inference time to specialize a SpeechLLM without textual instruction.Source: §1 · pp. 1–2
- Combining steering with prompting outperforms prompting alone while reducing prompt-wording sensitivity; the vectors also transfer to out-of-domain data.The paper further reports using the same injection technique to enforce a non-Latin target script for transcription.Source: §1 · p. 2
2. ACTIVATION ADDITION
The protocol contrasts activations for target-task and other-task instructions paired with the same audio to derive a task direction. At inference, it adds that direction at a selected layer and token positions, and can combine it with a script-normalization direction.
2.1. Extracting a task direction
- The task direction is the difference between mean activations for positive and negative prompts, paired with the same pool of audio clips.The protocol assumes that shared audio makes acoustic, speaker, and lexical contributions cancel on average, leaving the instruction-induced component.Source: §2.1 · p. 2
2.2. Task induction via activation addition
- At inference, CAA shifts residual activations by a task-specific strength at a selected layer and set of token positions.The candidate layer and strength are selected by grid search on a small audio pool from the target task’s training set; steering can include audio tokens or leave them untouched.Source: §2.2 · p. 2
- A script-normalization direction derived from translation prompts can be injected with the transcription direction to enforce a target alphabet.After setting task strength, a second grid search selects the script direction’s site and amplitude.Source: §2.2 · p. 2
3. EXPERIMENTS
The experiments evaluate task steering on two instruction-following SpeechLLMs across ASR, emotion recognition, and audio captioning, using small task-specific training sets. Prompting results are averaged over three prompts, and LoRA-SFT provides a fine-tuning comparison.
3. EXPERIMENTS
- Evaluation covers ASR, emotion recognition, and audio captioning on two SpeechLLMs, with task directions extracted from task-specific training data.ASR uses 300 utterances across LibriSpeech and five CommonVoice languages; emotion recognition uses IEMOCAP, and captioning uses AudioCaps.Source: §3 · p. 2
- Table 1 reports task-steering results for ASR, emotion recognition, and audio captioning, with prompted rows averaged over three prompts and LoRA-SFT shown for comparison.The best score for each system is highlighted.Source: Table 1 · p. 3
- Prompting results in Tables 1, 3, and 4 are averaged over three prompts to reduce variability from prompt choice.For non-Latin transcription, prompts specify the target script.Source: §3 · p. 3
- LoRA-SFT uses the same task datasets as direction extraction, but audio-only fine-tuning with 50–300 examples shows strong overfitting.Early stopping uses development-set loss; the authors suggest overfitting likely explains the lack of improvement on some datasets, including LibriSpeech.Source: §3 · p. 3
4. RESULTS
Across the evaluated tasks, steering combined with prompts generally improves on prompting alone, while steering without prompts shows task- and dataset-dependent results. The results also examine injection choices, script normalization, and cross-domain transfer.
4.1. Activation sites and direction injection
- Effective injection sites cluster in layers 10–16 for ASR and emotion recognition, with deeper layers requiring lower amplitudes to avoid oversaturation.The authors attribute this pattern to injections persisting through later blocks and being renormalized by fewer blocks when inserted deeper.Source: §4.1 · p. 3, Figure 2 · p. 3
- Steering all positions, including audio tokens, improves transcription over text-position-only steering by 43.9% and 19.1% relative on LibriSpeech and CommonVoice, respectively.The authors report that acoustic content remains mostly intact despite shifting audio activations.Source: §4.1 · p. 3, Table 2 · p. 3
4.2. Contrastive Activation Addition results
- High WER and CER transcripts often reflect continued generation after the audio ends or correct semantic content rendered in Latin rather than the target script.The authors note that target-script errors also occur with prompting alone despite explicit script instructions.Source: §4.2 · p. 4
- Adding task vectors to prompts beats prompting alone on every Qwen2-Audio benchmark and all but CommonVoice Russian and AudioCaps CIDEr for Phi-4-Multimodal.On Qwen2-Audio, CommonVoice CER falls by 5.1 points on Hindi and 6.4 on Arabic; Phi-4 gains include 21.2% relative improvement on LibriSpeech test-clean and 82.4% relative CER reduction on CommonVoice French.Source: §4.2 · p. 4
- Without prompts, steering outperforms LoRA-SFT on LibriSpeech for both SpeechLLMs; on the more challenging CommonVoice dataset, SFT remains stronger.On Qwen2-Audio, steering alone improves captioning over prompting: CIDEr rises from 0.26 to 0.43 and ROUGE-1 from 34.1 to 40.1. CommonVoice CER scores are mixed for both models, particularly on non-Latin-script languages.Source: §4.2 · p. 4
- Script normalization lowers Phi-4 Russian CER by 29.2 points and Hindi by 1.2, while Arabic stagnates; Qwen2-Audio CER falls by 5.4 points on Arabic and 2.5 on Hindi.The reported Table 3 comparison covers CommonVoice Arabic, Hindi, and Russian.Source: Table 3 · p. 4, §4.2 · p. 4
4.3. Cross-domain transfer and steering robustness
- A transcription direction transferred between LibriSpeech and CommonVoice with negligible performance drop, and task steering reduced prompt-to-prompt standard deviation by roughly half.With prompts, steering improved LS WER from 4.18 to 3.75 and CV CER from 28.6 to 25.7; an equal-norm random vector degraded performance below prompting alone.Source: §4.3 · p. 4
5. CONCLUSION
The conclusion presents low-resource, training-free CAA as a way to induce task-specific behaviors in SpeechLLMs, where speech tasks are linearly encoded in representations. It reports gains across benchmarks and describes transfer, semantic preservation, and script normalization.
5. CONCLUSION
- CAA improves over prompting alone on nearly all ASR, ER, and AC benchmarks across two SpeechLLMs, while reducing sensitivity to prompt wording.The authors also describe it as an alternative to low-resource SFT when no textual instruction is used.Source: §5 · p. 4
- The low-resource, training-free protocol extracts and induces task-specific behaviors through CAA, showing that speech tasks are linearly encoded in SpeechLLM representations.Source: §5 · p. 4
- Task steering vectors are localized and transfer to out-of-domain datasets, with negligible impact on ASR WER/CER.Source: §5 · p. 4
- Applying the appropriate CAA vector to input audio embeddings preserves input semantics and yields better overall steering performance.Source: §5 · p. 4
- A text-only script-normalization direction requires no speech data and can enforce a target alphabet for non-Latin-script languages, reducing ASR CER while preserving transcription semantics.Source: §5 · p. 4
7. COMPLIANCE WITH ETHICAL STANDARDS
The research used existing human-subject speech data retrospectively. No new human-subject data were collected, and ethical approval was not required under the corpus licenses.
7. COMPLIANCE WITH ETHICAL STANDARDS
- The study retrospectively used speech data made available through open access or institutional licenses.The listed corpora are LibriSpeech, CommonVoice, IEMOCAP, and AudioCaps.Source: §7 · p. 5
- No new data were collected from human subjects.Source: §7 · p. 5
- Ethical approval was not required, as confirmed by the corpus licenses.Source: §7 · p. 5