Source-linked AI summary
Are Audio-Language Models Listening? Audio-Specialist Heads for Adaptive Audio Steering
Neta Glazer, Lenny Aharon, Ethan Fetaya
TL;DR
LALMs can under-use decisive audio evidence because linguistic priors dominate multimodal predictions. The paper identifies audio-specialist heads as listening signals, uses audio–silence activation steering to amplify audio effects, and reports MMAU accuracy gains of up to +8.05 percentage points without parameter updates.
Problem
Text dominance can cause LALMs to rely on linguistic priors instead of informative audio evidence, motivating indicators of audio engagement.
Method
The paper identifies audio-specialist attention heads from audio attention predictive of correctness and uses their localization to construct an audio–silence steering direction for inference-time activation intervention.
Results
Head-guided layer steering improves MMAU test-mini accuracy from 49.20%→57.25% on Qwen2-Audio (+8.05 pp) and 64.50%→69.40% on R1-AQA (+4.90 pp).
Takeaways & Limitations
The listening signal tracks when audio affects decisions, and localized steering provides an actionable way to amplify audio effects without parameter updates.
Abstract
from arXiv · showhide
Multimodal large language models can exhibit text dominance, over-relying on linguistic priors instead of grounding predictions in non-text inputs. One example is large audio-language models (LALMs) where decisive audio evidence can be under-utilized even when it contains important information. To address this issue we use mechanistic interpretability to identify a small set of audio-specialist attention heads whose audio attention yields a ``listening'' signal. We show that this signal increases when audio evidence affects the model's output, providing an indicator of audio engagement under standard prompting. Leveraging this localization, we construct an audio--silence steering direction and apply an inference-time activation intervention to the final representation, amplifying the model's audio effect. To demonstrate the utility of this intervention, we show on MMAU that this improves accuracy by up to +8.0 percentage points on two Qwen-based LALMs, without any parameter updates.
1. Introduction
LALMs can over-rely on linguistic priors even when audio evidence is informative. This work uses mechanistic interpretability to identify audio engagement signals and steer inference toward audio evidence.
- Text dominance causes LALMs to rely disproportionately on linguistic cues despite informative non-text evidence.The imbalance is linked to fusion design choices and attention dilution from redundant non-text tokens.
- Mechanistic interpretability localizes internal components that causally drive model behavior and supports inference-time activation steering.The framework extends from text-only LLMs to multimodal architectures and uses interventions such as ablation and activation patching.
- The paper asks whether head-level signals can indicate audio engagement and provide a practical handle for inference-time steering.
- Audio-specialist heads yield an instance-level listening signal, while controlled activation intervention amplifies audio effects and improves MMAU performance without parameter updates.The evaluation covers Qwen2-Audio-7B and R1-AQA.
2. Related Work
Prior work characterizes text dominance and modality arbitration failures in audio-language models, while mechanistic interpretability motivates localized analysis and intervention. The paper’s steering approach builds on this component-level perspective.
- Text Dominance in Multi Modal LLMs: LALMs may let linguistic cues override informative audio evidence, including when text directly contradicts acoustic ground truth.Speech-affect evaluations also find failures to separate acoustic prosody from lexical content.
- Specialist-Guided Steering: Specialist-Guided Steering identifies audio-specialist heads using audio-attention share predictive of correctness, then derives a steering direction from audio–silence residual differences.The direction is aggregated over layers containing the discovered heads and added to the final representation after scaling by β.
- Mechanistic Interpretability: Mechanistic interpretability studies localized transformer mechanisms and specialized attention-head roles to support targeted interventions.
3. Preliminaries and Notation
The paper defines audio attention and residual-stream quantities for analyzing audio use in a unified text-audio token sequence. Matched-duration silence provides the counterfactual baseline for isolating audio effects and steering.
- Audio-language transformers process text and audio tokens in one sequence, with Aℓ,h[i,j](x) denoting attention from query position i to key position j.The attention matrix has dimensions n×n and rows summing to one.
- For the final prompt position, aℓ,h(x) measures the fraction of a head’s attention directed to audio tokens.The final position is the last prompt token before generation.
- Residual-stream states hℓ(x) and hfinal(x) represent the final prompt position after layer ℓ and at the final layer, respectively.
- Audio ablation replaces each original audio input with matched-duration silence to isolate its effect on residual representations.The original and silenced inputs are denoted xaud and xsil.
- Steering modifies internal activations during the forward pass before predictions are computed with the language-modeling head.
4. Method
The method localizes audio-specialist attention heads using audio attention and correctness, then uses those heads to guide audio–silence activation steering. It also evaluates head-level and layer-localized intervention designs.
- Discovering Audio-Specialist Heads: The approach first selects audio-specialist heads whose final-position audio attention is associated with correctness on a calibration split.The top-K heads are ranked by the absolute correlation between audio attention and binary correctness, with K=20 used for specialist selection.
- Steering: Figure 2 varies steering strength β and specialist count K while measuring performance improvement in percentage points for R1-AQA and Qwen2-Audio-7B.Each K defines a specialist layer set L for layer-localized steering.
- Discovering Audio-Specialist Heads: The specialist heads provide an instance-level listening indicator by aggregating signed, correlation-weighted audio attention.The validation protocol tests whether this score predicts correctness and rises when audio-conditioned and audio-ablated predictions differ.
- Steering: Layer-localized steering restricts the intervention to the specialist layer set induced by the selected heads, while a direct head-level intervention serves as a comparison.At each selected layer, specialist head deltas are mapped into residual-stream space and added after the attention sublayer before normal continuation.
- Steering: The intervention constructs an audio–silence steering direction from two forward passes and applies it to the final representation with strength β.Predictions are then computed from the modified representation through the language-modeling head.
5. Experimental Setup
The experiments evaluate inference-time steering on MMAU using two Qwen-based audio-language models in a standardized multiple-choice accuracy protocol. They compare specialist-guided interventions with unsteered, single-layer, head-level, and matched random-head controls.
- Benchmark: MMAU contains audio-question multiple-choice examples across speech, environmental sound, and music, covering 27 skills.Accuracy is reported on the labeled 1,000-example test-mini split and by domain.
- Baselines and Interventions: The intervention comparisons include no steering, best single-layer audio–silence steering, specialist or random head-level steering, matched random-head layer steering, and specialist-guided layer steering.Random controls use the same specialist count and intervention procedure.
- Models: The study evaluates Qwen2-Audio-7B-Instruct and R1-AQA, an RL-optimized audio question-answering model built on the Qwen backbone.Both models process audio-conditioned tokens jointly with text in a single self-attention sequence.
- Evaluation Protocol: MMAU is scored as a 4-way multiple-choice task using the next-token logit for each answer label at the final prompt position.The highest-scoring option is selected, accuracy measures agreement with the ground-truth label, and paired comparisons use McNemar’s test.
- Implementation: Specialist selection examines all 1,024 attention heads and chooses the top K=20 by the absolute correlation between audio attention and correctness.Residual-stream states are cached for all layers, with per-head attention outputs additionally cached for the head-level baseline.
6. Results
The listening signal tracks when audio affects model decisions, while specialist-guided layer steering improves MMAU accuracy across models and domains. Matched random-head controls perform worse, and excessive steering can degrade performance.
- Listening signal: The specialist listening score predicts correctness and increases when audio changes the model’s predicted option.The increase is statistically significant (p < 0.001).
- Accuracy gains: +8.05 pp raises Qwen2-Audio accuracy from 49.20% to 57.25%, while +4.90 pp raises R1-AQA accuracy from 64.50% to 69.40%.Head-guided layer steering outperforms the best single-layer baseline.
- Domain breakdown: Improvements span speech, environmental sound, and music, with Qwen2-Audio’s largest gain in SPEECH (+14.1 pp) and R1-AQA’s in SOUND (+7.5 pp).Qwen2-Audio also improves SOUND by +4.9 pp and MUSIC by +5.1 pp; R1-AQA improves SPEECH by +3.3 pp and MUSIC by +3.9 pp.
- Selection and sensitivity: Matched random-head steering yields much smaller improvements than specialist-selected steering across K.The comparison uses the same layer-guided steering procedure for both selections.
- Selection and sensitivity: Performance peaks at moderate β and degrades for overly large β, while K≈20 is typically near-optimal.The induced specialist layer set remains sparse as K increases.
7. Discussion
Mechanistic interpretability identifies audio engagement signals and supports inference-time steering that amplifies audio effects without parameter updates. The results characterize text dominance as a diagnosable and steerable failure mode.
- Discussion: Audio-specialist heads provide an instance-level indicator of audio engagement, while audio–silence intervention produces consistent MMAU accuracy gains.The intervention improves performance by up to +8 pp without parameter updates.