Source-linked AI summary
Direct or Mediated? Task-Dependent Audio Information Routing in Large Audio Language Models
Yizhou Zhang, Wangjin Zhou, Xin Gu, Yichi Wang, Wei Tan, Yi Zhao, Zhi Gong, Keisuke Imoto, Tatsuya Kawahara
TL;DR
LALMs are usually evaluated on single coherent audio segments, leaving their behavior and internal information pathways under less conventional configurations underexplored. This paper studies concatenated audio with controlled interventions and finds task-dependent routing: ASR remains comparatively robust through direct audio-to-answer access, while AQA relies more on prompt-mediated pathways despite retained decodable audio attributes.
Problem
LALMs are typically evaluated on single coherent audio segments, leaving task robustness and internal audio-information pathways across configurations and tasks underexplored.
Method
The paper combines controlled two-segment audio concatenation, layer-wise attention knockout, and probing of task-relevant attributes in prompt-token representations across ASR and AQA.
Results
ASR remains comparatively robust and depends more on direct audio-to-answer access, whereas AQA degrades substantially and is more sensitive to audio-to-prompt and prompt-to-answer pathways.
Takeaways & Limitations
Task-dependent audio information routing and retained decodable prompt-token attributes indicate that AQA failures are not explained solely by complete disappearance of audio information.
Takeaways & Limitations
Attention-knockout results characterize pathways supporting successful standard-benchmark predictions but do not directly identify computation failures in incorrect concatenated-AQA cases.
Abstract
from arXiv · showhide
Large Audio Language Models (LALMs) have demonstrated strong performance across a wide range of audio understanding tasks. However, they are typically evaluated on single, coherent audio segments, leaving their behavior under less familiar input configurations underexplored. We study this issue through a controlled setting in which two audio segments are concatenated into a single input. Across multiple LALMs, we observe a striking task-dependent robustness gap: automatic speech recognition (ASR) remains comparatively stable, whereas audio question answering (AQA) degrades substantially. To investigate the mechanisms underlying this disparity, we analyze how audio information is routed through LALM decoders using layer-wise attention knockout. The results reveal distinct task-dependent pathways. ASR relies primarily on direct retrieval from audio tokens by answer tokens, whereas AQA depends more strongly on a mediated route in which audio information is first integrated into prompt tokens and subsequently accessed during generation. We further probe prompt-token representations under audio concatenation and find that task-relevant audio attributes remain readily decodable, particularly in middle and later decoder layers, even when AQA performance deteriorates sharply. This dissociation indicates that the failure cannot be explained by complete loss of audio information from the decoder states and is instead consistent with a downstream bottleneck in retrieving or utilizing prompt-mediated information during answer generation. Together, our findings reveal task-dependent audio information routing in LALMs and highlight information utilization as a potential limitation on their generalization.
Introduction
The introduction frames a controlled two-segment audio-concatenation test of LALMs, revealing a task-dependent robustness gap between comparatively stable ASR and substantially degraded AQA. Layer-wise analyses indicate distinct routing mechanisms, with ASR relying on direct audio-to-answer access and AQA relying more on prompt-mediated pathways despite retained decodable attributes.
- Motivation: Most LALMs are evaluated on familiar, single-segment audio, leaving robustness and audio grounding under less conventional input configurations underexplored.Prior work also reports insufficient audio grounding and hallucinated responses when inputs depart from familiar settings.
- Task distinction: ASR primarily recovers linguistic content, whereas AQA must identify evidence, preserve attributes, and retrieve that evidence in response to a question.The introduction asks whether these task differences correspond to distinct internal routing mechanisms.
- Robustness gap: Across four LALMs, ASR remains comparatively stable under two-segment concatenation, whereas AQA performance deteriorates substantially.The study concatenates two waveform-level audio segments and evaluates the same models on both tasks.
- Information routing: Layer-wise attention knockout shows that ASR depends on direct answer-token access to audiotoken representations, while AQA depends more on audio-to-prompt and prompt-to-answer pathways.Direct audio-to-answer sensitivity is comparatively weak in AQA, whose routing is more prompt-mediated.
- Representation probing: Task-relevant attributes remain linearly decodable from prompt-token representations, particularly in middle and later decoder layers, even when two-segment AQA accuracy is low.This means the failure cannot be explained solely by complete disappearance of the probed attributes from decoder states.
Related Work
Prior work establishes LALMs as unified frameworks for audio understanding, while distinguishing ASR’s transcription-focused demands from AQA’s question-conditioned evidence use. Existing robustness studies largely characterize failures behaviorally, motivating controlled input perturbations and internal probing of multimodal information processing.
- Audio understanding tasks: LALMs unify diverse audio understanding tasks, with ASR systems projecting speech representations into an LLM embedding space for autoregressive transcription.Prior ASR research addresses cross-modal alignment, contextual conditioning, multilingual recognition, streaming inference, and real-world acoustic robustness.
- Audio understanding tasks: AQA requires recognizing acoustic or linguistic content, identifying question-relevant evidence, and generating an answer conditioned on the question.The field has progressed from task-specific datasets and architectures to broader benchmarks for open-ended audio understanding and reasoning.
- Robustness: Robustness research has examined hallucinations, adversarial and naturally occurring failures, temporal localization, long-audio understanding, multiple audio inputs, and reasoning across clips.These directions are represented by AHa-Bench, HalluAudio, AHA, HALAS, and studies of temporal and multi-clip processing limitations.
- Robustness: Prior studies show that strong conventional-benchmark performance may not ensure reliable behavior under challenging input conditions, with failures usually characterized behaviorally through benchmark construction and evaluation.This work instead uses controlled waveform-level concatenation to expose a task-dependent ASR–AQA robustness gap and examine associated internal routing mechanisms.
- Mechanistic analysis: Probing and causal-intervention studies analyze multimodal information processing across decoder layers and token positions, including grounding, cross-modal integration, reasoning, and answer decoding.Related speech and audio-language-model work uses probing, activation patching, and attention-based analysis.
Methods
The methods combine controlled audio concatenation with layer-wise attention knockout and probing to analyze task-dependent robustness and audio-information flow in LALM decoders. The evaluation distinguishes direct audio-to-answer retrieval, audio-to-prompt transfer, and prompt-mediated retrieval while testing whether task-relevant attributes remain decodable.
- Method Overview: The study uses audio concatenation, attention knockout, and layer-wise probing as complementary tools for analyzing audio-information flow in LALM decoders.The decoder sequence is partitioned into audio tokens, prompt tokens, and autoregressive tokens.
- Concatenated-Audio Evaluation: Composite inputs concatenate two waveform-level audio segments, with ASR requiring ordered transcription and AQA testing content, order, and location attributes.AQA content questions include distractor candidate attributes, while order and location questions target segment-wise organization.
- Attention Knockout: Layer-wise attention knockout suppresses selected directed attention edges within contiguous decoder-layer windows while leaving other attention connections unchanged.The intervention sets selected attention logits to −∞ after softmax and evaluates effects on cleanly solved examples.
- Attention Knockout: The three tested pathways are P ←A, Y ←A, and Y ←P, representing audio-to-prompt transfer, direct audio-to-answer retrieval, and prompt-mediated retrieval.Intervention effects are interpreted through performance degradation relative to clean inference, with larger degradation indicating greater functional importance.
- Layer-wise Probing: Layer-wise probing mean-pools prompt-token hidden states and uses an independently trained probe to measure linear decodability of task-relevant audio attributes.High probing accuracy alongside poor end-task performance indicates that the attribute remains decodable from the prompt representation.
Experimental Setup
The study evaluates four LALMs on concatenated-audio ASR and AQA, then uses attention knockout and layer-wise probing to analyze decoder information routing and representation. Inputs are constructed from waveform-level concatenation after resampling audio to 16 kHz, with task-specific datasets, controls, and evaluation metrics.
- Models and analyses: Four LALMs are evaluated on concatenated-audio ASR and AQA, followed by attention-knockout and layer-wise probing analyses of decoder information routing.The models are Audio-Flamingo-Next-Instruct, MiMo-Audio-Instruct, Step-Audio-R1, and Qwen3-Omni-Instruct.
- Audio construction: All clips are resampled to 16 kHz and concatenated directly at the waveform level without added silence or explicit boundary markers.Each model then applies its official audio preprocessing pipeline.
- ASR evaluation: Concatenated ASR uses LibriSpeech test-clean utterance pairs, orders front and back transcripts sequentially, and reports WER and CER against single-segment controls.The ASR intervention analysis also uses LibriSpeech because its controlled speech-only setting reduces variation unrelated to audio-to-text information flow.
- AQA evaluation: Concatenated AQA uses balanced multiple-choice questions on SpeechCommands, GTZAN, and ESC-50, covering content, order, and location, with accuracy averaged across question types.Single-segment controls test attribute recognition for each constituent clip using randomized four-option questions.
- Mechanistic analyses: Attention knockout evaluates ASR with WER and CER and AQA with multiple-choice accuracy on MMAU test-mini across independent contiguous five-layer windows.The intervention examines P ←A, Y ←A, and Y ←P, reporting performance changes relative to clean inference.
Experiments
Waveform concatenation exposes a task-dependent robustness gap: ASR remains comparatively stable, whereas AQA degrades substantially despite individually recoverable audio attributes. Pathway interventions and probing indicate that ASR favors direct audio-to-answer access, while AQA depends more on prompt-mediated computation whose information remains present but is not reliably used.
- Behavioral robustness: ASR remains comparatively robust after concatenation, with WER increasing by only 0.10–1.30 percentage points across models and staying below 4.5%.The linguistic content of both segments remains largely recoverable after waveform-level concatenation.
- Behavioral robustness: AQA accuracy falls to 35.33–67.33% for Audio-Flamingo-Next, MiMo-Audio, and Step-Audio-R1 across three datasets, despite isolated-clip accuracies of 85.17%–98.00%.The concatenated-input decline cannot be attributed simply to lacking basic attribute-recognition ability.
- Information routing: Layer-wise knockout shows ASR depends more on direct audio-to-answer access, whereas AQA is more sensitive to audio integration into prompt states and prompt-to-answer influence.The contrast concerns relative pathway importance rather than exclusive routing mechanisms.
- Prompt-state probing: Prompt-token probing shows constituent-segment attributes become increasingly decodable through early and middle layers and remain relatively high in later layers.Best macro-averaged accuracies reach 86.8% on SpeechCommands, 90.1% on GTZAN, and 81.8% on ESC-50.
- Prompt-state probing: Task-relevant audio attributes remain present in prompt states when AQA fails, particularly in middle and later layers, ruling out complete disappearance of relevant audio information.The findings instead implicate a downstream difficulty retrieving or utilizing prompt-mediated information during answer generation.
Conclusion
Across four Large Audio Language Models, concatenated two-segment inputs exposed task-dependent differences: ASR remained comparatively robust, while multi-segment AQA declined and relied more on mediated audio-to-prompt-to-answer pathways. Task-relevant attributes often remained linearly decodable from prompt-token states, separating representational availability from behavioral robustness.
- Behavioral robustness: Across four Large Audio Language Models, ASR remained comparatively robust under two-segment waveform concatenation, whereas multi-segment AQA performance was substantially lower than single-segment controls.The controls were attribute-recognition controls.
- Pathway sensitivity: Attention knockout showed that ASR depended more on direct audio-to-answer access, while AQA was generally more sensitive to audio-to-prompt and prompt-to-answer pathways.These results indicate different task-dependent pathway sensitivities.
- Representational decodability: Under concatenated inputs, task-relevant attributes often remained linearly decodable from prompt-token states despite task-dependent differences in behavioral robustness and pathway sensitivity.Together, the findings distinguish representational decodability from task performance.
Limitations
The study’s mechanistic analysis is indirect because attention knockout is applied to reliably solved standard examples rather than incorrect concatenated-AQA cases. Its controlled two-segment concatenation setting also simplifies realistic multi-audio conditions, motivating broader evaluation.
- Mechanistic analysis: Attention knockout characterizes pathways supporting successful predictions but does not directly locate failures in incorrect concatenated-AQA cases.Interventions require correct and stable clean predictions, so analysis uses standard benchmark examples that models solve reliably.
- Evaluation setting: The experiments study only a controlled two-segment waveform-concatenation setting, which may not capture realistic multi-audio challenges.The simplified design facilitates systematic analysis but limits ecological coverage.
- Future directions: Future work should test overlapping audio, background noise, longer recordings, additional segments, and more complex cross-segment interactions.These conditions would assess whether the observed patterns generalize beyond simple concatenation.