PaperlayerView exact revision on arXiv

Source-linked AI summary

Where Does the Sound Go? Tracing Acoustic Information Loss in Audio-Conditioned LLMs

Song-ha Jo, Sehyun Lee, Soyoon Kim, Jaesik Choi, Sanghyuk Choi

arXiv:2609.05871v1cs.SDcs.AI

TL;DR

Audio-LLMs often underuse acoustic cues, and it is unclear whether ASR-supervised frontends discard them before they reach the LM. The study compares encoder families and traces acoustic information through the model, finding recoverable structure alongside weak MCQA readout and evidence that readout alignment can be a dominant bottleneck in this controlled setting.

  • Problem

    Audio-LLMs often rely on transcript-like content while underusing acoustic cues such as prosody, emotion, and non-speech sounds, and the origin of this gap in the pipeline is unclear.

    Source: §1 · p. 1

  • Method

    The study compares Whisper-Tiny and Whisper-Small with reconstruction-based encoders in a shared Qwen3.5-4B pipeline and traces task-relevant information through model stages using probes, geometric analyses, and a targeted LM-head intervention.

    Source: §1 · p. 2

  • Results

    Encoder replacement alone does not improve emotion recognition or sound captioning over Whisper, while acoustic structure remains recoverable at the final LM layer despite weak MCQA predictions.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    In this controlled setting, locating the bottleneck supports targeting readout alignment rather than assuming acoustic underuse results solely from encoder-side loss.

    Source: §1 · p. 2, §5 · p. 9

    The readout analysis covers MCQA-format classification, and whether its bottleneck persists at larger scales or under different instruction-tuning regimes remains open.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Audio-conditioned language models often underuse acoustic cues such as prosody, emotion, and non-speech sounds, raising the question of whether ASR-supervised frontends discard this information before it reaches the LM. We test whether the frontend is responsible by comparing Whisper-Tiny and Whisper-Small with EnCodec, DAC-VAE, and WavTokenizer in a shared Qwen3.5-4B audio-LM pipeline on ASR, emotion recognition, and sound captioning. Encoder replacement alone does not resolve this underuse: Whisper variants remain strongest overall, including on emotion and environmental sound captioning. To localize the failure, we trace task-relevant information through the encoder, projector, LM layers, and LM head. Linear probes and geometric analyses show that discriminative acoustic structure remains recoverable at the final LM layer, even when MCQA accuracy trails probe accuracy by up to 83 points. Because the answer format and decoding procedure are controlled, this task-dependent gap points to content-specific readout failure rather than generic format bias. LogitLens analyses and a targeted LM head intervention support the conclusion that acoustic underuse is not explained solely by encoder-side information loss and that readout alignment can be a dominant bottleneck.

1 Introduction

Audio-LLMs often rely on transcript-like content while underusing prosody, emotion, and non-speech cues, but where this gap originates in the pipeline is unclear. The study finds that encoder replacement alone is insufficient and that acoustic structure can remain recoverable even when MCQA readout is weak.

1 Introduction

  • The study traces acoustic information across the encoder–projector–LM pipeline and tests whether it remains recoverable or is lost at each stage.The analysis includes the encoder, projector, LM internals, and LM head.Source: Figure 1 · p. 1
  • The study tests whether ASR-supervised frontends cause acoustic underuse because their objectives may favor lexical invariance over paralinguistic detail.Reconstruction-based codecs are used as a diagnostic counterfactual because they are trained to reconstruct waveforms.Source: §1 · pp. 1–2
  • Encoder replacement alone is insufficient: Whisper variants match or outperform EnCodec, DAC-VAE, and WavTokenizer on emotion recognition and sound captioning, while leading on ASR.The comparison uses a shared Qwen3.5-4B LM and identically trained projectors.Source: §1 · p. 2
  • Probe accuracy exceeds MCQA accuracy by as much as 83 pp on NSynth, although task-relevant acoustic structure remains linearly recoverable at the final LM hidden states.Under the same prompt template, letter set, and decoding procedure, the task-dependent gap points to a content-specific readout problem rather than generic formatting bias.Source: §1 · p. 2
  • Freezing the stack and updating only choice-letter rows of WU provides readout-level causal evidence; the study identifies LM-head alignment as a useful diagnostic for underuse.The authors frame readout alignment as a possible dominant bottleneck in their controlled setting, not as the sole explanation in all settings.Source: §1 · p. 2

2 Related Work

Prior work makes the audio frontend a plausible source of acoustic underuse, while leaving open whether replacing it is sufficient. This study connects pipeline-wide representation analysis and LM readout bias to the problem of acoustic underuse.

Audio frontends in audio-LLMs

  • Whisper’s ASR-oriented objective may encourage invariance to speaker, prosodic, and environmental variation, making frontend-related acoustic underuse plausible but not established.Prior probing reported Whisper underperforming self-supervised WavLM on speaker- and emotion-related tasks.Source: §2 · p. 3
  • Codec-based systems and reconstruction-trained encoders offer a contrast to transcription-oriented frontends, but prior work had not established whether frontend replacement explains acoustic underuse.The study compares these frontend families under a shared LM setup.Source: §2 · p. 3

Acoustic underuse and representation analysis

  • LISTEN shows that audio-LLMs can follow transcript-like content over conflicting acoustic emotion cues, but does not localize where those cues are lost or suppressed.Possible stages include the frontend, projector, LM, or final LM head.Source: §2 · p. 3
  • Earlier probing and similarity work often focused on one encoder or stopped at its output; this study extends diagnostics across the full encoder–projector–LM pipeline.It uses identical probing targets at each stage to distinguish representation-level loss from later readout failure.Source: §2 · p. 3

Readout bias in verbalized classification

  • Prior verbalized-classification work highlights sensitivity to label words and answer formats; this study links that readout problem to acoustic underuse in audio-LLMs.The paper reports recoverable acoustic structure in the final LM layer alongside failures to map it to the correct choice-letter tokens.Source: §2 · p. 3

3 Experiment Design

The experiments compare five frozen audio encoders in a shared encoder–projector–LM architecture, training the projector and evaluating a multi-task suite with common decoding and evaluation procedures.

3.1 Architecture

  • The system uses a frozen audio encoder, a trainable projector that aligns its latent sequence with LM embeddings, and a pretrained autoregressive LM.The projector is a four-layer causal Transformer decoder.Source: §3.1 · p. 3
  • Five pretrained encoders span reconstruction-based EnCodec, DAC-VAE, and WavTokenizer and ASR-based Whisper-Small and Whisper-Tiny; all encoders remain frozen.The study uses the encode path for reconstruction-based models and the encoder stack for ASR-based models; self-supervised encoders are outside this comparison.Source: §3.1 · pp. 3–4
  • Training first updates only the projector, then uses LM LoRA while continuing projector updates; the main experiments use Qwen3.5-4B and a multi-task English ASR, sound-captioning, and emotion mixture.Appendix D.4 repeats the analysis with Ministral3-3B (Instruct).Source: §3.1 · p. 4

3.2 Analysis Settings and Methods

  • Evaluation covers ASR, emotion recognition, and sound captioning datasets, using task-specific metrics and greedy decoding with identical evaluation code, tokenization, and normalization across encoders.ASR uses WER and CER; emotion uses accuracy and macro-F1; captioning uses BLEU-1, BLEU-4, and CIDEr.Source: §3.2 · p. 4

4 Analysis

Encoder choice changes downstream performance but does not remove acoustic underuse; stage-wise diagnostics find acoustic structure recoverable through the final LM layer even when MCQA predictions are weak.

4.1 Acoustic Underuse Persists across Encoders

  • Whisper-Small leads every task family on average, followed by Whisper-Tiny, yet substantial headroom remains on emotion and captioning across encoders.The encoder ranking is descriptive because candidates differ in training data and architecture.Source: §4.1 · p. 4

4.2 Acoustic Information Survives Encoder, Projector, and LM Internals

  • LoRA effects vary by encoder: Whisper-Tiny gains little on most non-ASR metrics and regresses on FSD50K captioning, while weaker-align encoders recover captioning headroom.EnCodec also substantially reduces ASR WER; family-level effects are summarized in Table 4.Source: Table 4 · p. 6, §4.2 · p. 6
  • Probe recoverability increases from encoder to projector on non-ASR tasks and is largely retained through LM depth, remaining above final-LM MCQA accuracy.The probes are fitted separately, so they establish information is present in hidden states, not that the model routes it to the answer.Source: §4.1 · p. 4, §4.2 · p. 5, Figure 2 · p. 7
  • All speaker-geometry ratios stay below 1 yet above collapse; DAC-VAE falls from 0.978 at encoder output to 0.882 at lm.out, while Whisper-Small and Tiny reach 0.897 and 0.888.The below-1 values indicate transcript-dominated geometry, while speaker identity remains preserved across the pipeline.Source: §4.2 · p. 5, Figure 3 · p. 7
  • On LibriSpeech, LogitLens top-1 accuracy changes from 0.873 to 0.882 for Whisper-Tiny but from 0.331 to 0.520 for EnCodec after fine-tuning.AudioCaps shows a flat align-to-fine-tuning trajectory, consistent with little additional captioning structure emerging inside the LM.Source: §4.2 · p. 6, Figure 4 · p. 7

4.3 The LM Readout, Not Its Representations, Drives MCQA Failure

  • Audio adds 27 to 74 pp on speech tasks but near-zero or negative gains on music tasks, with this split consistent across encoders.The pattern points to the LM’s answer mapping rather than a purely encoder-side explanation.Source: Figure 5 · p. 8, §4.3 · p. 8
  • Fine-tuning only WU’s choice-letter rows recovers 30 to 42 pp of MCQA accuracy on average across five encoders and six tasks.The update changes approximately 10–26k scalars per task while the encoder, projector, and LM internals remain frozen.Source: Figure 6 · p. 8, §4.3 · p. 8

5 Conclusion

Encoder replacement alone does not resolve acoustic underuse: acoustic structure remains recoverable at the final LM layer even when MCQA predictions are weak. The study points to readout alignment as a dominant bottleneck in this controlled setting.

5 Conclusion

  • Whisper replacement with reconstruction-based encoders did not improve emotion recognition or sound captioning, while acoustic structure remained recoverable at the final LM layer.The conclusion distinguishes recoverable information from what MCQA predictions use.Source: §5 · p. 8
  • Fine-tuning a small subset of LM-head choice-letter rows recovered accuracy across all five encoders, whereas LM-internal LoRA gains were limited and encoder-dependent.This supports readout alignment as a dominant bottleneck in the tested setting.Source: §5 · p. 8
  • Stage-by-stage localization separates information that the model fails to encode from information it fails to use, supporting diagnosis before targeting a bottleneck.Source: §5 · p. 9

Limitations

The study’s readout analysis is limited in scale and task coverage, and its encoder rankings do not isolate encoder effects. The model setup uses a shared projector and injects audio conditioning at the input-embedding level.

Limitations

  • Readout localization covers MCQA-format classification, while speaker-geometry analysis covers only IEMOCAP Session 5.Whether the readout bottleneck persists at larger scales or under different instruction-tuning regimes remains open.Source: Limitations · p. 9
  • Off-the-shelf encoders make cross-encoder rankings end-to-end comparisons rather than controlled attributions of capacity or pretraining differences.Source: Limitations · p. 9

A.1 Model

  • The projector uses four decoder layers and shared output components, with only its encoder-dimensional input projection varying across encoder configurations.Audio conditioning is injected at the input-embedding level; the pretrained LM architecture and weights are unchanged.Source: §A.1 · p. 12

A.2 Training

Training interleaves ASR, environmental-sound, and emotion data, with audio capped at 30 seconds and modality-specific sampling weights. The row-level schedule produces uneven modality exposure, and the authors note possible sound-dataset overlap.

A.2 Training

  • The training mixture combines ASR, environmental-sound, and emotion corpora, interleaving rows with sampling weights of 0.65/0.25/0.10 and stopping when all datasets are exhausted.Audio is capped at 30 seconds and resampled to each encoder’s native rate.Source: §A.2 · p. 12, Table 6 · p. 13
  • Samples use a ChatML wrapper with audio placeholders, task-specific instructions, and a response-only causal cross-entropy objective.System, user, audio-placeholder, and instruction tokens are masked from the loss.Source: §A.2 · pp. 12–13
  • Because the sound collections draw on overlapping public sources, intra-LAION and inter-corpus overlap cannot be ruled out; evaluation uses official held-out splits to mitigate leakage.Source: §A.2 · p. 12
  • DAC-VAE training consumes about 0.51 ASR epochs versus 4.41 environmental-sound and 24.3 emotion epochs, so some ASR shards may be skipped while small emotion subsets recur many times.Other encoders consume roughly twice as many rows.Source: §A.2 · p. 13
  • Row-level interleaving trades off modality exposure, and the authors leave per-modality early stopping or annealed sampling weights to future work.Source: §A.2 · pp. 12–13
  • The two training phases update the projector alone for alignment, then add LM LoRA adapters for fine-tuning; their shared hyperparameters differ in peak learning rate.DAC-VAE uses a halved global batch, while EnCodec uses a larger token cutoff to fit its audio sequence.Source: §A.2 · p. 13, Table 5 · p. 13, Table 7 · p. 13

B.1 Dataset Summary

The appendix describes datasets and per-task metadata for evaluations spanning speech, vocal, environmental-sound, and music domains. Benchmark details specify task metrics and filtering, including caption-based evaluation for four sound datasets.

B.1 Dataset Summary

  • The dataset summary spans speech, vocal, environmental-sound, and music domains, with some tasks using collapsed class sets or random-distractor MCQA.The listed filtering examples include a four-class IEMOCAP subset and a five-supercategory ESC-50 collapse.Source: §B.1 · pp. 13–14, Table 8 · p. 14

B.3 Per-Benchmark Results

  • FSD50K, AudioSet, AudioCaps, and Clotho are evaluated as captioning tasks with BLEU and CIDEr, while IEMOCAP Session 5 uses a filtered four-class split with 1,241 test utterances.The IEMOCAP filtering merges excited into happy and drops other labels; MELD uses its official held-out test split.Source: §B.3 · pp. 14–15, Table 10 · p. 15

B.2 Per-Task Metadata

  • Per-task metadata records sample count, class count, class balance, speech content, median duration, and chance accuracy; FSD50K and AudioSet use four-way MCQA with random distractors.Source: Table 9 · p. 14, §B.2 · p. 14

C LoRA Fine-Tuning: Per-Encoder Details

LoRA’s effects vary by encoder and task: gains are greatest where aligned performance leaves headroom, while strong captioning can regress. Probe accuracy changes little even when MCQA improves.

C LoRA Fine-Tuning: Per-Encoder Details

  • On Whisper-Tiny, LoRA stays near zero on most non-ASR metrics but sharply regresses FSD50K captioning, where align was already strong.The passage describes this pattern as consistent with overfitting to the ASR-supervised path.Source: C LoRA Fine-Tuning: Per-Encoder Details · p. 15
  • On EnCodec and DAC-VAE, LoRA produces substantial gains in headroom-rich tasks but does not improve AudioCaps or Clotho captioning; FSD50K shows the clearest alignment potential.FSD50K recovery is large for both encoders, while emotion changes remain small for DAC-VAE.Source: C LoRA Fine-Tuning: Per-Encoder Details · p. 15
  • Across Whisper-Tiny’s six representative tasks, L31 probe accuracy changes by at most ±1.5 pp on five and −3.1 pp on VocalSound, while MCQA rises on every non-emotion task.The probe-to-MCQA gap shrinks on all five non-emotion tasks, most on GTZAN and NSynth; IEMOCAP is the exception because align MCQA already exceeded its probe by 23.0 points.Source: C LoRA Fine-Tuning: Per-Encoder Details · p. 16, Table 12 · p. 16

D.1 LogitLens across All Tasks

LogitLens traces candidate-letter accuracy across five LM depths. Speech-task readout becomes strong by the final layer, while music and environmental tasks remain near chance despite recoverable encoder-stage information.

D.1 LogitLens across All Tasks

  • Figure 7 decodes candidate-letter logits at L0, L8, L15, L23, and L31 using final RMSNorm and WU; its shaded band shows E4 encoder-probe accuracy ranges.Each curve represents one encoder family.Source: §D.1 · p. 16, Figure 7 · p. 17
  • Speech tasks reach high L31 LogitLens accuracy that tracks audio-on MCQA, whereas music tasks and UrbanSound8K stay near chance at every depth despite high encoder-probe accuracy.For NSynth, the encoder-probe range is 38–97%; for GTZAN, it is 47–86%.Source: Figure 7 · p. 17, §D.1 · p. 18
  • Figure 8 shows that fine-tuning choice-letter rows yields large recoveries on five tasks, up to +78 pp, while IEMOCAP recovery is near zero.The before-fine-tune bars match final-layer LogitLens accuracy; gains across encoders mirror the Figure 7 curves.Source: Figure 8 · p. 17

D.2 WU Letter-Row Fine-Tune by Task

Task-specific letter-row updates recover MCQA accuracy without changing the encoder, projector, or LM internals. Vocabulary restriction alone has little effect, and the readout gap also appears with a second base LM.

D.2 WU Letter-Row Fine-Tune by Task

  • The D.2 update trains only the candidate-letter rows on saved L31 states, freezing the encoder, projector, and all other LM weights.The reported protocol uses 200 full-batch Adam steps at learning rate 10^-3 with an 80/20 stratified split.Source: §D.2 · p. 18
  • On five tasks, fine-tuning only WU choice-letter rows recovers the readout gap, with task-specific gains up to +78 pp; IEMOCAP changes by at most a couple of points.AudioMNIST/EnCodec gains +78 pp, NSynth/Whisper-Small +58 pp, and GTZAN/EnCodec +58 pp, each from a base accuracy below 20%.Source: §D.2 · p. 18

D.3 Logit Processor Baseline: Vocabulary Restriction Alone Is Insufficient

  • Across 30 task–encoder combinations, restricting argmax to candidate letters changes accuracy by only +0.23 pp on average, while letter-row fine-tuning closes the readout gap.Candidate-letter emissions already average 98.5% and never fall below 94%, so the failure is letter-versus-letter ranking rather than non-letter emission.Source: §D.3 · pp. 18–19, Figure 9 · p. 18

D.4 Generality of the Readout Bottleneck across Base LMs

  • With Ministral3-3B, WU letter-row updates bring accuracy within about a point of the probe on every task, while direct MCQA remains far below both.This pattern holds for Whisper-Small and DAC-VAE despite their different downstream strength.Source: §D.4 · p. 19, Table 14 · p. 19

D.5 Robustness of the Readout Intervention

The readout intervention is robust across answer formats and separates learned letter-row directions from simple bias calibration. Its fitted rows use disjoint evaluation examples and leave other parameters frozen.

D.5 Robustness of the Readout Intervention

  • The robustness ablations fit letter rows on each task’s standard training split and evaluate on a disjoint partition using paired original and post-intervention scores.Only candidate-letter rows receive gradients; the remaining parameters stay frozen.Source: Table 15 · p. 19, §D.5 · p. 19
  • Across prompt, option-order, letter-mapping, and label-word variants, post-intervention accuracy stays within roughly one point of the baseline condition.AudioMNIST is an exception for label words because spelling out digits nearly saturates accuracy before intervention.Source: §D.5 · pp. 19–20, Table 16 · p. 20
  • Learning a readout direction for letter rows recovers most of the gap and approaches an unconstrained final-layer probe, while bias-only calibration recovers little.Updating the full vocabulary adds nothing further, supporting a corrected mapping from final-layer representations to answer tokens.Source: §D.5 · p. 20, Table 17 · p. 20
Loading 2609.05871v1…