Source-linked AI summary
What Does LALM Post-training Learn? Rethinking Audio Understanding and Reasoning Capabilities
Hua-Hsuan Chen, Xuanjun Chen, Wei-Chung Lu, Chih-Kai Yang, Yu-Xiang Lin, Sung-Feng Huang, Haibin Wu, Hung-yi Lee
TL;DR
Audio-language models often improve on audio-reasoning benchmarks after post-training, but it is unclear whether those gains reflect listening or text-based answer strategies. Controlled experiments compare audio and silence conditions and multiple-choice versus open-ended evaluation. Gains often persist without audio and largely disappear when answer options are removed, suggesting reliance on textual cues rather than broad audio-reasoning improvement.
Problem
Method
The study evaluates base and post-trained LALMs under audio and silence conditions and compares multiple-choice with open-ended formats.
Source: §1 · p. 1Results
Takeaways & Limitations
Standard benchmark scores cannot distinguish text-based shortcuts from genuine listening, motivating a rethink of how LALM audio understanding and reasoning are evaluated.
Source: §7 · pp. 8–9The evidence covers two base models, four post-training methods, six open-source models, and MMAU and MMAR; other architectures, scales, or benchmarks may behave differently.
Source: Limitations · p. 9
Abstract
from arXiv · showhide
Post-training often improves the performance of large audio-language models on audio understanding and reasoning benchmarks. However, these improvements may reflect adaptation to specific question formats, such as multiple-choice questions, rather than genuine advances in audio understanding and reasoning capabilities. To investigate what LALMs actually learn from post-training, we conduct controlled experiments that disentangle improvements in audio understanding from adaptation to specific question formats. Across different models and post-training methods, we find that much of the reported improvement persists even without informative audio, but largely disappears when the question format changes. These findings suggest that post-training often teaches models to exploit task-specific textual cues and answer patterns rather than developing generalizable audio understanding and reasoning capabilities.
1 Introduction
The paper asks whether LALM post-training improves audio understanding or instead helps models answer through text-only shortcuts. Controlled comparisons show gains persist with silence, while changing the question format sharply reduces them.
1 Introduction
- The study tests whether LALMs use audio by comparing informative-audio and silence conditions in training and evaluation, then removing answer options to test format dependence.A correct answer with audio present alone cannot rule out text-only shortcuts.Source: §1 · p. 1
- Post-training gains persist without informative audio: MMAU and MMAR gains of +14.8 and +7.2 points retain +11.2 and +4.9 points under silence, respectively.Models trained entirely on silence also improve on real audio.Source: §1 · p. 2
- In open-ended evaluation, models suffer severe performance degradation and retain negligible reported gains after post-training.The format removes multiple-choice answer options.Source: §1 · p. 2
- Audio-grounding gains are marginal, with improvements depending on audio accounting for a negligible fraction across evaluation settings.This calls into question the models’ actual audio understanding and reasoning capabilities.Source: §1 · p. 2
2 Related Work
Prior work has carried text-model post-training recipes into audio reasoning and reported benchmark gains, while evidence of audio reliance remains inconclusive. This paper addresses whether post-trained LALMs answer from audio or text capabilities.
Post-training for audio reasoning
- Audio-reasoning post-training has adopted methods developed for text, including supervised fine-tuning and reinforcement learning, with rewards that verify final answers rather than derivations.Prior work therefore primarily reports overall accuracy, the metric optimized during training.Source: §2 · p. 2
Whether LALMs listen
- Prior studies find that benchmark questions can be answered without audio, models may follow text over conflicting acoustics, and strong reasoning scores can coexist with weak perception.Whether post-trained LALMs rely on audio or text capabilities remains unclear, and absolute accuracy does not capture that distinction.Source: §2 · p. 2
3 Controlled Evaluation Protocol
The protocol separates gains associated with audio from gains that persist under silence, and tests whether performance depends on multiple-choice options. It compares post-trained models with their base models under controlled input and answer-format conditions.
3 Controlled Evaluation Protocol
- Audio Gain and Silence Gain compare post-trained performance with the base model under real-audio and silence input, respectively.Silence Gain measures how much improvement remains without audio.Source: §3 · p. 2
- Open-ended evaluation removes answer options and uses an LLM judge to assess semantic equivalence to the reference, while other evaluation settings remain unchanged.Questions that become unanswerable without options are rewritten using rules and judge prompts.Source: §3 · p. 2
- Table 1 compares Audio and Silence Gains across post-training methods.The paired gains show whether improvement is specific to audio or also persists under silence.Source: §3 · p. 2, Table 1 · p. 3
- Table 2 compares Audio and Silence Gains across open-source models.The comparison extends the silence-control analysis beyond the paper’s post-training methods.Source: Table 2 · p. 3
4 Experimental Setting
Experiments use audio-question-answering data, four post-training methods applied under a shared recipe, and two benchmarks. Open-ended answers are evaluated by an LLM judge, and post-training results are averaged over three seeds.
4 Experimental Setting
- The training pool contains 12,267 multiple-choice audio QA items across 7,129 clips and three public corpora, split into 10,446 training items and 1,821 held-out in-domain items.Audio-level matching was used to verify that no training audio leaks into evaluation.Source: §4 · p. 3
- SFT, RFT (Yuan et al., 2023), GRPO (Shao et al., 2024), and VCRL (Jiang et al., 2025) share a unified training budget and final-answer reward.The recipe uses 200 LoRA steps, three seeds, and is applied unchanged to Qwen2.5-Omni-3B and Qwen2-Audio-7B-Instruct.Source: §4 · p. 3
- Evaluation uses MMAU test-mini and MMAR, each with 1,000 items, with model-specific prompts and fixed per-benchmark settings shared with each base model.Open-ended answers are judged for semantic equivalence by Qwen3.6-27B; post-trained accuracies are averaged over three seeds at fixed checkpoints.Source: §4 · p. 3
5 The Necessity of Audio for LALMs
The section tests whether LALM post-training gains depend on audio, examining both evaluation-time silence controls and models trained on silence. Across models and fine-grained capabilities, substantial gains often persist without audio; silence-trained models also retain gains on MMAU, while MMAR shows greater reliance on training audio.
5 The Necessity of Audio for LALMs
- The evaluation compares model performance with informative audio against silence to test whether post-training gains reflect genuine reliance on audio.The section also asks whether audio is necessary during training, which it tests by replacing training audio with same-length silence.Source: §5 · p. 3, §5.3 · p. 5
5.1 Evaluating Overall Gains Across Models
- Across open-source models, Audio Gains and Silence Gains remain closely aligned; AudioMCQ’s MMAU gain is +5.0 points with audio versus +6.4 under silence.The authors report the same broad pattern across Tables 1 and 2, with textual artifacts and guessing strategies offered as the explanation.Source: §5.1 · p. 4
5.2 Evaluating Fine-Grained Capabilities
- Across MMAU reasoning and information extraction, Silence Gains are often substantial; on MMAR, semantic and perception gains are usually positive, while signal-layer results range from −7.0 to +14.0 points.The section concludes that silence gains frequently depart from the expected near-zero baseline across fine-grained capabilities.Source: §5.2 · p. 4
5.3 Post-Training with Audio versus Silence
- On MMAU, silence-trained GRPO on Qwen2-Audio-7B-Instruct retains +7.6 points versus +14.8 points after real-audio training, while MMAR retains less gain with greater variance.Both training conditions are evaluated on real audio; the authors interpret the MMAR pattern as stronger reliance on training audio.Source: Table 4 · p. 5, §5.3 · p. 5
6 What Does LALM Post-training Learn?
The section tests whether post-training gains reflect audio understanding or dependence on multiple-choice text. Across answer-pattern analysis, open-ended evaluation and training, correction tracing, and a case study, the evidence points to gains that rely heavily on textual cues and answer options.
6.1 Analyzing shifts in answer patterns
- After post-training, invalid responses decrease more than incorrect listed answers, consistent with improved adherence to the multiple-choice format rather than better discrimination among options.For Qwen2-Audio-7B-Instruct on MMAU, invalid responses fall from 17.2% to 7.6% after GRPO, while incorrect listed answers fall from 37.6% to 32.4%.Source: §6.1 · p. 5, Table 5 · p. 5
6.2 Assessing open-ended evaluation
- Most post-training gains largely fail to transfer when answer options are removed, and the results suggest models exploit textual cues rather than gain broad audio understanding.On MMAU, Qwen2-Audio-7B-Instruct’s GRPO Audio Gain falls from +14.8 to +2.3 points; VCRL falls from +12.4 to +2.1.Source: §6.2 · p. 6, Table 6 · p. 6
6.3 Examining open-ended training
- Open-ended training does not recover the lost gains: Qwen2-Audio-7B-Instruct’s MMAU Audio Gain is +2.0 with open-ended GRPO, versus +2.3 under mismatched training and +14.8 under multiple-choice.VCRL reaches only +3.0 Audio Gain, while GRPO’s Silence Gain remains +0.6.Source: §6.3 · pp. 6–7, Table 7 · p. 7
6.4 Tracing individual error transitions
- Ideal corrections that require audio and generalize beyond options are rare, while many corrections are spurious or option-bound and fail under silence and open-ended controls.The Ideal case peaks at 7 items in any setting, compared with up to 48 Spurious corrections.Source: §6.4 · p. 7, Table 8 · p. 7
6.5 Case Study: Revealing option priors
- In the whistling example, GRPO answers correctly under silence but fails when options are removed, showing the answer depends on the options rather than the audio.Across six open-ended phrasings, the post-trained model’s “Twinkle, Twinkle, Little Star” response appears only when nursery-rhyme or children’s-song terms are present.Source: §6.5 · pp. 7–8, Figure 2 · p. 8
7 Conclusion
The paper investigates what LALM post-training learns on audio understanding and reasoning benchmarks. Its controlled experiments find gains that persist without informative audio but largely vanish without answer options, motivating a rethink of how current models’ capabilities are assessed.
7 Conclusion
- Across the evaluated benchmarks, reported gains persist with silence but largely vanish when question formats change, indicating reliance on textual cues rather than improved audio perception.The conclusion describes these as text-prior shortcuts that standard benchmark scores cannot distinguish from genuine listening.Source: §7 · pp. 8–9
- The study investigates what different LALM post-training methods teach on audio understanding and reasoning benchmarks.Its controlled experiments examine whether gains reflect audio understanding or adaptation to question formats.Source: §7 · p. 8
- The results highlight the need to rethink claims about current LALMs’ audio understanding and reasoning capabilities.Standard benchmark scores cannot distinguish text-based shortcuts from genuine listening.Source: §7 · pp. 8–9
Limitations
The findings are based on a finite set of models, methods, and benchmarks, so the observed pattern may differ elsewhere. The section also documents shared training settings and the silence control used in the experiments.
Limitations
- The evidence covers two base models, four post-training methods, six open-source models, MMAU, and MMAR; other architectures, scales, or benchmarks may behave differently.The authors report a small Audio Gain–Silence Gain gap across independently developed models, not that the pattern must hold everywhere.Source: Limitations · p. 9
A Global Training Configurations
- All four methods share a LoRA training recipe, with adapters applied to attention and MLP projections, including the audio encoder’s attention projections.The only model-specific setting is microbatch size, chosen because of memory constraints; results use only the step-200 checkpoint.Source: A Global Training Configurations · p. 10
- The silence control zeroes each training waveform before feature extraction while preserving audio-token length, removing audio information without dropping the audio turn.The reward checks whether the final answer matches the gold option; unparseable responses score zero.Source: A Global Training Configurations · p. 10
B Method-Specific Implementations
The appendix specifies how the four post-training methods are implemented and how blind training alters the audio input. GRPO and VCRL differ in group selection, while SFT and RFT use verified answer targets.
B Method-Specific Implementations
- SFT trains on the gold option alone, while RFT samples eight base-model rollouts and uses every verified-correct rollout as a target.Items never solved by the base model contribute nothing to RFT’s effective training pool.Source: B Method-Specific Implementations · p. 10
- GRPO samples groups of eight rollouts and assigns each rollout its reward minus the group mean; VCRL differs by probabilistically selecting groups based on reward variance.Groups with uniformly correct or uniformly wrong rollouts are retained with zero advantage; VCRL targets κ = 0.75 after warming up from 0.3.Source: B Method-Specific Implementations · p. 11
- Blind training zeroes waveforms before feature extraction, preserving audio tokens and their length rather than removing the audio turn.The judge template fills three slots per item; blue marks the same answer and purple a different answer.Source: B Method-Specific Implementations · p. 11, Figure 3 · p. 11
C Details of Open-Ended Evaluation
The evaluation converts multiple-choice questions to open-ended form through syntactic rewrites intended to preserve what is asked. It also retains six residual option-dependent questions and uses a single text-only LLM judge.
C Details of Open-Ended Evaluation
- Rewriting removes answer-set references while preserving references to the audio stimulus, and the rules affect 172 MMAU and 407 MMAR questions.The rewritten questions are intended to keep the underlying intent unchanged.Source: C Details of Open-Ended Evaluation · p. 11
- Figure 4 presents one training item in both forms.Source: Figure 4 · p. 12
- Six questions still presuppose an option list, but retaining them changes no checkpoint’s open-ended accuracy by more than 0.13 MMAU or 0.07 MMAR points.Removing them leaves ∆Acca and ∆Accs unchanged.Source: C Details of Open-Ended Evaluation · p. 12
- Qwen3.6-27B grades every response greedily for at most eight tokens with reasoning mode off, without audio, scoring agreement between strings under a fixed system message and three per-item user-message slots.Across the 132,000 judgements in this paper there were none; a verdict containing neither word is scored as no.Source: C Details of Open-Ended Evaluation · p. 12
D Details of Open-Ended Post-Training
Open-ended post-training removes choices from the question text and changes the reward to accept gold answers expressed in free form. The reward uses normalized equality or word-boundary containment, while the remaining audio and training schedule stay the same.
D Details of Open-Ended Post-Training
- The open-ended training pool cuts option text from 12,267 questions and rewrites the remaining questions using the evaluation’s rules.Figure 4 shows an example item in both forms.Source: D Details of Open-Ended Post-Training · p. 12
- The reward accepts a response containing the gold answer after first checking normalized equality, then searching for the gold string on word boundaries.Word boundaries avoid treating “male” as a match inside “female.”Source: D Details of Open-Ended Post-Training · p. 12
- The modified reward mean settles between 0.3 and 0.5 during training, versus about 0.5 for the multiple-choice reward on the same pool.The audio and schedule remain the same, including two hundred training steps.Source: D Details of Open-Ended Post-Training · p. 12