Source-linked AI summary
Reward-Tilted On-Policy Distillation for Acoustic Grounding in Audio-Language Models
Kaiyang Li, Shaobo Han, Yue Tian, Shihao Ji
TL;DR
Audio-language models can use textual shortcuts instead of acoustic evidence, and standard on-policy distillation does not explicitly distinguish acoustic support from linguistic predictability. RT-OPD reshapes teacher targets using audio-present/absent probability contrasts. Across two students and three benchmarks, it outperforms Vanilla OPD, and audio perturbations suggest stronger acoustic reliance.
Problem
Textual shortcuts can let audio-language models answer questions without relying on recordings, and standard OPD does not explicitly emphasize the recording's additional evidence.
Source: §1 · p. 1Method
Results
Across two students and three benchmarks, RT-OPD consistently outperforms Vanilla OPD; Mizar-3B reaches 72.72% accuracy on MMAU, the highest among compared 3B models.
Source: §1 · p. 2Takeaways & Limitations
Audio perturbation analysis suggests RT-OPD relies more strongly on acoustic evidence than Vanilla OPD.
Source: §4.4 · p. 4, §5 · p. 4
Abstract
from arXiv · showhide
Audio-language models (ALMs) can exploit textual shortcuts to answer questions while overlooking acoustic evidence, weakening audio understanding. On-policy distillation (OPD) trains compact ALMs by supervising student-generated responses with teacher predictions, but does not explicitly distinguish acoustic support from linguistic predictability. We propose Reward-Tilted On-Policy Distillation (RT-OPD) to strengthen acoustic grounding. Given the same question and student-generated text, a frozen teacher predicts the next token with and without audio inputs. Their log-probability contrast defines a reward that reshapes the teacher distribution for reverse-KL distillation, emphasizing the additional evidence provided by audio. Across two compact students and three benchmarks, RT-OPD consistently outperforms Vanilla OPD. Experiments with silenced and replacement audio further suggest that RT-OPD strengthens the student's reliance on acoustic evidence. Our 3B model achieves 72.72% accuracy on MMAU, the highest among the compared 3B models and competitive with several 7B and 8B models. Code and model checkpoints are available at https://github.com/KaiyangLi1992/RT-OPD.
1. INTRODUCTION
RT-OPD uses audio-present/absent teacher predictions to emphasize acoustic evidence in distillation targets. It outperforms Vanilla OPD across two students and three benchmarks, with Mizar-3B leading the compared 3B models on MMAU.
1. INTRODUCTION
- OPD transfers larger-model reasoning to compact ALMs by supervising student-generated responses, reducing exposure bias through teacher guidance on student-generated prefixes.Compact ALMs are needed for deployment under limited memory and computation while retaining reasoning capabilities.Source: §1 · p. 1
- Textual shortcuts can let ALMs answer audio questions without using recordings, while standard OPD does not explicitly emphasize the recording's additional evidence.A linguistically predictable answer such as “a dog is barking” does not establish that the model identified barking in the recording.Source: §1 · p. 1
- RT-OPD rewards tokens whose teacher probability rises with audio, reweights and normalizes the teacher distribution, then trains the student by reverse KL.The teacher compares next-token predictions with and without audio for the same question and student-generated text.Source: §1 · pp. 1–2, Figure 1 · p. 2
- 72.72% accuracy on MMAU makes Mizar-3B the highest-scoring compared 3B model, while remaining competitive with several 7B and 8B models.Table 1 reports ALM performance on MMAU, ranked by accuracy.Source: Table 1 · p. 1, §1 · p. 2
2. RELATED WORK
Prior work improves audio-language reasoning through reinforcement learning and distillation, while other methods contrast predictions across audio conditions. RT-OPD applies that contrast to student-generated trajectories and reshapes token-level teacher targets.
2. RELATED WORK
- Reasoning-focused work includes GRPO-based approaches, weighted reverse-KL alignment with sequence-level GRPO, and distillation along student-generated trajectories.The cited examples are Ke-Omni-R and CESAR, CORD, and X3-OPD, respectively.Source: §2 · p. 2
- Audio-aware decoding contrasts predictions with and without audio at inference, while CAAD brings that contrast into distillation using responses generated in advance by the teacher.RT-OPD instead applies the contrast along student-generated trajectories and reshapes the teacher's next-token distribution before reverse-KL distillation.Source: §2 · p. 2
- Visual-Advantage OPD uses modality contrast to weight trajectories or token groups, whereas G-OPD contrasts a teacher with a separate reference model.RT-OPD compares the same frozen teacher with and without audio, directly measuring how audio changes support for each candidate token.Source: §2 · p. 2
3. METHOD
RT-OPD measures how audio changes the teacher’s next-token support and uses that contrast to reshape distillation targets along student-generated responses. The training objective combines this distillation with gold-answer cross-entropy and gates distillation on teacher correctness.
3.1. Standard OPD and Setup
- Standard OPD matches teacher next-token distributions to the student’s generated response, using contexts formed from audio, the question, and the generated prefix.Source: §3.1 · p. 2
3.2. Acoustically Grounded Teacher-Target Shaping
- Audio-present/absent teacher log-probability contrasts define token rewards that reweight and normalize the teacher distribution, amplifying candidates whose support increases with audio.Setting α = 0 recovers standard OPD; α > 0 strengthens amplification for candidates with larger audio-present/absent probability ratios.Source: §3.2 · pp. 2–3
3.3. Student Training Objective
- The student combines gold-answer cross-entropy with distillation only when the teacher answers correctly with audio; excluded examples retain cross-entropy.The implementation uses λ = 0.25, α = 1, and rank-64 LoRA; the audio-absent teacher is used only during training.Source: §3.3 · p. 3
4. EXPERIMENTS
RT-OPD is evaluated against existing training methods on three benchmarks, with target and rollout ablations and audio perturbations probing its results and acoustic grounding.
4.1. Experimental Setup
- Training uses 10,000 AudioMCQ examples and a frozen Ke-Omni-R-7B teacher; evaluation averages accuracy across disjoint-audio MMAU, MMAR, and ADQA-cl.Trained methods are reported as means over five seeds; Macro-3 is the arithmetic mean of accuracy across the three benchmarks.Source: §4.1 · p. 3
4.2. Main Results
- RT-OPD outperforms Vanilla OPD and CAAD on all three benchmarks for both students.Table 2 reports main results, with five-seed means for trained methods.Source: §4.2 · p. 3, Table 2 · p. 4
- 63.08 Macro-3 for Mizar-3B is close to the 7B teacher’s 63.47, and Mizar-3B exceeds the teacher on ADQA-clean.Mizar-3B also has the highest MMAU accuracy among compared 3B models and is competitive with several 7B and 8B models.Source: §4.2 · p. 3
4.3. Ablation Study
- RT-OPD beats unmatched, sharpening, and model-reference targets on Macro-3, and improves over off-policy by 0.84 points for Ke and 1.02 for Qwen.The ablations compare audio-reference choices, teacher-distribution sharpening, model-reference contrast, and pre-generated teacher responses.Source: §4.3 · p. 4, Table 3 · p. 4
4.4. Audio Grounding Analysis
- On Ke-based models, silence and replacement audio generally cause larger performance drops for RT-OPD than Vanilla OPD, suggesting stronger acoustic reliance.Table 4 reports five-seed mean accuracy and percentage-point changes from original audio.Source: §4.4 · p. 4, Table 4 · p. 4
5. CONCLUSION
The paper concludes that RT-OPD uses audio-present/absent teacher contrasts to reshape distillation targets, improving benchmark performance and strengthening apparent reliance on acoustic evidence.
5. CONCLUSION
- RT-OPD reshapes the teacher distribution using audio-present/absent log-probability contrast during on-policy distillation.The paper presents this as a way to strengthen acoustic grounding.Source: §5 · p. 4
- Across two students and three benchmarks, RT-OPD consistently outperforms Vanilla OPD and the other trained methods in Macro-3.Source: §5 · p. 4
- Audio perturbation analysis suggests stronger acoustic reliance, while Mizar-3B has the highest MMAU accuracy among compared 3B models and is competitive with several 7B and 8B models.Source: §5 · p. 4