PaperlayerView exact revision on arXiv

Source-linked AI summary

OP-CAD: On-Policy Clean-Audio Distillation for Robust Audio-Visual Reasoning

Xingming Shui, Dapeng Chen, Bowei Liu, Jingqi Tian, Minfu Li, Kun Yi, Jiapeng Hong, Yansong Tang

arXiv:2609.39150v1cs.CVcs.SD

TL;DR

OP-CAD addresses how audio-visual question answering models can resist environmental noise and competing speech when uniform on-policy distillation does not prioritize acoustically sensitive positions. It uses clean-teacher supervision with sensitivity-based token weighting and a mild-to-severe noise curriculum, outperforming uniform OPSD on noisy inputs across two benchmarks.

  • Problem

    Uniform on-policy distillation gives equal supervision across response positions, rather than prioritizing predictions affected by acoustic interference.

    Source: §1 · p. 2

  • Method

    OP-CAD trains a student on corrupted audio-visual inputs with a clean-input teacher, using audio-removal and corruption sensitivities for bounded token weights and a mild-to-severe noise curriculum.

    Source: §1 · pp. 2–3

  • Results

    OP-CAD raises environmental-noise and competing-speech averages by 3.48 and 2.92 points on Daily-Omni, and 2.83 and 3.80 on OmniVideoBench, versus OPSD.

    Source: §4.2 · p. 7

  • Takeaways & Limitations

    OP-CAD improves noisy-input accuracy over uniform OPSD under matched training conditions, with no observed decrease in aggregate clean accuracy.

    Source: §5 · p. 10

    Under OmniVideoBench environmental noise, Base retains 71.12% accuracy on clean-correct questions versus 66.04% for OP-CAD.

    Source: §4.4 · p. 9

Abstract

from arXiv · show

Omni-modal large language models deployed in real-world environments encounter external noise that can interfere with their perception and understanding of multimodal inputs. We study their robustness in audio-visual understanding, focusing on question answering under environmental noise and competing speech. The challenge is to resist acoustic interference while preserving useful audio evidence. On-policy distillation provides dense teacher feedback on student-generated responses, but uniform token weighting does not explicitly prioritize positions affected by acoustic interference. We introduce OP-CAD (On-Policy Clean-Audio Distillation), a curriculum-based privileged self-distillation framework for robust audio-visual understanding. Training progresses from mild to severe environmental noise and competing speech, with selective token-level supervision at each stage. The student generates responses from corrupted audio-visual input, while a frozen teacher uses clean audio and the verified answer to supervise the same response prefixes. To allocate this supervision, OP-CAD compares teacher predictions under clean, corrupted, and visual-only contexts without revealing the answer. These matched comparisons measure sensitivity to audio removal and corruption; a bounded weighting rule emphasizes positions identified by either signal while retaining supervision throughout the response. OP-CAD outperforms the compared methods across all evaluated noise conditions. Paired analyses further show improved preservation of clean-correct answers under strong interference, with no observed aggregate clean-accuracy penalty. These results demonstrate the value of directing clean-teacher supervision toward acoustically sensitive predictions for robust audio-visual reasoning.

1 Introduction

The paper studies audio-visual question answering under environmental noise and competing speech, where acoustic interference can mislead models even though the video remains unchanged. OP-CAD uses clean privileged teacher guidance and selectively increases supervision at acoustically sensitive response positions; it outperforms compared methods under evaluated interference conditions.

1 Introduction

  • Acoustic interference can turn a Base model’s correct answer into an incorrect one despite unchanged video and question, while OP-CAD answers correctly from corrupted audio-visual input.The motivating task is audio-visual question answering under environmental noise or background speech.Source: §1 · p. 1
  • Prior privileged and on-policy distillation work guides audio-conditioned understanding, while this study retains shared visual context and uses clean audio-visual input to guide learning under interference.The related approaches named are CORD and EchoDistill.Source: §1 · p. 1
  • OP-CAD compares clean input with corrupted and visual-only inputs to identify acoustically sensitive positions and increase clean-teacher supervision when either signal is strong.Baseline supervision remains throughout the response.Source: §1 · pp. 2–3
  • OP-CAD outperforms compared baseline training methods under environmental noise and competing speech, with no observed decrease in aggregate clean accuracy.Paired analyses also find better preservation of clean-correct answers under strong interference than OPSD.Source: §1 · p. 2
  • Training progresses from mild to severe interference while selective token-level supervision directs stronger guidance within each response.The framework uses a frozen teacher with clean audio-visual input and the verified answer to supervise prefixes generated by the student from corrupted input.Source: §1 · pp. 2–3

2 Related Work

The paper situates OP-CAD among work on audio-visual robustness, privileged information, and on-policy distillation. Its distinction is to use clean audio-visual input to measure prediction changes from audio removal and corruption and allocate token-level supervision accordingly.

Audio-visual understanding and acoustic robustness

  • Prior robustness research spans evaluation, input processing, and modality utilization, while this paper studies privileged training supervision for audio-visual QA with corrupted audio and unchanged video.The related work includes benchmarks and methods addressing cross-modal conflict, acoustic degradation, modality reliability, and noise-robust understanding.Source: §2 · p. 3

Privileged information and on-policy distillation

  • On-policy distillation supervises student-generated prefixes, and OPSD extends it with privileged teacher information; OP-CAD uses clean audio-visual input as a reference and studies prediction-change-based token allocation.The passage describes prior privileged information as access to additional information unavailable to students during training.Source: §2 · p. 3

Audio-conditioned distillation and supervision allocation

  • CORD and EchoDistill address audio-conditioned tasks, whereas OP-CAD retains shared visual context and measures teacher prediction changes caused by audio removal and corruption.The paper compares OP-CAD with SFT, GRPO, and uniform OPSD using the same backbone, data, and training protocol.Source: §2 · pp. 3–4

3 Method

OP-CAD trains a student on corrupted audio-visual inputs using clean privileged teacher targets and token weights based on teacher sensitivity to audio removal and corruption. The method pairs data and interference conditions, warms up the models, follows a noise curriculum, and allocates supervision across student-generated response prefixes.

3 Method

  • OP-CAD combines paired clean and corrupted inputs, supervised warm-up, progressively stronger interference, and teacher supervision weighted by prediction changes from audio removal and corruption.The student responds to corrupted input while clean audio-visual input supplies privileged teacher guidance.Source: §3 · p. 4

3.1 Paired Audio-Visual Data Construction

  • Training examples alter only audio, using environmental noise or competing speech at 10, 5, and 0 dB, with disjoint interference recordings and room impulse responses across training and evaluation.The sources include MUSAN, DNS Challenge, NOISEX-92, and VoxCeleb2.Source: §3.1 · p. 4
  • Paired base-model predictions select clean-correct, corrupted-incorrect examples for robustness training, examples incorrect in both conditions for warm-up, and exclude examples correct under corruption.The warm-up set teaches structured evidence and answer outputs.Source: §3.1 · pp. 4–5

3.2 Supervised Warm-Up and Noise Curriculum

  • The warm-up student initializes both student and frozen teacher, then robustness training proceeds through 10→5→0 dB stages, with the student generating new responses under its current policy.Each SNR stage contains both interference types and is trained for one epoch; curriculum orders difficulty across examples while token weighting controls guidance within responses.Source: §3.2 · p. 5

3.3 On-Policy Clean-Teacher Supervision

  • The student generates responses from corrupted input, while a frozen teacher uses clean audio-visual input plus the verified answer for targets and scores the same student-generated prefixes under matched input conditions.The verified answer is excluded from the weight computation, whose inputs share the video, question, and options.Source: §3.3 · p. 5

3.4 Supervision Allocation through Audio Condition Comparisons

  • OP-CAD contrasts uniform OPSD weights with weights based on teacher prediction sensitivity to audio removal and corruption, using clean audio-visual input as the common reference.Audio-removal sensitivity captures prediction differences associated with audio beyond visual context; corruption sensitivity captures changes caused by acoustic interference.Source: §3.4 · pp. 5–6, Figure 3 · p. 6
  • The method normalizes each sensitivity signal within a response, takes their maximum, and adds a bounded increment to baseline weight 1, preserving supervision at every generated position.Weights lie in [1, 1 + λ]; the objective normalizes the weighted distillation losses by total weight, and λ = 0 recovers uniform OPSD.Source: §3.4 · p. 6

4 Experiments

The experiments compare robustness strategies on two audio-visual benchmarks under clean audio, environmental noise, and competing speech. OP-CAD leads aggregate accuracy across these conditions, while analyses examine its allocation signals, curriculum, prediction stability, and token-level supervision.

4.1 Experimental Design

  • The evaluation covers 1,197 Daily-Omni questions and 1,000 OmniVideoBench questions across clean audio and two interference types at 10, 5, and 0 dB.The continued-training strategies share warm-up initialization, data, and noise severity curriculum; OPSD is the direct control for token-weight allocation.Source: §4.1 · pp. 6–7

4.2 Robustness Across Acoustic Conditions

  • OP-CAD achieves the highest aggregate accuracy among listed methods on clean audio and both interference families, with Table 1 reporting full-test accuracy.The comparison includes Base, supervised warm-up, and continued-training strategies under shared conditions.Source: §4.2 · p. 7, Table 1 · p. 7
  • OP-CAD improves environmental-noise and competing-speech averages over OPSD on both benchmarks and exceeds OPSD at every tested SNR.The average gains are 3.48 and 2.92 points on Daily-Omni, and 2.83 and 3.80 points on OmniVideoBench, respectively.Source: §4.2 · p. 7, Table 1 · p. 7
  • Across question types, audio types, and video lengths, OP-CAD exceeds OPSD in every corrupted-condition category and duration column.Tables 2 and 3 break accuracy down by question type and duration on Daily-Omni and by audio type and duration on OmniVideoBench.Source: Table 2 · p. 7, §4.2 · p. 8, Table 3 · p. 8

4.3 Component Analysis

  • Both audio-removal and corruption sensitivity alone improve all four corrupted-condition averages over uniform OPSD; their combination performs best in three aggregates.On OmniVideoBench competing speech, corruption sensitivity alone scores 39.00% versus 38.93% for OP-CAD.Source: §4.3 · p. 8, Table 4 · p. 8
  • Under matched training budgets, the 10, 5, then 0 dB curriculum improves accuracy over randomly mixed SNRs for both interference types on both benchmarks.Clean accuracy is higher with the curriculum on Daily-Omni but lower on OmniVideoBench.Source: §4.3 · p. 8, Table 5 · p. 9

4.4 Prediction Stability under Interference

  • At 0 dB, OP-CAD has lower harm rates than OPSD in all four conditions and the lowest rate among compared methods in three.For OmniVideoBench competing speech, the harm rate falls from 39.86% to 33.65%; SFT is slightly lower at 32.05%. Retention on Base’s clean-correct questions is highest for OP-CAD in three conditions, but not under OmniVideoBench environmental noise.Source: §4.4 · p. 9, Figure 4 · p. 9

4.5 Training Example of Supervision Allocation

  • On a competing-speech rollout at 5 dB, audio-removal and corruption sensitivity peak at different response positions, and their combination preserves both sets of peaks.The combined weighting also preserves baseline supervision elsewhere in the response.Source: §4.5 · p. 10, Figure 5 · p. 10

5 Conclusion

OP-CAD directs clean-teacher supervision toward predictions sensitive to audio removal or corruption. On two benchmarks it improves noisy-input accuracy over uniform OPSD without an observed aggregate clean-accuracy decrease, while evidence remains limited to one backbone, one run per method, and controlled acoustic corruptions.

5 Conclusion

  • OP-CAD weights clean-teacher supervision according to sensitivity to audio removal and corruption.The student is used alone at inference, without a clean audio reference, verified answer, or additional teacher passes.Source: §5 · p. 10
  • On two audio-visual benchmarks, OP-CAD improves noisy-input accuracy over uniform OPSD with no observed decrease in aggregate clean accuracy.Single-signal ablations find both comparisons useful, but their combination is not best in every condition.Source: §5 · p. 10
  • The reported evidence is limited to one backbone, one run per method, and controlled acoustic corruptions.Source: §5 · p. 10

A Implementation Details

The appendix specifies full-vocabulary teacher and student distributions, response handling, and numerical loss reduction for OP-CAD.

A.1 Full-Vocabulary Distributions and Response Handling

  • Teacher and student distributions, the reverse-KL target loss, and both allocation divergences use the complete vocabulary V.Teacher logits are detached, with gradients flowing only through the student.Source: §A.1 · p. 15
  • The student samples one current-policy response of at most 128 new tokens, and all teacher contexts score that same trajectory.Every generated position contributes to the objective, including incomplete or truncated responses, which are not regenerated.Source: §A.1 · p. 15

A.2 Numerical and Reduction Details

  • Each allocation score is normalized by its response maximum when that maximum exceeds 10^-12, and otherwise set to zero.The loss sums weighted token losses across the distributed mini-batch and divides by total weight; the configuration uses one microbatch per optimizer update.Source: §A.2 · p. 15
  • Sampled prefixes, teacher distributions, and weights are detached, so gradients flow only through the student distribution.Source: §A.2 · p. 15

B.1 Dataset Construction and Supervision

The appendix describes the training and evaluation pools, paired acoustic corruption, and the evidence used for supervised warm-up.

B.1 Dataset Construction and Supervision

  • WorldSense, OmniBench, and the AVQA subset of OmniInstruct-v1 supply training data, while Daily-Omni and OmniVideoBench are evaluation-only.The robustness partition contains 4,660 corrupted acoustic views from 1,809 source questions.Source: §B.1 · p. 15, Table 6 · p. 15
  • Applying the warm-up routing criterion to the screened pool yields 1,790 views from 615 questions across the same three SNRs.A question can contribute multiple corrupted views, so view and question counts are reported separately.Source: §B.1 · p. 15, Table 7 · p. 16
  • Warm-up evidence is generated by Gemini 3 Flash, paired with verified answers, and reviewed by humans for support from audio-visual evidence.The evidence generator is separate from the frozen Qwen teacher used for distillation.Source: §B.1 · p. 15
  • Corruptions combine measured room impulse responses and an 8-kHz µ-law channel with environmental noise or competing speech at 10, 5, and 0 dB.Training and evaluation use disjoint interference recordings and room impulse responses, with disjoint speakers for competing speech.Source: §B.1 · p. 16

C Complete Robustness Results

OP-CAD exceeds OPSD across all 12 benchmark–interference–SNR conditions. The reported tables cover benchmark partitions and paired acoustic conditions.

C Complete Robustness Results

  • OP-CAD exceeds OPSD in all 12 benchmark–interference–SNR conditions.Tables 8 and 9 report Base performance by partition and the compared methods at each SNR.Source: C Complete Robustness Results · p. 16
  • Table 8 reports Base accuracy by benchmark category and duration under paired acoustic conditions.Source: Table 8 · p. 16

C.1 Robustness across Categories and Durations

The appendix details robustness results across benchmark partitions and noise severities, paired clean-correct retention and correctness transitions, and a limited teacher-context diagnostic.

C.1 Robustness across Categories and Durations

  • Table 9 reports full-test accuracy at each noise severity; corrupted-condition accuracy averages the 10, 5, and 0 dB views within each interference family.Table 10 gives category and duration partition sizes; five OmniVideoBench records slightly longer than 30 minutes remain in the final published duration bin.Source: §C.1 · p. 16, Table 9 · p. 17, Table 10 · p. 17

C.2 Preserving Clean-Correct Answers

  • Clean-correct retention uses a fixed set of questions Base answered correctly under clean audio, so it should be read alongside complete-test accuracy.A nonzero count of questions becoming correct under corruption does not alone imply a benefit; it must be interpreted with the count becoming wrong and the net change.Source: §C.2 · p. 17
  • OP-CAD exceeds OPSD in every displayed clean-correct retention column on both benchmarks.Tables 11 and 12 report retention at each SNR and correctness transitions at 0 dB.Source: §C.2 · p. 17, Table 11 · p. 18, Table 12 · p. 18
  • At 0 dB, OP-CAD reduces net clean-to-noisy accuracy loss versus OPSD across both benchmarks and interference families.Losses fall from 8.35 to 5.68 and 13.20 to 10.28 points on Daily-Omni, and from 2.80 to 0.40 and 7.10 to 3.40 points on OmniVideoBench, for environmental noise and competing speech, respectively.Source: §C.2 · p. 17

C.3 Context Sensitivity

  • The teacher-context diagnostic compares fixed-response prediction disagreement across contexts, but its small sample does not establish how frequently these effects occur in training.It scores 639 saved tokens from eight diagnostic trajectories under seven contexts.Source: §C.3 · p. 18, Table 13 · p. 19

C.4 Training Efficiency

OP-CAD’s measured robustness-training cost falls between OPSD and an eight-rollout GRPO configuration, although the runs are not compute-matched.

C.4 Training Efficiency

  • OP-CAD uses 36.5% more GPU-hours than OPSD and about 65% fewer than the eight-rollout GRPO configuration.These measured run costs exclude shared SFT warm-up, offline evidence generation, and evaluation; they are not comparisons at equal compute.Source: §C.4 · p. 19, Table 14 · p. 19

D Reproducibility Details

The reproducibility details describe the source package and computing setup, prompt and scoring procedures, offline evidence generation, and training configuration and checkpoint selection.

D.1 Artifact Availability and Computing Environment

  • An anonymous source package provides training and evaluation code and configuration templates; fine-tuning used 32 NVIDIA L20X GPUs across four Linux nodes.The acoustic corruption procedure and parameters are described in Appendix B.Source: §D.1 · p. 19, Table 15 · p. 20

D.2 Prompt and Scoring Protocol

  • Students answer video questions with brief evidence and one option letter; the privileged teacher receives the verified answer, while allocation contexts omit that private block.Evaluation uses deterministic generation, designated answer-field extraction, and counts unparseable outputs as incorrect without regeneration.Source: §D.2 · p. 20

D.3 Offline Evidence Generation

  • Gemini 3 Flash generates offline evidence from clean audio, video, the question, options, and verified answer; exported evidence is checked and human-reviewed.The offline generator is distinct from the frozen Qwen teacher used during distillation.Source: §D.3 · p. 21

D.4 Configuration and Checkpoint Selection

  • Final curriculum checkpoints are evaluated without using either benchmark for hyperparameter tuning or checkpoint selection.Training uses one warm-up epoch followed by one epoch at each disjoint 10, 5, and 0 dB stage; OP-CAD uses τ = 1, λ = 0.5, and weights in [1, 1.5].Source: §D.4 · p. 21

Related papers

Loading 2609.39150v1…