PaperlayerView exact revision on arXiv

Source-linked AI summary

When Vision Speaks for Sound

Xiaofei Wen, Wenjie Jacky Mo, Xingyu Fu, Rui Cai, Tinghui Zhu, Wendi Li, Yanan Xie, Muhao Chen, Peng Qi

arXiv:2605.16403v1cs.CVcs.SD

TL;DR

Video-capable multimodal models often infer sound from visual cues instead of verifying whether the audio is synchronized and consistent. The paper introduces THUD’s Shift, Mute, and Swap interventions and a preference-based alignment recipe, achieving a 28% average gain across the three intervention dimensions.

  • Problem

    Current video-capable multimodal models may rely on visual-semantic shortcuts rather than genuinely grounding sound-related answers in temporally and semantically consistent audio.

  • Method

    THUD probes audio-visual grounding with Shift, Mute, and Swap counterfactual interventions, while preference alignment combines intervention-derived pairs with general video preferences.

  • Results

    28% average gain over vanilla Qwen3-Omni across Shift, Mute, and Swap was achieved, with slight improvements on general video and audio-visual QA benchmarks.

  • Takeaways & Limitations

    Intervention-based training can mitigate multiple audio-visual shortcut modes, although audio existence and cross-modal consistency still need targeted supervision.

  • Takeaways & Limitations

    The recipe was evaluated on a limited set of base models, and complete training studies for Mute and Swap have not yet been conducted.

Abstract

from arXiv · show

Despite rapid progress in video-capable MLLMs, we find that their apparent audio understanding in videos is often vision-driven: models rely on visual cues to infer or hallucinate acoustic information, rather than verifying the audio stream. This issue appears across both state-of-the-art open-source omni models and leading closed-source models from providers such as Google and OpenAI. We characterize this failure mode as an audio-visual Clever Hans effect, in which models appear (falsely) audio-grounded, but actually exploit visual-acoustic correlations without verifying whether the audio and visual streams are truly aligned. To systematically study this behavior, we introduce Thud, an intervention-driven probing framework based on three counterfactual audio edits: Shift, which tests temporal synchronization; Mute, which tests sound existence; and Swap, which tests audio-visual consistency. Beyond diagnosis, we further study a two-stage alignment recipe: intervention-derived preference pairs teach audio verification, while event-level general video preferences regularize the model against over-specialization. Our best 10K-sample recipe improves average performance across the three intervention dimensions by 28 percentage points, while slightly improving performance on general video and audio-visual QA benchmarks.

1 Introduction

Current video-capable MLLMs often infer sound from visual-semantic shortcuts rather than verified audio, producing nearly unchanged descriptions when audio changes. THUD probes this Clever Hans effect by counterfactually disrupting temporal synchronization, audio existence, and sound consistency.

  • Problem: Current video-capable MLLMs often rely on visual cues instead of verified audio when reasoning about acoustic information.This visual dominance can yield nearly unchanged descriptions even when the audio track changes substantially.
  • Problem: This behavior constitutes an audio-visual Clever Hans effect: apparent audio grounding arises from unintended visual-acoustic correlations.The models appear audio-grounded but produce sound-related outputs primarily from visual cues.
  • THUD: THUD is an intervention-driven diagnostic protocol that tests whether models perform genuinely grounded audio-visual reasoning.It neutralizes semantic shortcuts by counterfactually perturbing audio-visual correspondences in natural sounded videos.
  • THUD: THUD probes three dimensions of audio-visual grounding: temporal synchronization, audio existence, and sound consistency.Its interventions are named Shift, Mute, and Swap, respectively.
  • Contributions: Controlled interventions quantify how strongly Video-LLMs substitute genuine audio-visual grounding with visual-semantic shortcuts.Representative failures include missed temporal shifts, hallucinated sounds, and visually biased predictions.

2 How Can We Align Models Beyond Visual Shortcuts?

The section presents an intervention-driven alignment framework that trains models to verify audio against visible events rather than rely on visual shortcuts. It constructs Shift, Mute, and Swap counterfactuals, converts them into preference data, and combines them with general video supervision to limit over-specialization.

  • Intervention construction: The framework breaks natural audio-visual correlations with Shift, Mute, and Swap interventions targeting timing, sound presence, and physical consistency.Shift displaces audio temporally, Mute replaces it with silence, and Swap substitutes acoustically plausible audio from another video.
  • Annotation and verification: Event-time annotations are cross-verified across models and retained only when visual and acoustic timestamps agree within strict tolerances.Gemini provides initial annotations, while GPT, Claude, and human verification support cross-model checking and correction of disagreements.
  • Preference pair construction: Intervention cases become chosen–rejected preference pairs in which chosen responses verify audio evidence and rejected responses follow visually plausible but inconsistent interpretations.For Shift, Mute, and Swap, the chosen responses respectively detect temporal offsets, identify silence, and flag source inconsistency.
  • Two-stage alignment: The two-stage recipe uses SFT warm-up on intervention data followed by DPO on intervention preference pairs mixed with general video data.The general-video mixture regularizes against over-specialization while favoring audio-verified responses over visually plausible shortcuts.
  • Two-stage alignment: General video supervision is drawn from FineVideo and enriched with visual and audible event-level information organized by timestamp segments.Selected clips are re-annotated with Gemini and checked for human agreement before producing instruction data.

3 Experiments

The experiments diagnose whether video-capable multimodal models verify audio or exploit visual-acoustic shortcuts, then evaluate intervention-based alignment training. Models show fragile audio-visual grounding under counterfactual edits, while a targeted 10K DPO recipe improves synchronization and broader intervention performance without sacrificing general capabilities.

  • Experimental Setup: Evaluation uses Original, Shift, Mute, and Swap conditions with paired accuracy for audio existence, temporal synchronization, and sound consistency.Original videos are positive controls; counterfactual interventions test whether audio-visual correspondence is verified.
  • Shortcut Analysis: 80.7% and 78.4% are the largest Avg Gaps for MiniCPM-o-4.5 and MiMo-V2.5, showing that strong Original performance is fragile under intervention.Qwen3-Omni’s temporal-sync accuracy drops to 1.4% under Shift, consistent with a synchronized-default prior.
  • Shortcut Analysis: Above 0.63 across models, Mute Hallucination and Swap False-Match dominate, while False Silence and Swap False-Mismatch remain near zero.Models commonly invent visually plausible audio but rarely reject audio that is present; Qwen3-Omni misses 98% of ±2 s offsets.
  • Targeted Alignment: The best 10K mixture raises Sync from 34.3% to 83.1% and VGGSync from 36.8% to 56.4%, increasing the six-benchmark average from 51.3% to 63.3%.The recipe maintains or improves V-MME, LVB, and WS and remains competitive on DO.
  • Broader Intervention Results: 28% average gain over vanilla Qwen3-Omni results across Shift, Mute, and Swap after adding a small amount of Mute/Swap SFT to the best recipe.The resulting model ranks first on Swap and second on Mute.

4 Related Work

Related work spans native omni-modal models that unify video, audio, images, and text, as well as preference-based alignment methods adapted to video-capable multimodal models. However, unified multimodal processing does not guarantee that models verify the audio stream.

  • Native Omni Models and Cross-Modal Shortcuts: Native omni-modal models unify video, audio, images, and text through shared interfaces or architectures, moving beyond frame-centric video-language pipelines.This integration is associated with stronger audio-visual grounding but does not ensure audio-stream verification.
  • Preference Alignment for Video-Capable Multimodal Models: Video-capable multimodal modeling has developed through video-language instruction tuning and native omni-modal architectures.Preference-based methods such as Direct Preference Optimization have also been adapted to video-language modeling.

5 Conclusion

The paper shows that video-capable multimodal models’ apparent audio understanding can be strongly vision-driven, revealing an audio-visual Clever Hans effect. THUD makes this failure measurable through Shift, Mute, and Swap interventions.

  • 5 Conclusion: Audio-related answers can exploit natural visual-acoustic correlations instead of verifying the observed audio stream.This vision-driven behavior is identified as an audio-visual Clever Hans effect.
  • 5 Conclusion: The audio-visual Clever Hans effect describes models that appear to understand sound while relying on visual cues.The paper characterizes the behavior as strongly vision-driven.
  • 5 Conclusion: THUD measures this failure with Shift, Mute, and Swap interventions targeting temporal synchronization, sound existence, and audio-visual consistency.These interventions probe whether models verify the relationship between video and its audio stream.

A Schematic Overviews of Data Construction and Alignment · A.1 Data Construction Pipeline · A.2 Intervention Summary

The paper constructs THUD through rigorously verified event-time annotations and controlled audio interventions that preserve the visual stream while probing distinct audio-grounding dimensions. Its pipeline combines model-based timestamp validation, human acoustic inspection, strict filtering, and Shift, Mute, and Swap perturbations.

  • A.1 Data Construction Pipeline: The intervention-driven preference dataset begins with event-time labeling using Gemini.This initializes the systematic construction pipeline before crossverification and filtering.
  • A.1 Data Construction Pipeline: Visual timestamps are validated through GPT–Claude consensus using frame-unit analysis.The crossverification targets reliable visual temporal annotations.
  • A.1 Data Construction Pipeline: Acoustic timestamps undergo human inspection to ensure ground-truth reliability.Human review provides the reliability check for audio timing annotations.
  • A.1 Data Construction Pipeline: Samples are filtered using strict agreement criteria before interventions are applied.The filtering step follows the crossverification process in the dataset pipeline.
  • A.2 Intervention Summary: Each THUD intervention keeps the visual stream fixed while perturbing the audio track.This design isolates audio-grounding behavior under controlled counterfactual cases.
  • A.2 Intervention Summary: Shift probes temporal synchronization, Mute probes sound existence, and Swap probes source consistency.Together, the interventions target three distinct grounding dimensions.
  • A.2 Intervention Summary: These controlled cases test whether models verify observed audio or infer plausible sounds from visual priors.The intervention summary frames the benchmark as a test of audio verification versus visual inference.

A.3 Preference Data Sources · A.4 Alignment Pipeline · B Annotation and Verification Details

The alignment recipe combines intervention-focused and general-video preference sources in a two-stage pipeline, while annotation and verification enforce localized, audibly verifiable audio-visual events. Stage 1 uses intervention data for SFT warm-up, and Stage 2 mixes intervention preferences with general video data during DPO to encourage audio verification without sacrificing broad video understanding.

  • A.3 Preference Data Sources: Preference examples pair a desired chosen response with a shortcut-prone or incorrect rejected response.This common structure underlies the alignment recipe’s data sources.
  • A.3 Preference Data Sources: Original synchronization preferences perturb visual or acoustic events and timestamps to create plausible but incorrect synchronization explanations.The chosen response is derived from the original aligned event, while the rejected response changes one or more components.
  • A.3 Preference Data Sources: SFT-policy negatives use incorrect or shortcut-prone SFT outputs as rejected responses, encouraging correction of post-SFT failure modes.The reference annotation remains the chosen response for the same video-question input.
  • A.3 Preference Data Sources: Counterfactual temporal preferences reverse chosen and rejected answers across original and shifted videos to force discrimination of true temporal synchronization.The original synchronized answer is preferred for original videos, while the shifted-condition answer is preferred for shifted videos.
  • A.3 Preference Data Sources: FineVideo sources provide descriptive, audio-dependent, explanatory, and general-video preferences, with LV-MCQA regularizing against intervention-specific over-specialization.FV-AVQA retains examples where GPT-based text-only answering fails, and FV-AVQA-L requires explanations grounded in audio-visual evidence.
  • A.4 Alignment Pipeline: The two-stage pipeline uses intervention-data SFT warm-up followed by DPO on mixed intervention and general-video preference data to detect Shift, Mute, and Swap failures while preserving general understanding.The targeted intervention dataset is combined with re-annotated general video instructions derived from FineVideo.
  • B Annotation and Verification Details: Temporally ordered frame units let GPT and Claude localize candidate visual events, while retained samples require visual agreement, human audio verification, clear events, salient sounds, and unambiguous interventions.Each unit includes representative frames and a timestamp range; samples failing automatic agreement receive manual review, and unclear events or sounds are discarded.

C Experimental Configuration · D Preference Pair Examples · E FineVideo-derived general instruction data

The appendix specifies the Qwen3-Omni-30B-A3B-Instruct setup and training infrastructure, illustrates Shift, Mute, and Swap preference-pair construction, and lists FineVideo-derived general instruction tasks. These materials operationalize audio verification through synchronization, sound-existence, source-consistency, and audio-dependent questions.

  • C Experimental Configuration: Evaluation across six datasets takes approximately 5 hours on average.The reported average inference time covers the evaluation datasets collectively.
  • C Experimental Configuration: The experiments use Qwen3-Omni-30B-A3B-Instruct with video audio enabled via use_audio_in_video=true.This is the base omni-modal model configuration.
  • C Experimental Configuration: Training runs on a single 8-GPU node with DeepSpeed ZeRO-3, while experiments use 8 NVIDIA H200 GPUs and evaluation uses 8 NVIDIA H200 or H100 GPUs.A single SFT run takes approximately 6 hours, and DPO training on 10K examples takes approximately 20 hours.
  • D Preference Pair Examples: Shift pairs contrast a fall at ∼5.0s with impact sound at ∼3.1s against a response claiming synchronization.The chosen response identifies the temporal mismatch.
  • D Preference Pair Examples: Mute pairs treat a silent audio track as chosen and reject richly detailed but unsupported descriptions of music, speech, ambient noise, and sound effects.The example explicitly states that no audio cues are detected throughout the clip.
  • D Preference Pair Examples: Swap pairs favor recognizing that optics-diffraction visuals paired with centrifuge audio indicate an audio-source mismatch over a visually consistent narration.The rejected response explains the diffraction setup rather than verifying the audio source.
  • E FineVideo-derived general instruction data: FineVideo-derived general instruction data covers describing visible and audible cues, cue-source attribution, temporal localization, and audio-dependent question answering.These task types extend beyond the three intervention-specific preference dimensions.

F Qualitative GPT-5.5 Outputs (Visual-Only Input) … C. Question Avoidance.

The appendix shows GPT-5.5 hallucinating and accepting audio from visual-only input, while defining prompts, judge extraction, failure metrics, and question-avoidance criteria for systematic evaluation. It also specifies how temporal, mute, and swap errors are operationalized.

  • G Evaluation Prompts: G Evaluation Prompts: MUTE and SWAP use neutral open-ended description prompts, whereas SHIFT uses a structured synchronization prompt requesting offset direction and seconds.Free-form responses are converted into structured predictions by GPT-based judges.
  • G Evaluation Prompts: G Evaluation Prompts: GPT judges classify MUTE and SWAP free-form responses into discrete categories and return a single JSON object.The MUTE judge distinguishes muted from hallucinated audio, while the SWAP judge distinguishes mismatched from apparently synchronized audio.
  • H Failure-mode definitions: H Failure-mode definitions: Mute Hallucination is the audio_described rate on silenced videos, while Swap False-Match measures treating unrelated replacement audio as a plausible visual match.Both metrics use independent GPT-5.4 judging of neutral-prompt responses.
  • H Failure-mode definitions: H Failure-mode definitions: False Silence measures claiming no audio when original audio exists, and Swap False-Mismatch measures spuriously claiming mismatch on naturally synchronized pairs.These are symmetric audio-denial errors on non-intervened videos.
  • C. Question Avoidance.: C. Question Avoidance.: Audio Dodge is the mean rate of responses that describe only visuals and neither describe sound nor claim silence across silenced and real-audio conditions.The averaging treats non-engagement as a model property rather than a consequence of audio presence.
  • C. Question Avoidance.: C. Question Avoidance.: The sync benchmark includes synced originals and approximately ±2 s DELAY and EARLY variants, with judge outputs for synchronization and offset direction.The parsed fields are pred_synced and pred_direction ∈{DELAY, EARLY, NONE}.
  • C. Question Avoidance.: C. Question Avoidance.: Offset Blindness, Direction Confusion, and False Sync Alarm isolate missed desynchronization, wrong offset direction, and false desynchronization claims, respectively.For Gemini-3.1-pro, these quantities are derived from aggregate saved metrics because per-row sync predictions were unavailable.

I Limitations. … J.2 Broader Impacts

The paper reports limited validation of its training recipe across model families and intervention types, while outlining ethical safeguards, potential benefits, and risks of intervention-based audio-visual diagnostics.

  • I Limitations.: The training recipe was evaluated on a limited set of base models, leaving its effectiveness across broader omni-modal model families for future study.Experiments primarily validate DPO after SFT for temporal synchronization, not the complete Mute and Swap settings.
  • I Limitations.: The recipe’s effects on audio existence and cross-modal consistency remain incompletely studied because complete training studies for Mute and Swap have not been conducted.The current recipe experiments primarily validate DPO after SFT for improving temporal synchronization.
  • J.1 Ethics.: The study follows the NeurIPS Code of Ethics and involves no human-subject experiments, crowdsourcing, or personally identifiable information collection.Its video data comes from public or properly licensed sources and is used for evaluation and training under controlled audio-visual interventions.
  • J.2 Broader Impacts: The work aims to improve model reliability by revealing visual-semantic shortcuts and enabling diagnosis of pseudo-alignment through controlled Mute, Swap, and Shift interventions.These interventions assess whether audio is present, synchronized, and consistent with the visual scene.
  • J.2 Broader Impacts: Intervention-based diagnostics could be misused to construct adversarial examples or optimize models for benchmark performance instead of robust real-world grounding.This is identified as the main potential risk of the diagnostic approach.
  • J.2 Broader Impacts: Improved audio-visual verification does not eliminate hallucination risks, and deployed systems may still fail under out-of-distribution sounds, noisy environments, edited videos, or subtle cross-modal inconsistencies.The passage identifies these deployment conditions as residual failure modes requiring mitigation.

K New Assets

The paper introduces intervention-based evaluation assets that test whether multimodal models verify audio rather than relying on visual-semantic shortcuts. The assets modify only audio across Mute, Swap, and Shift conditions, with team verification and stated coverage limitations.

  • Intervention assets: The assets probe audio-visual grounding through three controlled interventions: Mute, Swap, and Shift.They are designed to test audio-stream verification rather than visual-semantic shortcuts.
  • Intervention assets: Mute removes audio, Swap inserts mismatched audio, and Shift temporally displaces audio to test existence, cross-modal consistency, and synchronization, respectively.Each variant includes its intervention type, evaluation condition, and target label.
  • Validation and limitations: Research-team members verify intervention validity and label clarity, rejecting examples with unclear events, defective audio, failed edits, or ambiguous labels.The assets support model evaluation and alignment research, not guarantees of real-world robustness.
  • Validation and limitations: The assets have restricted intervention coverage, possible residual annotation noise, and limited coverage of real-world audio-visual failure modes.These limitations constrain the scope of the diagnostic assets.
Loading 2605.16403v1…