PaperlayerView exact revision on arXiv

Source-linked AI summary

Seeing What Should Be Heard: Diagnosing and Repairing Cross-Modal Shortcuts in Omni-Modal LLMs

Yueran Ma, Ronghao Lin

arXiv:2609.36798v1cs.CVcs.CLcs.LG

TL;DR

Omni-modal models may answer audio questions from images because training rarely checks which modality supports the answer. The paper diagnoses this reliance with crossed image–audio samples and trains DMC-Repair using labels from the question’s designated modality. DMC-Repair reduces the Shortcut Index by 59.9% on a sealed confirmation set, with reported generalization and persistence through subsequent RL.

  • Problem

    Training paradigms rarely verify whether omni-modal models follow the modality specified by a question, since same-sample image and audio often support the same answer.

    Source: §1 · pp. 1–2

  • Method

    The Factorized Modality Diagnostic swaps images and audio between samples to isolate their answer effects, while DMC-Repair trains on the crossed samples labeled by the question’s designated modality.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    DMC-Repair reduces the Shortcut Index by 59.9% on a sealed confirmation set and generalizes across two model families and zero-shot to an unseen dataset and benchmark.

    Source: §5.2 · p. 7, §6 · p. 9

  • Takeaways & Limitations

    The shortcut reduction persists through subsequent RL post-training, including post-training that improves task accuracy.

    Source: §6 · p. 9

Abstract

from arXiv · show

Omni-modal large language models (LLMs) are expected to answer a question using the modality it explicitly refers to. However, existing training paradigms rarely verify whether models actually follow this modality, because multimodal inputs from the same sample often provide redundant evidence for the same answer. In this work, we uncover a pervasive cross-modal shortcut in omni-modal LLMs: when asked an audio-related question, models rely on the image as much as on the audio, and sometimes even more. To systematically diagnose this behavior, we introduce the Factorized Modality Diagnostic, which independently swaps audio and images between samples to isolate each modality's causal contribution. Across two model families in different settings, we find that this shortcut persists throughout supervised fine-tuning and reinforcement learning post-training, while judge-based RL may further amplify such reliance on irrelevant visual information. Based on this finding, we propose DMC-Repair, which trains models on the same kind of cross-modal swapped samples while assigning supervision according to the modality specified by the question. This prevents models from exploiting the spurious correspondence between modalities within the same clip. Experiments demonstrate that DMC-Repair reduces the image-induced share of the answer effect by 59.9%, effectively suppressing the cross-modal shortcut without compromising audio-question answering performance. The reduction in shortcut reliance generalizes across two model families and zero-shot to an unseen dataset and an unseen benchmark, and persists through subsequent post-training. Code is available at https://anonymous.4open.science/r/DMC-Repair.

1 INTRODUCTION

Omni-modal models can answer audio questions from the image because training rewards correctness without checking which modality supports the answer. The paper diagnoses this shortcut with crossed image–audio samples and proposes DMC-Repair, which labels each sample according to the question’s designated modality.

1 INTRODUCTION

  • Existing decoding, preference, and RL approaches require extra inference passes, preference pairs, or rewards, yet rarely verify which modality drives an answer.Correct-answer rewards can be fully satisfied when a model answers from the image rather than the audio.Source: §1 · pp. 1–2
  • Across two model families, the image affects audio answers about as much as the audio, and the shortcut persists through SFT and RL post-training.Judge-based RL rewards designed to encourage modality use further increase the image effect; on conflict cells, answers often follow the image.Source: §1 · p. 2
  • DMC-Repair trains on counterfactual image–audio combinations, labels each cell by its designated modality, and supervises only answer tokens.This removes the spurious image–audio correspondence as a basis for inferring the answer and requires no judge, reward model, or preference pairs.Source: §1 · p. 2
  • 59.9% lower Shortcut Index and 13.7-point higher audio-following on held-out test questions after DMC-Repair.The Shortcut Index measures the image’s effect on audio answers; audio-following measures how often answers on conflict cells follow the audio.Source: §1 · p. 2
  • The repair generalizes across two model families and zero-shot to an unseen dataset and benchmark, and persists through subsequent RL post-training.The diagnostic measures each modality’s effect by comparing answer margins across crossed image–audio combinations.Source: §1 · p. 2, Figure 2 · p. 3

2 RELATED WORK

Related work addresses modality dependence through inference-time interventions, preference optimization, and modality-aware RL, but these approaches rarely directly measure which modality drives an answer. DMC-Repair uses a complete crossed grid for both diagnostic measurement and training.

2 RELATED WORK

  • RL post-training rewards judge context or frame order, but do not verify whether answers follow the designated modality (Yang et al., 2025; Feng et al., 2025; Zhao et al., 2025).GRPO gains in audio QA have also been reported to persist when audio is removed.Source: §2 · p. 3
  • Media interventions such as swaps, mutes, shifts, and modality shuffling diagnose audio-visual failures, while this work measures each modality’s effect on answer margins (Wen et al., 2026; Fang et al., 2026).The diagnostic tests whether the non-designated modality has zero effect for audio and visual questions.Source: §2 · p. 3
  • Closest to this setting, Chen et al. (2026b) fine-tune on aligned and misaligned clips; DMC-Repair completes each pair into a 2×2 grid where mismatch status carries no answer information.The same grid supplies both training cells and diagnostic measurements.Source: §2 · p. 3

3 CROSS-MODAL SHORTCUTS IN OMNI-MODAL LLMS

Natural multimodal accuracy cannot establish whether an omni-modal model follows the question-designated modality. The Factorized Modality Diagnostic uses crossed media swaps to measure modality effects, revealing a persistent cross-modal shortcut across tested checkpoints.

3.1 PROBLEM SETUP

  • Natural-corpus accuracy cannot verify modality use because designated and non-designated streams are correlated; swapping media between samples breaks that correlation.The diagnostic applies to audio-visual QA datasets with multiple samples sharing a question.Source: §3.1 · p. 3, §3.2 · p. 3

3.3 PILOT STUDY

  • The pilot found that image effects on audio answers were about as large as audio effects across tested checkpoints, prompts, and post-training stages.In free generation, the starting checkpoint followed audio on fewer than half of conflict cells.Source: Figure 3 · p. 4, §3.3 · pp. 4–5

3.2 THE FACTORIZED MODALITY DIAGNOSTIC

  • The diagnostic crosses own and partner images with own and partner audio across four cells, then compares answer-margin effects to separate modality contributions.For Audio questions, the Shortcut Index is zero when the image has no effect and 0.5 when image and audio contributions are equal.Source: §3.2 · p. 4

4 DMC-REPAIR

DMC-Repair builds counterfactual media combinations and labels them according to the modality designated by each question. It trains on answer tokens so the designated input determines the target rather than the original image–audio pairing.

4 DMC-REPAIR

  • DMC-Repair combines counterfactual grids, designated-modality relabeling, and answer-token supervision to train on examples where modalities disagree.The recipe uses question types and answers, rather than requiring complete reference responses.Source: §4 · p. 5

4.1 COUNTERFACTUAL GRID CONSTRUCTION

  • Pairing same-question samples with different answers yields four real-media image–audio combinations, including both mismatched cross-sample cells.The two mismatched cells combine one sample’s image with the other sample’s audio.Source: §4.1 · p. 5

4.2 DESIGNATED-MODALITY RELABELING

  • Each cell is labeled from its designated modality, making the non-designated input and matched status individually chance-level predictors; joint-modality questions retain only original cells.In joint Audio and Visual grid training, the same image is informative for Visual questions but uninformative for Audio questions.Source: §4.2 · p. 5

4.3 ANSWER-TOKEN SUPERVISION

  • DMC-Repair teacher-forces candidate answers and applies a thresholded hinge to prefer each cell’s label over the other answer, averaging loss across labeled cells.For one-token answers, the score gap is the diagnostic’s answer margin; cells at or above threshold τ receive no gradients.Source: §4.3 · p. 6

4.4 THE RECIPE IN GENERAL

  • The recipe extends to N modality streams when questions designate the answer-determining streams and real samples share a question but differ in answers.A cell’s label is the answer supplied by the sample contributing all designated streams.Source: §4.4 · p. 6

5 EXPERIMENTS

The experiments test whether DMC-Repair suppresses the cross-modal shortcut on held-out data, through construction controls, and beyond its training setting. The repair reduces image influence, generalizes across models and benchmarks, and remains effective after subsequent RL.

5.2 MAIN RESULTS

  • The repair also shifts answers toward the question’s designated modality on held-out development families, without reducing Audio-question accuracy.On Audio questions the Shortcut Index falls by two thirds; on Visual questions it moves toward its ideal value of one.Source: §5.2 · p. 7
  • 59.9% lower Shortcut Index and 13.7-point higher audio-following on the sealed confirmation set show that DMC-Repair suppresses the shortcut on held-out questions.All seven pre-registered tests met their criteria, reducing concern that the result reflects development-set tuning.Source: §5.2 · p. 7
  • On MiniCPM-o-2.6, LoRA reduces the Shortcut Index by about half on both sets and improves zero-shot AVHBench accuracy by 1.4 points.The model’s encoders and output format differ from Qwen2.5-Omni’s.Source: §5.2 · p. 7

5.3 ABLATION STUDY

  • Both conflicting inputs and designated-modality labels are needed for the strongest construction-control gains; removing either leaves the index about twice that of the complete grid.Single-direction cells retain about 80% of the index reduction, while the complete grid leads controls by 12 to 31 points in free generation.Source: §5.3 · p. 7, Figure 5 · p. 8
  • Loss and parameterization matter less: cross-entropy reaches 89% of the hinge-loss reduction, while LoRA matches full fine-tuning with 148 times fewer trainable parameters.The index falls by 0.30 to 0.36 across all four seeds.Source: §5.3 · pp. 7–8

5.4 ZERO-SHOT GENERALIZATION PERFORMANCE

  • Zero-shot on AVQA, DMC-Repair reduces the image effect by 62%, improves audio-following by 18 points, and raises matched-cell accuracy by 14 points.The frozen checkpoints were not trained, selected, or prompt-tuned on AVQA.Source: §5.4 · p. 8
  • On AVHBench audio-dependent tasks, the repair improves zero-shot accuracy without hallucination-specific training data, and subsequent RL nearly doubles the audio-hallucination gain.Video-hallucination accuracy is nearly unchanged; accuracy on audible objects rises from 49.1% to 61.7% after repair and 75.4% after RL.Source: §5.4 · p. 8

5.5 ANALYSIS

  • Adding a modality reward to RL does not remove the shortcut, and the three judge rewards evaluated on all families increase the image effect.GRPO with format and accuracy rewards increases the image effect about twice as much as the audio effect; mixing grid cells into RL data achieves less than a tenth of the repair’s effect.Source: §5.5 · pp. 8–9

6 CONCLUSION

The conclusion reports that the diagnostic exposes a cross-modal shortcut and that DMC-Repair reduces it across models and evaluation settings. The accompanying statements describe AI assistance, ethics, and reproducibility.

6 CONCLUSION

  • The Factorized Modality Diagnostic finds that tested models rely on images about as much as audio at every tested SFT and RL stage; DMC-Repair lowers the Shortcut Index by 59.9% on a sealed confirmation set.The repair generalizes across two model families and zero-shot to an unseen dataset and benchmark, and persists through subsequent RL.Source: §6 · p. 9

AI USE STATEMENT

  • Generative AI assistants supported implementation, data cleaning, framing, experimental design, hypothesis assessment, result interpretation, and paper and artifact preparation, with authors reviewing the work.The authors state that AI did not synthesize datasets or labels or formulate or prove mathematical claims.Source: AI USE STATEMENT · p. 10

ETHICS STATEMENT

  • The study uses public benchmarks, collects no new human-subject data, and reports no known ethics concerns specific to its evaluation and training-data construction.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

  • The authors report that core diagnostic, construction, training, and analysis scripts and diagnostic-family lists are available in the linked anonymous repository.They state that complete code will be released upon acceptance.Source: REPRODUCIBILITY STATEMENT · p. 10

OVERVIEW OF THE APPENDICES

The appendix supplies details and extensions for the paper’s setup, diagnostic, repair, generalization, case study, and related work.

OVERVIEW OF THE APPENDICES

  • Appendices A–L cover settings and prompts, diagnostic extensions and checks, repair ablations and confirmation tests, zero-shot protocols, added case-study cells, and further related work.Source: OVERVIEW OF THE APPENDICES · p. 13

A TRAINING AND EVALUATION DETAILS

The appendices document training and evaluation details and extend the diagnostic and repair analyses. They report that the shortcut persists across configurations, modality rewards do not remove it, and repair lowers conditional image effects.

A TRAINING AND EVALUATION DETAILS

  • The repair uses a 6,176-cell MUSIC-AVQA training pool after diagnostic families and media are removed, avoiding shared evaluation content.The pool includes 461 Audio pairs, 591 Visual pairs, and 984 Audio-Visual pairs.Source: A TRAINING AND EVALUATION DETAILS · p. 13
  • The diagnostic retains families with at least two yes and two no samples; one MUSIC-AVQA confirmation pair was later excluded for missing media.The appendix reports 773 retained MUSIC-AVQA pairs rather than 774.Source: Table 3 · p. 14

B THE DIAGNOSTIC ACROSS MODEL FAMILIES AND PROMPT FORMATS

  • The pilot diagnostic shows a shortcut across tested configurations and prompts: baseline SI values lie near 0.5, while both repairs move toward EV = 0.HumanOmniV2’s native-prompt SI is within |∆SI| < 0.1 of its base model, indicating the shortcut predates its post-training.Source: B THE DIAGNOSTIC ACROSS MODEL FAMILIES… · p. 14, Table 4 · p. 15
  • Validity checks support that the diagnostic measures modality use: shuffled mappings center effects on zero, and swaps pull answers toward incoming audio and image ground truths at similar rates.The observed effects fall outside permutation nulls (p < 0.001).Source: B THE DIAGNOSTIC ACROSS MODEL FAMILIES… · p. 15

C MODALITY REWARD DESIGNS

  • The three judge rewards evaluated on all families increase the image effect, while GRPO without a modality judge increases it about twice as much as the audio effect.The graded-description judge awards visual scores roughly three times as often as audio scores; mixing grid cells into RL lowers the index by less than a tenth of the repair’s effect.Source: C MODALITY REWARD DESIGNS · p. 16

D INTERACTION TERMS

  • The interaction term ψ measures how much the effect of swapping one modality depends on the source of the other modality; across configurations, its magnitude is smaller than either main effect.Under the stated orientation, negative ψ means the audio-swap effect is smaller when the image is the own item’s than when it is the partner’s.Source: D INTERACTION TERMS · p. 16

CONDITIONAL IMAGE EFFECTS

  • After repair, both conditional image effects decrease, and the own-audio effect’s confidence interval includes zero; the mean of per-family maximum absolute effects falls by more than half.At the RL endpoint, the partner-audio effect and both per-family means remain well below their pre-repair values.Source: CONDITIONAL IMAGE EFFECTS · p. 16, Table 7 · p. 17

E PER-QUESTION-TYPE BREAKDOWN

DMC-Repair improves matched-cell accuracy for Audio questions, while changes for the other question types are statistically inconclusive. On Visual questions, all models show a larger image effect than audio effect.

E PER-QUESTION-TYPE BREAKDOWN

  • Every model gives the image effect more than twice the audio effect on Visual questions, where the image is designated.For Audio questions, the image is non-designated and the Shortcut Index ideal is zero; for Visual questions, the audio is non-designated and the ideal is one.Source: E PER-QUESTION-TYPE BREAKDOWN · p. 17, Table 8 · p. 17
  • The repair improves matched-cell accuracy on Audio questions, while changes on the other two question types have intervals that include zero.Audio-Visual questions contribute only their original cells to training; their accuracy does not fall after repair and rises after subsequent RL.Source: E PER-QUESTION-TYPE BREAKDOWN · p. 17, Table 9 · p. 17

F PRE-REGISTERED CONFIRMATION PROTOCOL

The pre-registered confirmation tests support the repair’s effects and their retention through subsequent RL. A separate zero-shot evaluation finds reduced image reliance on AVQA, an unseen dataset of everyday sounds.

F PRE-REGISTERED CONFIRMATION PROTOCOL

  • All seven pre-registered tests met their criteria on the 257-family confirmation set, supporting the repair effect without development-set tuning.The stricter ±10% equivalence band for H2 was met, as was the specified ±20% band.Source: F PRE-REGISTERED CONFIRMATION PROTOCOL · pp. 17–18, Table 10 · p. 18
  • All three construction controls underperformed the complete-pool LoRA reference; mismatch-label outputs included explicit mismatch verdicts in about a third of conflict-cell outputs.Nearly all such verdicts were among outputs that could not be parsed as yes or no.Source: Table 10 · p. 18, F PRE-REGISTERED CONFIRMATION PROTOCOL · p. 18
  • Subsequent RL left the Shortcut Index unchanged and further improved audio-following, meeting both retention criteria.The development-set index remained near its repaired value throughout RL; Figure 9 plots Audio-question index change with pointwise 95% paired family-cluster bootstrap intervals.Source: F PRE-REGISTERED CONFIRMATION PROTOCOL · p. 18, Figure 9 · p. 19
  • Every construction control remained below the reference in audio-following when unparseable outputs were excluded or treated leniently, though that latter analysis was descriptive.The pre-registered lead-of-grid result held under all three treatments of unparseable outputs.Source: F PRE-REGISTERED CONFIRMATION PROTOCOL · pp. 18–19, Table 11 · p. 19

G ZERO-SHOT AVQA PROTOCOL

H ZERO-SHOT AVHBENCH PROTOCOL

On AVHBench, the repair improves both tasks that require listening, with subsequent RL extending the gain only on video-driven audio hallucination. Results across model families and output descriptions show that answer gains need not coincide with a general yes-rate shift or fuller audio narration.

H ZERO-SHOT AVHBENCH PROTOCOL

  • AVHBench evaluates three balanced-label yes/no tasks on original and audio-swapped AudioCaps and VALOR clips.The tasks cover video-driven audio hallucination, audio-driven video hallucination, and audio-visual matching.Source: H ZERO-SHOT AVHBENCH PROTOCOL · p. 20
  • The audio-task gain primarily reflects improved detection of audible objects, while accuracy on silent objects falls by less than four points.Among audio-task answers that changed from no at the starting checkpoint to yes at the RL endpoint, these outnumbered the reverse changes by almost three to one.Source: H ZERO-SHOT AVHBENCH PROTOCOL · p. 20, Table 14 · p. 21
  • Most outputs still omit an explicit audio description after repair and subsequent RL, despite improved answer accuracy.More than four in five outputs have empty audio descriptions; answer-token supervision sets no target for generated descriptions.Source: H ZERO-SHOT AVHBENCH PROTOCOL · pp. 20–21
  • The repair improves both listening tasks on AVHBench, and subsequent RL extends the gain only on video-driven audio hallucination.The benchmark covers 5,302 questions across three yes/no tasks; its table reports accuracy and yes-rate, and compares each model family with its own starting checkpoint.Source: H ZERO-SHOT AVHBENCH PROTOCOL · p. 20, Table 14 · p. 21
  • MiniCPM-o-2.6 also gains accuracy over its base model, most on the two hallucination tasks, while its yes-rate falls on the audio task and audio-visual matching.The model families use different answer readouts and are not directly comparable to each other.Source: H ZERO-SHOT AVHBENCH PROTOCOL · p. 21, Table 14 · p. 21

I GENERATED ANSWERS ON CONFLICT CELLS

Generated conflict-cell examples show both successful correction and persistent image-following. In some cases the repaired model and RL endpoint follow the audio where the starting checkpoint follows the image; other examples remain failures.

I GENERATED ANSWERS ON CONFLICT CELLS

  • Figure 10 shows two conflict cells where all three checkpoints follow the image, including a repaired answer that describes the audio’s violin melody.On the AVQA example, the RL endpoint describes the audio through the beach scene in the image.Source: Figure 10 · p. 21, I GENERATED ANSWERS ON CONFLICT CELLS · p. 21
  • In Figure 11, both the repaired model and RL endpoint follow the audio on four development examples and two confirmation examples where the starting checkpoint follows the image.The starting checkpoint follows the image on 67 of 260 development conflict cells; its audio descriptions are empty on every displayed cell, unlike those of the later checkpoints.Source: I GENERATED ANSWERS ON CONFLICT CELLS · p. 21, Figure 11 · p. 22
  • Figure 11’s six examples include both directions of one development pair and cells from four other pairs.Two displayed rows come from the confirmation set.Source: I GENERATED ANSWERS ON CONFLICT CELLS · p. 21, Figure 11 · p. 22

J LOSS AND PARAMETERIZATION ABLATIONS

The ablations show that counterfactual cells matter more than the choice of loss or parameterization, and the repair also reduces shortcut reliance in MiniCPM-o-2.6. The appendix documents experiment prompts and relates the diagnostic and repair to prior work.

J LOSS AND PARAMETERIZATION ABLATIONS

  • Cross-entropy achieves 89% of hinge-loss index reduction, and full fine-tuning and LoRA reach almost the same ∆SI; matched-cells-only yields less than a third of the reference reduction.Runs cover 65 development Audio families; audio-following averages, across families, the within-family share of conflict-cell answers following audio.Source: J LOSS AND PARAMETERIZATION ABLATIONS · p. 22, Table 15 · p. 23

K CROSS-MODEL GENERALIZATION ON MINICPM-O-2.6

  • On MiniCPM-o-2.6, the repair halves the index on development and confirmation sets; the image effect falls to about one tenth while the audio effect falls by about one fifth.The lower index reflects removal of most of the image effect while most of the audio effect remains.Source: K CROSS-MODEL GENERALIZATION ON… · p. 23, Table 16 · p. 23

L PROMPTS

  • The appendix documents experiment prompts and scoring, including Qwen’s HumanOmniV2 system prompt and direct scoring from log p(yes) − log p(no).Source: L PROMPTS · pp. 23–24

M EXTENDED RELATED WORK

  • The Factorized Modality Diagnostic complements prior analyses by measuring each modality’s effect on individual answers; DMC-Repair differs by changing which input predicts the training label.Source: M EXTENDED RELATED WORK · p. 25

Related papers

Loading 2609.36798v1…