PaperlayerView exact revision on arXiv

Source-linked AI summary

MoGround: Measuring and Mitigating Modality Distraction in Vision-Language Models

Luca Zhou, Bo Zhao, Rose Yu, Emanuele Rodolà, Roberto Dessì

arXiv:2609.33431v1cs.LG

TL;DR

Existing probes rarely verify that questions are answerable from exactly one modality, limiting the isolation of modality distraction. MoGround provides such a dataset and finds that distraction depends on models’ grounding strength. A MoGround-trained robustness vector reduces distraction across seven models with little average accuracy cost on standard multimodal tasks.

  • Problem

    Existing probes rarely establish single-modality answerability per item, making it difficult to isolate modality distraction.

    Source: §1 · p. 1

  • Method

    MoGround verifies that each question is answerable from exactly one modality, and its train split is used to learn a weight-space robustness vector.

    Source: §1 · pp. 1–2, §5 · p. 8

  • Results

    Distraction varies by model: the weaker-grounded modality is more distracted, and distraction scales inversely with grounding strength.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The MoGround-trained robustness vector reduces distraction across seven backbones, transfers to unseen pools, and has almost no general capability loss.

    Source: §1 · p. 2

    The dataset covers items answerable from exactly one modality and therefore does not address questions requiring both modalities together.

    Source: LIMITATIONS · p. 11

Abstract

from arXiv · show

We release MoGround, a vision-language dataset spanning four visual domains in which the answer to every question is guaranteed to be available from exactly one modality. This guarantee enables us to measure modality distraction, the failure in which a model answers a question correctly from one modality alone and then flips to a wrong answer once irrelevant content from the other modality is added. Existing probes rarely establish single-modality answerability this way, making it hard to isolate distraction in the first place. Across seven open-source VLMs, we find that modality distraction is not universal but model-dependent. The weaker-grounded modality is the more distracted one (r = +0.86), and distraction scales inversely with grounding strength (r = -0.90). The single-modality guarantee also enables a mitigation method that needs to distinguish between relevant and irrelevant context. Trained on one split of MoGround alone, a weight-space robustness vector reduces distraction on all seven models by 9% to 51%, at a cost of only 0.1 average points of accuracy on standard multimodal tasks.

1 INTRODUCTION

Existing probes rarely verify that an answer comes from exactly one modality, making distraction difficult to isolate. MoGround enables that measurement and shows that distraction varies by model and grounding strength.

1 INTRODUCTION

  • Across seven open VLMs, every model is affected by distraction, with larger drops in non-natural domains such as charts and radiology.The measurements use MoGround-Base.Source: §1 · p. 2
  • MoGround-Base verifies that each item is answerable from exactly one modality, across four visual domains, with two harder subsets.LLM judges verify the items, with confirmation by a human audit.Source: §1 · pp. 1–2
  • Training on MoGround-Base’s train split yields a robustness vector that reduces distraction across seven backbones, transfers to unseen pools, and has almost no general capability cost.The introduction presents the dataset as supporting this mitigation strategy.Source: §1 · p. 2
  • Distraction varies across models: the weaker-grounded modality is more distracted, and distraction scales inversely with grounding strength.The paper relates distraction to the model’s accuracy when answering from the modality containing the answer.Source: §1 · p. 2

2 RELATED WORK

Prior work studies modality bias, modality preference, and related failure modes, but the cited probes generally do not verify single-modality answerability per item. MoGround instead measures distraction across models and domains.

2 RELATED WORK

  • Prior work reports language-prior over-reliance, including for plots, while Lee et al. (2025) report that this bias decreases with scale; the paper says scale alone does not predict distraction.The authors use this distinction to separate distraction from the bias measured by Lee et al. (2025).Source: §2 · p. 2
  • Many probes assume visual dependency or create conflicts to test modality preference; they do not verify that each item is answerable from exactly one modality [1, 2, 3, 4, 5, 6].The paper distinguishes following one modality during a conflict from distraction, which it measures without engineered conflict.Source: §2 · pp. 2–3
  • Related studies use different protocols: surgical-VQA auditing, image removal, or irrelevant visual clutter, whereas MoGround builds a multi-domain set verified for single-modality answerability [1, 2, 3].The passage identifies Sun et al. (2026) as studying the mirror setting by adding irrelevant visual clutter.Source: §2 · p. 3
  • The paper contrasts its model-dependent account with universal-interference fine-tuning and notes that a router works in its experiments only when told which items are distracted [1, 2].It also reports that per-item distraction was not controlled by a shared direction in its tests.Source: §2 · p. 3

3 DATA CURATION PIPELINE

MoGround-Base uses a generate-then-verify pipeline to retain questions answerable from exactly one modality. MoGround-Human and MoGround-Retrieved add harder evaluation pools with distinct construction procedures.

3 DATA CURATION PIPELINE

  • MoGround-Base generates questions from image-caption pairs, then keeps an item only when a second oracle answers correctly with the grounded modality alone and with both inputs, but fails with the other modality alone.The symmetric rule applies to vision-grounded and text-grounded items.Source: §3 · pp. 3–4
  • The pipeline uses four visually different source datasets and two oracles; filtering removes 74.5% of generated candidates, ranging from 46.9% on radiology to 85.9% on charts.Each source dataset is generated and verified by one oracle.Source: §3 · p. 4
  • Cross-oracle agreement is 94%, human annotators agree on 98.4%, and the single-modality property holds for 97.6% and 96.8% of their audited sample.At most 1.6% of the human-audited items leak the answer into the wrong modality.Source: §3 · p. 4
  • MoGround-Human provides 125 human-written captions as a held-out test set and is not used for selection or training.The captions may tempt a wrong option while leaving the answer unavailable from caption alone.Source: §3 · p. 4
  • MoGround-Retrieved pairs A-OKVQA and RACE-high questions with distractors from the other modality, audits retrieved captions, and releases 4,757 items.The pool contains vision-grounded and text-grounded questions; the paper measures every asymmetry and transfer result on it.Source: §3 · p. 4

4 MEASURING MODALITY DISTRACTION

The section measures modality distraction as a model-relative conditional flip and finds that distraction varies by model and grounding strength. It also traces when caption-induced flips emerge and which image-grounded answers are vulnerable.

4 MEASURING MODALITY DISTRACTION

  • V-distraction ranges from 3.4% on InternVL3-8B to 13.6% on LLaVA-1.5-7B, while charts and radiology average 11.3% and 10.5%, versus 6.8% on photos.Same-length replacement text always lowers distraction, indicating the measured effect depends on caption content rather than prompt length alone.Source: §4 · p. 5, Table 1 · p. 5
  • T-distraction is approximately zero on MoGround-Base, where above-99% text-only accuracy leaves answers too far from the decision boundary for images to flip them.The section therefore measures v-distraction well but t-distraction barely at all on this pool.Source: §4 · p. 5

4.1 THE RELATION BETWEEN ACCURACY AND DISTRACTION RATE

  • r = −0.90 across 70 backbone×domain×modality configurations: distraction scales inversely with grounding strength, and weaker-grounded modalities are more distracted.The grounding-gap/distraction-asymmetry correlation is r=+0.86 across seven backbones; a within-model grounding ablation also correlates at r=−0.91.Source: §4.1 · pp. 5–6, Figure 3 · p. 6
  • The two best-grounded systems measured have V-only accuracies of 87% and 95% versus a 75% open-model average, and are among the least distracted (v-distraction 9.1% and 11.7% versus 27.4%).This shows that distraction is reduced for frontier models, not removed.Source: §4.1 · p. 7

4.2 TRACING DISTRACTIONS

  • Before captions are added, the image-only answer margin predicts distraction with AUROC 0.87 on MoGround-Base and 0.92 on MoGround-Retrieved.The margin is the correct-option logit minus the highest incorrect-option logit; smaller positive margins require less perturbation to flip an answer.Source: §4.2 · p. 7
  • Distraction emerges abruptly at a mid-to-late layer, where distracted items commit to the caption-induced answer; commitment occurs at 45–89% depth and later in better-grounded models.The commit-depth correlation with grounding across backbones is r=+0.88.Source: §4.2 · p. 7, Figure 4 · p. 7

5 MITIGATING DISTRACTION

The authors tune a LoRA-based robustness vector on MoGround-Base and evaluate its distraction reduction and capability cost across reporting pools. At the selected strength, it reduces distraction across all backbones and datasets while largely preserving benchmark capability.

5 MITIGATING DISTRACTION

  • The authors divide MoGround-Base into train, validation, and test splits and use MoGround-Retrieved, four benchmarks, and MoGround-Human across selection and reporting pools.The report pool is held out for one final test; benchmark questions 1–400 are used for selection and 401–800 for reporting.Source: §5 · p. 8
  • The method fine-tunes a LoRA adapter on MoGround-Base training data and scales its weight update with w, selected as 0.5.Distraction reaches a minimum around w=0.5–0.75 and regrows beyond it, while capability deteriorates monotonically past that range.Source: §5 · p. 8, Figure 5 · p. 8
  • V-distraction improves for every backbone on every dataset, by up to 73% on MoGround-Base test, 67% on MoGround-Human, and 39% on MoGround-Retrieved held-out.On six of seven backbones, capability changes on the four standard benchmarks range from +0.9 to −0.5pp.Source: §5 · p. 9
  • Among the tested alternatives, the robustness vector is the only method that reduces distraction on all three pools.Prompting and chain-of-thought fail to reduce distraction consistently; contrastive decoding reduces it on MoGround-Base and MoGround-Human but not MoGround-Retrieved.Source: §5 · p. 9, Table 4 · p. 9

6 CONCLUSION

The conclusion presents MoGround as a dataset for measuring and mitigating modality distraction, with per-item modality-grounding verification. The authors report model-dependent distraction and a fine-tuning strategy that reduces it at negligible average capability cost.

6 CONCLUSION

  • MoGround questions are guaranteed answerable from either the image or caption, with the guarantee checked by human audit.The authors frame per-item modality-grounding verification as a design principle for evaluating which modalities models rely on and how easily answers are distracted.Source: §6 · p. 9
  • Modality distraction depends on a model’s grounding strength, and the paper reports that it occurs abruptly in a mid-to-late layer.Source: §6 · p. 9
  • A fine-tuning strategy reduces distraction at negligible capability cost on average.Source: §6 · p. 9

AI USE STATEMENT

The statement identifies three roles for language models in the work: dataset construction, evaluation subjects, and writing and coding assistance.

AI USE STATEMENT

  • Two language models serve as data-pipeline oracles, building and verifying benchmark items; one also writes adversarial captions for fine-tuning.Each source is handled end-to-end by one oracle, with the other oracle cross-checking its items.Source: AI USE STATEMENT · p. 10
  • Gemini 3.5 Flash and Claude Sonnet 5 are evaluated as proprietary-model subjects and are not used to produce other results.Source: AI USE STATEMENT · p. 10
  • Claude Fable 5 and Opus 5 assist with code development, analysis tooling, and wording; the authors report that research design and analysis are theirs.The statement says reported numbers come from released scripts and artifacts rather than an LLM.Source: AI USE STATEMENT · p. 10

ETHICS STATEMENT

The dataset uses four public image-caption benchmarks, with de-identified ROCO radiology images and no new personal data. The authors identify source-benchmark biases and benchmark training as risks.

ETHICS STATEMENT

  • MoGround-Base draws on four publicly available image-caption benchmarks; ROCO radiology images are de-identified, and no new personal data are introduced.Source: ETHICS STATEMENT · p. 10
  • Only the authors provide human annotation; no external annotators or crowdworkers are involved.Source: ETHICS STATEMENT · p. 10
  • The authors cite source-benchmark biases and the possibility of training on the benchmark instead of using it only for evaluation as risks.Source: ETHICS STATEMENT · p. 10

LIMITATIONS

The paper bounds its measurements to specified models, inputs, and single-modality-answerable items, while noting residual oracle and reproducibility constraints. It releases the dataset and supporting code and records for reuse.

LIMITATIONS

  • The claims concern multiple-choice answering by open-weight 2–8B VLMs; retrieval and agentic pipelines, plus adversarial or answer-bearing context, are not tested.The grounding-strength relation is fitted over 70 configurations from seven backbones, a small sample for a scaling claim.Source: LIMITATIONS · pp. 10–11
  • 3.4%–13.6%: absolute distraction on MoGround-Base is modest across backbones, and the dataset excludes questions requiring both modalities.The answer to every MoGround item is available from exactly one modality.Source: LIMITATIONS · p. 11
  • The oracle filter may retain generating or judging-model biases, and rerunning the proprietary-oracle pipeline may not reproduce the fixed released dataset exactly.The authors mitigate but do not eliminate oracle bias; released per-item records are fixed artifacts.Source: LIMITATIONS · p. 11

A RELEASED ARTIFACTS

  • The open-licensed release includes MoGround datasets, audit records, pipeline and evaluation code, and trained sparse autoencoders.Image files are not redistributed for two sources, but upstream identifiers, checksums, and a rebuilding script are provided.Source: A RELEASED ARTIFACTS · p. 14

B DATASET CONSTRUCTION PIPELINE

This section describes the construction and evaluation setup for MoGround-Base, including modality-specific questions filtered through a three-condition oracle check. Examples show image- and passage-grounded questions paired with content from the other modality as a distractor.

B.1 THREE-CONDITION VERIFICATION

  • Items were retained only when image-only, text-only, and image-plus-text answers matched a vision-grounded or text-grounded signature.Verification establishes answerability relative to the verifying oracles, not absolute ground truth.Source: §B.1 · pp. 14–15
  • ROCO reports 632 items (Table 5); verification is operational: single-modality answerability is assessed relative to these oracles, not treated as absolute ground truth.Source: §B.1 · p. 15
  • In a 240-item cross-oracle check, the second oracle achieved 94% mean accuracy across the two labels, with 88.3% for vision and 99.2% for text.Each source was verified by one oracle and re-answered under all three conditions by the other.Source: §B.1 · p. 15

B.2 ORACLE PROMPTS

  • Generation prompts avoid generic perceptual questions when dense captions may reveal their answers, instead targeting omitted visual details and injected non-physical text facts.Questions use modality-neutral phrasing, and text facts are inserted naturally while preserving the original caption’s content.Source: §B.2 · pp. 15–16
  • A fixed 60/20/20 split supports supervised uses; mitigation trains on the 2,050-item training split, while its MoGround-Base effect is tested on held-out data.Frozen-model analyses use the full oracle-verified set, and transfer evaluation uses MoGround-Retrieved, which is never used for fine-tuning.Source: §B.2 · p. 17

C EXAMPLE MOGROUND-BASE AND MOGROUND-RETRIEVED ITEMS

  • Examples span natural photos, charts, paintings, and radiology, while hand-authored items use captions that prime a wrong option without answering the question.The examples show how captions can mention distractor options or related facts while leaving the correct answer grounded in the other modality.Source: C EXAMPLE MOGROUND-BASE AND… · pp. 17–18

C.6 MOGROUND-RETRIEVED POOL

  • For the image-grounded example, the answer is read from the image and the retrieved caption is the cross-modal distractor.The listed options include “mom,” “rabbit,” “little girl,” and “man.”Source: §C.6 · p. 22
  • For the passage-grounded example, the question asks “What is DNA?” and the retrieved image is the cross-modal distractor.The passage provides the grounding source, and the options include “DNA is a substance that makes up genes.”Source: §C.6 · p. 22

D.1 HUMAN AUDIT OF MOGROUND-BASE: SINGLE-MODALITY ANSWERABILITY

Human audits support the dataset’s single-modality and irrelevance checks, while supplementary analyses compare distraction predictors, caption controls, and effects across domains and backbones. The analyses also document how model inputs are processed and how caption additions alter answers.

D.1 HUMAN AUDIT OF MOGROUND-BASE: SINGLE-MODALITY ANSWERABILITY

  • 234/250 and 232/250 sampled MoGround-Base items passed the strict audit, and the annotators agreed on 236/250 items.An item passed only when its intended modality answered it and the other modality did not; counting grounding failures alone, the guarantee held on 97.6% and 96.8% of the sample.Source: §D.1 · pp. 22–23, Table 6 · p. 23

D.2 HUMAN AUDIT OF MOGROUND-RETRIEVED: IRRELEVANCE OF THE ADDED CONTEXT

  • 9.6% of retrieved captions violated irrelevance; after those items were dropped, the 4,757-item pool was nearly leak-free, with one genuine leak in 383 usable judgments.Two annotators judged 98.4% and 97.9% of usable kept items answer-irrelevant; leaks concentrated in the judge-dropped pool.Source: §D.2 · pp. 23–24

D.4 THE OPEN-WEIGHT BACKBONES

  • Across the seven backbones, image tokens are placed in the same sequence as text tokens, although image encoders and image-processing approaches differ.The models share a language-model-plus-image-encoder structure, with differing encoders, resolutions, and image handling.Source: §D.4 · p. 24

D.5 PER-DOMAIN BREAKDOWN

  • Non-natural domains show more vision distraction than photos in five of seven backbones; Table 9 reports counts and Wilson 95% intervals behind the rates.Text-side distraction is at or near zero in every domain, with 182–557 solved items per configuration.Source: §D.6 · p. 24, Figure 6 · p. 25, Table 8 · p. 25

D.7 BOTH DIRECTIONS OF THE CAPTION’S EFFECT

  • Caption additions both break image-answerable items and fix image-only misses on every backbone, but breaking dominates, leaving V+T accuracy below V-only accuracy.Table 10 defines V-Distraction as Broken over image-answerable items and Recovery as Fixed over the rest.Source: §D.7 · p. 25, Table 10 · p. 26

E.1 DOES ANYTHING ELSE PREDICT DISTRACTION?

  • Grounding strength predicts distraction best among tested alternatives (r = −0.899; held-out error 1.60 pp); model scale (r = +0.085) and distractor strength (r = −0.220) do not.After controlling for confounds, the correlation changes from −0.90 to −0.84; held-out prediction errors are reported in percentage points.Source: §E.1 · p. 26

E.2 DISTRACTION IS NOT AN ARTIFACT

  • The true caption out-flips scrambled and content-free controls on all seven backbones, with the ranking true > {mismatched, scrambled} > neutral holding throughout.All contrasts use the same V-only-correct items; the ordering is attributed to semantic content.Source: §E.2 · p. 27, Table 11 · p. 27

E.3 SAE-FEATURE ABLATION AND PATCHING

The ablation tests whether shared residual features or directions explain distraction, and whether restoring an item's own image-only state can recover its answer. Only the item-specific patch substantially outperforms its control.

E.3 SAE-FEATURE ABLATION AND PATCHING

  • Each edit is applied only to distracted items and only at the listed layer's final token.The controls use random features, a random direction, or another item's residual, depending on the intervention.Source: §E.3 · p. 27
  • 93.5% of distracted items recover with their own image-only residual, versus 24.3% with another item's residual.The grid evaluates recovery on distracted vision items; the same-item patch is the positive control.Source: §E.3 · p. 27, Table 12 · p. 28
  • SAE-feature removal and removal of a probe direction recover no more than their random controls across K values and both backbones.The edits target features that fire most on distracted rather than robust items, or the single direction separating those groups.Source: §E.3 · p. 27

E.4 ALL INTERVENTIONS ON ONE ITEM SET

Across the intervention comparisons, the robustness vector reduces distraction while limiting the text-side increase, and its end-to-end gains depend on how prevalent distraction is. The section also details its training recipe, evaluation splits, and prompting baselines.

E.4 ALL INTERVENTIONS ON ONE ITEM SET

  • Intervention comparisons score backbones on identical held-out items, while activation-edit results use distracted items only and are not comparable with the other methods.The broader comparison also reports capability costs for interventions that alter the served model.Source: §E.4 · p. 28, Table 14 · p. 29, Table 15 · p. 29
  • The oracle-gated image-only patch removes 75–100% of distraction, but a learned detector's patch removes only 2–22%, versus 75–100% with oracle selection.Applying the patch to every item instead reduces text accuracy by 27–70pp.Source: §E.4 · p. 30
  • On MoGround-Retrieved, the vector cuts Qwen2.5-VL-3B v-distraction by 20% for +0.3pp t-distraction; it is the only method reducing LLaVA-NeXT-8B v-distraction.Across methods that reduce vision-side distraction, the vector has the smallest text-side increase; LLaVA-NeXT-8B's t-distraction rises by 2.4pp from 3.0%.Source: §E.4 · p. 30, Table 16 · p. 31

E.5 FINE-TUNING RECIPE (THE ROBUSTNESS VECTOR)

  • Training uses attention-only LoRA on MoGround-Base's 2,050-item train split, with caption dropping or adversarial captions for vision-grounded items and retained, double-weighted captions for text-grounded items.A KL anchor against the frozen base model helps prevent drift.Source: §E.5 · p. 31

E.7 END-TO-END ACCURACY CHANGE

  • Every backbone gains on both MoGround-Base reporting surfaces; on MoGround-Retrieved, where baseline distraction is 2–3%, average accuracy is neutral at +0.6pp.Per-model changes on MoGround-Retrieved are within seed noise.Source: Table 17 · p. 32

E.6 SELECTION AND REPORTING SPLITS

  • Selection uses validation and development data, while reporting uses held-out MoGround-Base and MoGround-Retrieved halves and later capability questions.The capability benchmarks' first 400 questions are for selection and questions 401–800 for reporting.Source: §E.6 · p. 32

F.1 ONE-LINE INSTRUCTION

  • The prompt-instruction baseline states the policy the vector is trained to implement, yet still fails to reduce distraction.This is a generous baseline because the instruction is explicitly told which modality to prefer.Source: §F.1 · p. 32, F PROMPTING BASELINES · p. 32

F.2 ZERO-SHOT CHAIN-OF-THOUGHT

  • Zero-shot chain-of-thought tests whether deliberation alone helps, using a fixed instruction and up to 256 generated tokens per query.Generations without an extractable answer are excluded from reported numbers.Source: §F.2 · p. 33

G SAE TRAINING DETAILS

The SAE details specify the training corpus, architecture, and optimization setup, along with the reconstruction and feature-activity measures reported for released checkpoints.

G SAE TRAINING DETAILS

  • The SAEs are trained unsupervised on 1,242,328 image-caption pairs from four sources matched to MoGround-Base's visual domains.The sources are CC3M, PlotQA, WikiArt, and OpenI; training uses reconstruction alone.Source: G SAE TRAINING DETAILS · p. 33
  • The released SAEs are Top-K sparse autoencoders with K=32 and width 8 × dmodel, trained on the frozen backbone's last-token residual.The encoder and decoder are separate matrices, and both biases are learned.Source: G SAE TRAINING DETAILS · p. 33
  • Table 18 reports reconstruction FVU and the fraction of features that fire at least once on an evaluation batch.Lower FVU is better.Source: Table 18 · p. 33, G SAE TRAINING DETAILS · p. 33
Loading 2609.33431v1…