Source-linked AI summary
Grounding on Request: Learning Modality Control in Vision-Language Models
Maxime Dassen, Alexander Martin, Andrew Yates, Reno Kriz, William Walden
TL;DR
When image and text conflict, correctness alone does not establish whether a VLM followed the requested evidence. The paper learns modality control using grounding words and paired counterfactual supervision, finding stronger control than natural-language instructions while LoRA remains strongest overall.
Problem
When image and text disagree, it remains unclear whether a VLM can reliably follow the user’s chosen modality, since a correct answer alone does not establish evidence dependence.
Source: §1 · p. 1Method
The paper introduces visual and textual grounding words by training only their embeddings, and uses paired counterfactual edits to supervise answer-probability shifts.
Source: §1 · p. 2Results
Across benchmarks, learned modality words improve control over natural-language instructions, one embedding per modality provides lightweight visual control, and LoRA remains strongest overall.
Source: §1 · p. 2Takeaways & Limitations
Reliable modality control should be evaluated by whether answers follow edits to the requested evidence, not only by whether original answers are correct.
Source: §1 · pp. 1–2Evaluation uses fixed answer choices and held-out values from the same task type, so it does not test arbitrary distribution shifts or fully open-ended generation.
Source: §7 · p. 9
Abstract
from arXiv · showhide
When an image and its caption disagree, can a user reliably control which one a vision-language model (VLM) follows? Accuracy alone cannot answer this question, since a model might give the correct answer from parametric knowledge without following the requested evidence. We therefore evaluate modality grounding control through conflict-preserving counterfactual interventions. Specifically, Flip measures whether an initially correct answer follows an edit to the requested modality, while Stay measures whether it remains stable when the competing modality is edited. To learn this control, we build on neologism learning by introducing new words for visual and textual grounding, training only their embeddings while keeping the pretrained model fixed. We also introduce Counterfactual Twin-Margin (CTM) supervision, which uses paired interventions to teach how answer probabilities should change when the requested evidence changes. Across three VLMs on three existing benchmarks and our multi-attribute Ground-CF benchmark, we find that learned modality words improve control over natural-language instructions. Surprisingly, a single learned embedding per modality provides strong lightweight visual control and shows little average degradation on held-out values. LoRA remains strongest overall, although its visual advantage narrows under held-out evaluation. Moreover, counterfactual supervision improves control in several settings where standard supervision is weaker. Finally, we test our method on real-world news video and find that visual control improves when the relevant evidence is made easier to access. Overall, our results show that reliable modality control requires testing not only whether an answer is correct, but whether it continues to follow edits to the modality of choice.
1 INTRODUCTION
When modalities conflict, correctness alone cannot show whether a VLM followed the user’s requested evidence. The paper evaluates that control with paired counterfactual edits and learns it with modality words and counterfactual supervision.
1 INTRODUCTION
- FLIP tests whether an initially correct answer follows an edit to the requested modality, while STAY tests whether it remains stable when competing evidence changes.The frog example illustrates why an initially correct answer alone does not establish reliable grounding.Source: §1 · pp. 1–2, Figure 1 · p. 2
- The paper introduces <look> and <read> as visual and textual grounding words, training only their embeddings while keeping the pretrained model fixed.Its Counterfactual Twin-Margin objective uses paired interventions to make the original answer less likely and the replacement answer more likely after a requested-modality edit.Source: §1 · p. 2
- Learned modality words improve control over natural-language instructions across benchmarks; one embedding per modality gives lightweight visual control, while LoRA remains strongest overall.The visual advantage of LoRA narrows on held-out evaluations, and counterfactual supervision helps in several settings where standard supervision is weaker.Source: §1 · p. 2
- The work combines conflict-preserving evaluation, Ground-CF, learned user-selectable modality controls, and tests on realistic news-video frames.The authors report that visual control on news video improves when relevant evidence is easier to access.Source: §1 · p. 2
2 RELATED WORK
Prior work studies multimodal conflict, evidence dependence, and methods for steering model behavior. This paper focuses instead on user-selectable control over which modality grounds an answer, including under counterfactual edits.
Modality conflict and grounding
- Prior conflict research examines modality dominance and hallucinations, whereas this work asks whether users can select either modality reliably at inference time.Prior approaches include strengthening visual grounding, reducing textual influence, and explicit instructions.Source: §2 · p. 3
Counterfactual dependence and attribution
- Counterfactual interventions test whether predictions depend on evidence by changing it; this work tests both requested-source following and stability to competing-source edits.It also uses paired interventions during training, not only evaluation.Source: §2 · p. 3
Steering, neologisms, and training objectives
- Unlike prior neologism learning for concepts or behaviors, this paper uses new vocabulary embeddings as explicit user-selectable modality controls.CTM couples candidate answers across paired evidence edits, unlike standard SFT, which optimizes examples independently.Source: §2 · p. 3
3 MEASURING AND LEARNING MODALITY GROUNDING CONTROL
The section defines modality control through counterfactual behavior and uses paired evidence edits to train that behavior. It distinguishes following requested-source changes from remaining stable under changes to competing evidence.
3.1 MEASURING MODALITY GROUNDING CONTROL
- Modality control is measured by whether predictions follow edits to the requested source and remain stable when only competing evidence changes.The setup uses a question and evidence sources, scoring fixed candidate answers by sequence log-likelihood rather than open generation or an external judge.Source: §3 · p. 3, §3.1 · pp. 3–4
- Accuracy checks correctness on the original input, while FLIP and STAY test whether initially correct predictions behave appropriately under requested- and competing-source edits.Both measures are conjunctive at the item level and cannot exceed accuracy.Source: §3.1 · p. 4
3.2 LEARNING MODALITY CONTROL WITH COUNTERFACTUAL SUPERVISION
- Neologism learning adds <look> and <read> tokens to natural-language requests and trains only their embeddings while keeping the pretrained model fixed.CTM uses paired examples to encourage the old answer to become less likely and the replacement more likely after a requested-source edit; training combines CTM with SFT.Source: §3.2 · p. 4
- When both CTM margins are positive, the relative preference shifts toward the replacement answer, but this need not produce a successful FLIP or preserve STAY.CTM does not explicitly train invariance to edits of the competing modality.Source: §3.2 · pp. 4–5
4 EXPERIMENTAL SETUP
The experiments compare modality control across controlled image–text conflicts and a more realistic news-video setting. The setup specifies the benchmarks, models, and adaptation methods used for these comparisons.
4 EXPERIMENTAL SETUP
- The experiment sequence examines default modality preferences, tests natural-language requests, and then evaluates learned modality control.Controlled image–text conflicts are the main setting, followed by more realistic news-video frames.Source: §4 · p. 5
4.1 BENCHMARKS
- The study evaluates four controlled image–text benchmarks, including Ground-CF interventions across color, shape, count, and spatial relations, plus MultiVENT-G news-video frames.Primary evaluations use conflict-preserving twins whose edited value differs from both the original and the value supported by competing evidence.Source: §4.1 · p. 5
- Experiments use Gemma-3-4B, Qwen3.5-4B, and Qwen3.5-2B, comparing neologisms with frozen models, natural-language instructions, prompt tuning, and LoRA.Neologisms and prompt tuning use one learned embedding per request by default; LoRA uses rank 16 and a specialist adapter per modality.Source: §4.1 · p. 5
5 ANALYTICAL RESULTS
The analyses show that default modality preferences vary, so accuracy and natural-language requests alone do not establish reliable user control. Learned modality words provide stronger visual control than prompt tuning, while counterfactual supervision helps in several weaker settings.
5 ANALYTICAL RESULTS
- Default modality preferences vary across models and benchmarks, so no single modality should generally be amplified or suppressed.On COCO-CF all three models favor text, while both Qwen models favor images on Ground-CF and ColorSwap.Source: §5 · pp. 5–6, Table 1 · p. 6
5.1 ACCURACY DOES NOT MEASURE RELIABLE MODALITY GROUNDING CONTROL
- Natural-language instructions provide unreliable modality control, especially for visual requests, where every model–benchmark setting loses at least 10% of initially correct answers under FLIP.In 9 of 12 image-request settings, at least 25% fail to follow the image edit; Figure 2 identifies correct predictions that fail counterfactual tests.Source: §5.1 · p. 6, Figure 2 · p. 6
5.2 LEARNING RELIABLE MODALITY GROUNDING CONTROL
- Neologisms provide the strongest lightweight visual control, outperforming prompt tuning on ID and held-out FLIP and STAY, while LoRA remains strongest overall.Neologism visual FLIP and STAY decrease by only 0.7% and 1.6% on held-out values; LoRA's decrease is 9.4% and 8.6%, respectively.Source: §5.2 · p. 7, Table 2 · p. 7
- CTM improves visual FLIP beyond paired SFT in ID and held-out evaluation, with gains concentrated in settings where SFT is weaker.Paired counterfactual examples alone improve caption control but not visual control; CTM's effect is not uniform across models and metrics.Source: §5.2 · pp. 7–8, Figure 3 · p. 7
- Models often distinguish <look> and <read> but are substantially less reliable at identifying which modality each word denotes.The words can still control which modality the model follows under intervention.Source: §5.2 · p. 8
1. Distinguishing the learned words
The supplied passage poses a question about whether the learned words are distinguishable.
1. Distinguishing the learned words
- The section asks whether <look> and <read> are the same or different.Source: §1 · p. 8
2. Identifying the source
The models' ability to identify what the learned words denote is tested separately from their ability to distinguish the words.
2. Identifying the source
- When asked what <look> asks it to use, Gemma-3-4B answers that it means the caption.Source: §2 · p. 8
- Models often distinguish the learned words but are substantially less reliable at identifying the modality each denotes.Source: Figure 4 · p. 8
6 APPLIED RESULTS
On real-world news-video frames, visual control is difficult even when frame-level answers are often correct, and making relevant evidence easier to access improves visual FLIP.
6.1 MODALITY GROUNDING CONTROL ON REAL-WORLD NEWS VIDEOS
- The news-video interventions are not perfectly isolated because changing the frame can also change other parts of the scene.Source: §6.1 · p. 8
- On MultiVENT-G, full-frame visual control is considerably harder than textual control despite per-frame accuracy of .50–.66; models can answer frames correctly without reliably changing answers as visual evidence changes.After cropping, neologisms outperform parameter-matched prompt tuning on Qwen3.5-4B (.486 vs. .347) and Qwen3.5-2B (.295 vs. .214); Gemma-3-4B is the exception, where prompt tuning remains slightly stronger.Source: §6.1 · p. 9
- Cropping around annotated evidence improves visual FLIP for every method and model without retraining.For neologisms, FLIP rises from .202 to .486 on Qwen3.5-4B and from .145 to .295 on Qwen3.5-2B.Source: Table 3 · p. 8, §6.1 · p. 9
7 CONCLUSION
The conclusion argues that modality control should be judged by whether predictions track evidence edits, not correctness alone. Its evaluation uses conflict-preserving counterfactuals across controlled benchmarks, including Ground-CF.
7 CONCLUSION
- Neologisms provide strong lightweight modality control and remain competitive with parameter-matched prompt tuning, while LoRA is strongest overall.Neologisms are particularly robust in harder visual settings; LoRA’s visual control degrades more under held-out evaluation despite using 5,300–11,600× more trainable parameters than L=1 neologisms and prompt tuning.Source: §7 · p. 9
- The evaluation uses fixed answer choices and held-out values from the same task type, so it does not test arbitrary distribution shifts or fully open-ended generation.MultiVENT-G uses more realistic scenes, but its frame interventions are less controlled and its crop analysis relies on human-annotated evidence.Source: §7 · p. 9
A DATA AND EVALUATION PROTOCOL
- Conflict-preserving twins keep the sources in disagreement while changing the requested source for FLIP and the competing source for STAY.The question, grounding request, and candidate set remain fixed across the original example and its twins.Source: A DATA AND EVALUATION PROTOCOL · p. 14
- Ground-CF applies the intervention structure to color, shape, count, and spatial relations, with specified attribute values and request–attribute combinations held out.It contains 748 ID and 800 held-out items; valid image twins are available for 707 ID and 800 held-out items.Source: A DATA AND EVALUATION PROTOCOL · p. 14
- Accuracy measures original-input correctness, while FLIP and STAY require correct answers across edits to the requested and competing sources, respectively.Ground-CF image FLIP and text STAY use the eligible image-twin subset; image STAY and text FLIP use all examples.Source: A DATA AND EVALUATION PROTOCOL · p. 14
B TRAINING AND EVALUATION DETAILS
The section details how modality requests are represented and trained, including counterfactual supervision, and reports comparisons across the full model–benchmark grid. Neologisms improve visual FLIP over natural-language instructions across that grid, while LoRA provides the strongest average control.
B TRAINING AND EVALUATION DETAILS
- The controlled experiments use one learned embedding per request for neologisms and prompt tuning; the main comparison uses rank-16 LoRA with one specialist adapter per modality.The lightweight methods use 4,096–5,120 trainable parameters across both requests, compared with 21.8–59.6 million for the two specialist LoRA adapters.Source: B TRAINING AND EVALUATION DETAILS · p. 14, Table 5 · p. 15
- Methods receive the same multimodal input and differ in how the request is represented: English wording, a learned prompt vector, a neologism in natural language, or a modality-specific LoRA adapter.For neologisms and prompt tuning, gradients are restricted to added embedding rows, which are inserted into the original pretrained embedding matrix at evaluation.Source: B TRAINING AND EVALUATION DETAILS · pp. 14–15
- Moving the Ground-CF neologism to the prompt-tuning position changed visual FLIP from .725 to .716 in ID and .608 to .609 on held-out values; placement alone did not explain the interface difference.Neologisms retained strong text control at either position, while prompt tuning remained stronger on Ground-CF visual control.Source: B TRAINING AND EVALUATION DETAILS · p. 15, Table 8 · p. 16
- Candidate scoring sums token log probabilities for each fixed answer candidate conditioned on the input, request, and preceding candidate tokens, then selects the highest-scoring candidate.The reported controlled results are forced-choice likelihood results rather than free-generation accuracy.Source: B TRAINING AND EVALUATION DETAILS · p. 16
C NEOLOGISM LEARNING AND COUNTERFACTUAL SUPERVISION
- CTM compares answers across an original context and a requested-source twin, encouraging the old answer’s likelihood to fall and the new answer’s likelihood to rise after the edit.If both margins are positive, the model’s relative preference shifts toward the replacement answer.Source: C NEOLOGISM LEARNING AND… · pp. 16–17
- CTM gives a directional signal but does not guarantee that the replacement answer becomes highest-scoring, and it does not explicitly train invariance to competing-source edits.The Stay ablation additionally changes the competing source while keeping the requested answer fixed.Source: C NEOLOGISM LEARNING AND… · p. 17
- Comparing CTM with paired SFT tests whether coupling original and counterfactual contexts adds benefit beyond independently optimizing the same examples.The comparison is reported for Gemma-3-4B and Qwen3.5-2B, and conclusions are restricted to those two models.Source: C NEOLOGISM LEARNING AND… · p. 17
D.1 FULL MODEL–BENCHMARK RESULTS
- Across the full grid, neologisms improve visual FLIP over natural-language instructions in every cell, while the substantially larger LoRA adapters provide the strongest average control overall.Prompt tuning and neologisms are closer, with their relative performance depending on the model and benchmark.Source: §D.1 · p. 17
D.2 GROUND-CF RESULTS BY ATTRIBUTE
Ground-CF visual control is strongest on color and shape and harder on count and spatial relations, especially for held-out values. Across the additional analyses, counterfactual supervision improves control in some settings, while wording and source-order results have specific scope limits.
D.2 GROUND-CF RESULTS BY ATTRIBUTE
- Visual FLIP for neologisms falls from .725 to .608 on Ground-CF from ID to held-out evaluation, with count and spatial relations driving much of the difficulty.Neologisms remain particularly strong on color and shape, while text control is substantially stronger across attributes.Source: §D.2 · p. 17
D.3 CONTROL CONDITIONAL ON AN INITIALLY CORRECT ANSWER
- 16%–60% of initially correct visual answers to natural-language instructions still fail to follow the image edit.Text requests are generally more reliable, so correctness does not guarantee control even when the initial answer is correct.Source: §D.3 · p. 17, Table 12 · p. 21
D.4 OBJECTIVE COMPARISON
- CTM improves held-out visual FLIP/STAY beyond paired SFT by +.065/+.059 for Gemma-3-4B and +.031/+.115 for Qwen3.5-2B.Adding explicit STAY supervision does not consistently help: Qwen3.5-4B text FLIP falls from 1.000 to .512 in ID evaluation and from .994 to .610 on held-out values, so CTM without that term is the reference objective.Source: §D.4 · pp. 17, 21, Table 13 · p. 21
D.5.1 HELD-OUT REQUEST CARRIERS AND QUESTION WORDING
- Joint LoRA shows little sensitivity to tested question and request wording changes, with a largest absolute image-FLIP change of .024.The reported comparisons do not establish a wording-transfer advantage for neologisms over joint LoRA.Source: §D.5.1 · p. 22, Table 14 · p. 22, §D.5 · p. 22
D.5.2 SENSITIVITY TO IMAGE–CAPTION ORDER
- For Gemma-3-4B L = 5 neologisms, the ID caption-control gap is .000 under CTM versus .060 under paired SFT; the reported held-out gaps are also smaller.Reduced order sensitivity is not specific to neologisms: joint LoRA reaches 1.000 caption FLIP in both orders, and the L = 1 neologism has larger gaps under CTM+Stay.Source: §D.5.2 · p. 22, Table 15 · p. 23
D.6 UNCERTAINTY AND AGGREGATION
- Neologism and prompt-tuning visual control change little on average from ID to held-out evaluation, whereas LoRA loses more of its visual advantage.These macro-average results do not imply uniform behavior across benchmarks; the analysis uses paired exact McNemar tests and item-level bootstrap intervals.Source: §D.6 · p. 23
E REAL-WORLD NEWS-FRAME EVALUATION
The real-world evaluation tests modality control on news-video frames using visual- and description-swap pairs, with an additional crop around human-annotated visual evidence. On full frames, visual accuracy exceeds FLIP for many methods and models, underscoring that correct answers do not ensure reliable response to changed evidence.
E REAL-WORLD NEWS-FRAME EVALUATION
- Evidence crops enlarge a human-annotated visual region by 25% without crop-specific training, testing access to evidence rather than the ability to find it.The evaluation uses separate checkpoints trained directly for this setting, and none of its videos appears in the learned methods’ training set.Source: E REAL-WORLD NEWS-FRAME EVALUATION · p. 23
- At full frame, visual Accuracy is .50–.66 across methods and models, while FLIP is often substantially lower.The evaluation uses 173 visual-swap pairs and 300 description-swap pairs; two independent uniform guesses yield an expected FLIP of .25, not a general chance threshold.Source: E REAL-WORLD NEWS-FRAME EVALUATION · p. 23, Table 17 · p. 24
F RESULT VERIFICATION
- Reported tables are generated from canonical item-level results and checked for fixed candidate sets, preserved conflicts, and reproducible aggregate scores.Source: F RESULT VERIFICATION · p. 23