PaperlayerView exact revision on arXiv

Source-linked AI summary

Diagnosing the Reference Gap: What Visual Scaffolds Can and Cannot Fix in Multimodal Reasoning

Guizhen Chen, Weiwen Xu, Wotao Yin, Anh Tuan Luu, Hou Pong Chan

arXiv:2610.39419v1cs.CVcs.CL

TL;DR

MLLMs can recognize visual content yet struggle to maintain stable references to it during multi-step reasoning, a problem often approached as perception. The paper introduces a paired benchmark that varies visual references while fixing the scene, question, and answer, and tests a training-free skill for selecting references. Task-matched references improve performance across five tasks, while mismatched references may provide little benefit or harm performance.

  • Problem

    MLLMs can recognize visual content yet struggle to maintain stable links to relevant visual entities, regions, or locations across multi-step reasoning.

    Source: §1 · p. 1

  • Method

    ReGap-Bench pairs reference conditions across five visual reasoning tasks while keeping the scene, question, and answer fixed, and the paper tests a training-free skill for dynamically selecting references.

    Source: §1 · p. 2

  • Results

    Task-matched references improve performance across all five tasks, with numbering gains reaching 25% on clock ranking; mismatched references provide little benefit or can harm performance.

    Source: §1 · p. 2, §7 · p. 10

  • Takeaways & Limitations

    The results identify the reference gap as distinct from visual perception, and the training-free skill demonstrates dynamic reference construction at inference time without additional model training.

    Source: §7 · p. 10

Abstract

from arXiv · show

Multimodal large language models can recognize visual content yet struggle to reason reliably over it across multiple steps. In counting, a model may see all objects but lose track of which ones it has counted. We argue that such failures, commonly attributed to perception, reveal a reference gap: the failure to bind language to the relevant visual entities, regions, or locations. Motivated by the theory of visual routines, we test whether external references such as numbering can narrow this gap. We introduce ReGap-Bench, a paired benchmark of five visual reasoning tasks in which the scene, question, and answer are fixed and only the reference varies. Results show that task-matched references improve accuracy by up to 25\%, while mismatched references provide little benefit or even hurt performance. These findings identify the reference gap as a bottleneck distinct from visual perception and show that constructing task-appropriate visual references can serve as a training-free visual skill for improving multimodal reasoning.

1 INTRODUCTION

MLLMs can recognize visual content yet struggle to maintain references to it across multi-step reasoning, a gap distinct from perception. The paper introduces a paired benchmark and a training-free visual skill to test task-matched references, which improve performance across five tasks.

1 INTRODUCTION

  • Prior approaches do not isolate the effect of the reference or establish whether one reference suits every task.The paper notes that prior work changed training or varied images and prompts together, and applied the same reference across tasks.Source: §1 · p. 1
  • The reference gap arises when models fail to maintain stable links between language-based reasoning and relevant visual entities, regions, or locations despite recognizing scene content.In the example, numbering lets the model list ellipses by label rather than use vague locations and double-count.Source: §1 · p. 1, Figure 1 · p. 2
  • Task-matched references improve accuracy on all five tasks, with numbering gains reaching 25% on clock ranking; mismatched references provide little benefit or can hurt performance.The paper also reports that the reference gap persists across Qwen3-VL models from 2B to 235B parameters.Source: §1 · p. 2
  • The proposed training-free visual skill dynamically identifies a task’s visual routine and constructs a corresponding reference at inference time.On CartoMapQA, the skill improves complex map understanding without finetuning.Source: §1 · p. 2
  • ReGap-Bench pairs reference conditions across five tasks while keeping scene semantics, questions, and answers fixed.The evaluated references include numbering, shading, and grid lines.Source: §1 · p. 2

2 DIAGNOSING THE REFERENCE GAP WITH VISUAL INTERVENTIONS

The section frames the reference gap as failure to maintain links between language and visual content during reasoning, rather than necessarily a failure of perception. It proposes task-matched visual scaffolds and a paired intervention that varies references while holding the underlying problem fixed.

2.1 THE REFERENCE GAP

  • Most benchmarks do not distinguish a perception failure from failure to maintain reference during reasoning; the reference gap concerns losing stable links to visual entities, regions, or locations.In counting, this can cause omissions or duplicates when the model loses track of processed objects.Source: §2.1 · pp. 2–3

2.2 EXTERNAL REFERENCES FOR VISUAL ROUTINES

  • Visual routines motivate three reference types: numbering supports indexing and marking, shading makes bounded regions explicit, and grids support focus shifts and boundary tracing.The scaffolds provide different reference information, so the authors hypothesize that a reference helps most when it matches the visual routines a task requires.Source: §2.2 · p. 3

2.3 PAIRED REFERENCE-INTERVENTION DESIGN

  • The paired intervention renders the same underlying scene with different references, preserving the scene content, question, and answer while preventing the reference from revealing the answer.Figure 2 illustrates the unchanged scene and question across reference conditions.Source: §2.3 · p. 3, Figure 2 · p. 4
  • The average reference effect compares correctness under a reference condition with correctness in the plain condition; positive values suggest the reference helps solve more of the same problems.A near-zero effect may indicate no added benefit or errors elsewhere, while a negative effect may indicate interference such as clutter.Source: §2.3 · pp. 3–4

3 REGAP-BENCH: A REFERENCE-INTERVENTION BENCHMARK

ReGap-Bench applies paired reference interventions to five visual reasoning tasks, using scenes rendered with and without added references. The tasks are selected to probe different visual routines, and evaluation uses exact match alongside pairwise accuracy for ranking tasks.

3 REGAP-BENCH: A REFERENCE-INTERVENTION BENCHMARK

  • ReGap-Bench contains five tasks with 200 scenes each, rendered under reference conditions that add numbers, gray fills, or faint grid lines without covering content or revealing answers.The benchmark applies the paired design to shape counting, angle ranking, size ranking, clock ranking, and transformation prediction.Source: §3 · p. 4

3.1 TASKS

  • The five tasks probe distinct routines: counting uses indexing and marking; angle ranking uses bounded activation; size ranking uses area or scale references; clock ranking uses a circular frame; transformations use position and scale.The authors identify the reference expected to match each routine: numbering for counting and clock positions, shading for angles and shape area, and grids for size comparisons and transformations.Source: §3.1 · pp. 4–5

3.2 EVALUATION SETUP

  • Evaluation covers five proprietary and open-source MLLMs, with six additional Qwen3-VL Thinking scales from 2B to 235B parameters; the main metric is exact match.Ranking tasks also use pairwise accuracy, the share of predicted pairs in the correct order.Source: §3.2 · p. 5

4 EXPERIMENT

Task-matched visual references improve reasoning, while mismatched scaffolds can be neutral or harmful. Numbering, shading, and grids support different tasks, and numbering continues to help as model scale increases.

4.1 TASK-MATCHED REFERENCES IMPROVE VISUAL REASONING

  • Numbering yields gains of 6.0%–17.5% on shape counting and 8.5%–25.0% on clock ranking.With numbering, 100% of Gemini’s and 86.0% of GPT’s responses refer to printed numbers; without it, GPT uses systematic references in only 15.5% of responses.Source: §4.1 · pp. 5–6
  • Shading raises size-ranking accuracy from 4.0% to 11.0% and improves angle-ranking pairwise accuracy for four of five models.For angle ranking, pairwise accuracy rises from 76.4% to 85.2% for Gemini and from 59.7% to 68.0% for Qwen3-VL.Source: Figure 4 · p. 6, §4.1 · p. 6
  • For GPT, numbering primarily improves clock-reading completeness: clocks without a stated time fall from 34% to 12%, while stated-time accuracy changes from 90.3% to 90.8%.The clock-reading measure counts a stated time as correct when it is within five minutes of ground truth.Source: §4.1 · p. 6
  • For size ranking, pairs with scale differences below 0.10 are ordered correctly only 67.5%–82.2% of the time, versus at least 97.5% when differences are 0.20 or more.Shading helps most on close pairs, suggesting its exact-match gains primarily reflect fewer swaps between similarly sized shapes.Source: Table 1 · p. 6, §4.1 · p. 6
  • Grid lines yield gains of 2.0%–8.0% on size ranking and 2.0%–5.5% on transformation prediction, with smaller size-ranking gains than shading in four models.Grid-based measurement requires models to measure shapes in cells, retain readings, and compare them.Source: Figure 5 · p. 7, §4.1 · p. 7

4.2 ANALYSIS ON MATCHED VERSUS MISMATCHED REFERENCES

  • In shape counting, numbering improves accuracy by 11.3% on average, versus 1.7% for shading, while grids reduce it by 4.1%.Across shape counting and clock ranking, operation-matched scaffolds produce the largest gains; added visual marks alone do not ensure improvement.Source: Figure 6 · p. 7, §4.2 · p. 8

4.1 TASK-MATCHED REFERENCES IMPROVE VISUAL REASONING

  • Shading nearly halves wrong-side angle readings for Claude (3.3% to 1.7%), Gemini (18.4% to 8.5%), and InternVL (23.9% to 13.1%).GPT-5.4-mini shows no such reduction (13.3% vs. 13.8%), consistent with nearly unchanged pairwise accuracy.Source: §4.1 · pp. 6–7

4.4 ANALYSIS OF MODEL SCALE

  • Across six Qwen3-VL models from 2B to 235B parameters, numbering retains a shape-counting advantage of 9.0%–23.5% that does not decline monotonically with scale.Figure 7 also reports improving performance as dial numbering becomes more complete.Source: §4.4 · p. 8, Figure 7 · p. 8

5 TASK-MATCHED REFERENCES AS TRAINING-FREE VISUAL SKILLS

The paper packages task-matched visual references as a training-free skill that an agent can use when a task requires them. On CartoMapQA, the skill improves mean score while selecting overlays for only some tasks.

5 TASK-MATCHED REFERENCES AS TRAINING-FREE VISUAL SKILLS

  • Increasing model size helps both plain and numbered inputs but does not eliminate the reference gap.The passage does not specify model sizes or numerical effects.Source: Figure 8 · p. 8
  • The skill’s written rules match references to visual operations, with before-and-after examples and a script that overlays a copy of the map.Rules select numbering, calibrated grids, shading, or no overlay depending on the operation.Source: §5 · p. 9
  • All references improve mean score on CartoMapQA, and the agent draws overlays on 57 of 100 tasks.Table 2 reports visual-skill accuracy by overlay type.Source: §5 · p. 9
  • The selected references broadly align with each subtask’s visual operation, including numbering for counting and route navigation, grids for route length, and shading for region isolation.The agent uses no overlay for symbol-recognition and presence questions.Source: §5 · p. 9

6 RELATED WORK

Prior work studies visual perception, spatial reasoning, and visual aids, but does not isolate failures to maintain visual references during reasoning. This paper varies references while holding the scene, question, and answer fixed.

6 RELATED WORK

  • Existing perception and reasoning benchmarks do not distinguish failures to encode visual elements from failures to maintain references during reasoning.The paired design holds the scene, question, and answer fixed while varying the reference system.Source: §6 · p. 9
  • Prior work grounds reasoning with generated points and boxes, cognitive maps, or prompting strategies, while this study tests references supplied in the input image for fixed, off-the-shelf models.The cited approaches include trained coordinate generation and reasoning prompts or cognitive maps.Source: §6 · p. 9
  • Visual aids range from region identifiers and auxiliary drawings to modular computation; this study systematically tests rendered reference systems for task alignment and reusable, training-free skills.Examples include Set-of-Mark, Visual Sketchpad, and code-based vision modules.Source: §6 · p. 9

7 CONCLUSION

The paper identifies a reference gap in multi-step multimodal reasoning and finds that task-matched references improve performance, while mismatched references may not help. It also demonstrates inference-time construction of references with a training-free visual skill.

7 CONCLUSION

  • The reference gap is a distinct bottleneck: models may perceive relevant visual content yet fail to maintain stable references during multi-step reasoning.The conclusion says the gap persists at larger model scales and that mismatched references can be harmful.Source: §7 · p. 10

AI USE STATEMENT

  • The authors used generative AI to refine hypotheses, support qualitative analysis, edit code, improve readability, and identify literature, while reviewing the assisted work.They report no AI use for generating, cleaning, or reformatting datasets.Source: AI USE STATEMENT · p. 10

A PRELIMINARY CHECK: CAN MODELS PERCEIVE THE SHAPES?

In a preliminary check, Claude-Opus-4.7 recognizes shapes more accurately than it counts target-class shapes. The benchmark generates matched scene conditions from a shared specification.

A PRELIMINARY CHECK: CAN MODELS PERCEIVE THE SHAPES?

  • The preliminary probes used Claude-Opus-4.7 on plain shape-counting scenes, testing total-shape counts and single-shape class recognition.For the recognition probe, one shape was cropped from each scene and tested against the target class.Source: A PRELIMINARY CHECK: CAN MODELS… · p. 14
  • 96.5% of scenes were counted correctly when counting all shapes, and single-shape recognition reached 95.9%, versus 72.0% for target-class counting.The authors conclude that most counting errors do not come from missing or misreading individual shapes.Source: A PRELIMINARY CHECK: CAN MODELS… · p. 14, Table 3 · p. 14

B REGAP-BENCH DETAILS

  • Scenes were generated from a fixed random seed, with every reference condition sharing item positions, sizes, orientations, and a gold answer derived from the same specification.Source: B REGAP-BENCH DETAILS · p. 14

B.1 DATASET STATISTICS

ReGap-Bench combines generated scenes for five reasoning tasks with paired visual-reference conditions. Its dataset varies the reference while retaining the underlying task content.

B.1 DATASET STATISTICS

  • Each task has 200 scenes; the benchmark contains 1,000 scenes rendered as 3,800 images.Table 4 summarizes the benchmark statistics.Source: Table 4 · p. 14, §B.1 · p. 14

B.2 TASK GENERATION

  • Shape-counting scenes draw from 11 classes, with random orientations and sizes and class-inclusion rules determining the gold count.The target class can be absent, so the correct count may be zero.Source: §B.2 · pp. 14–15
  • Transformation-prediction scenes use a three-step chain of rotations, dilations, reflections, or translations, with four options generated from faulty transformations.The source and four options appear in a five-panel layout; the point P and any reflection line XY are shown in all panels.Source: §B.2 · p. 15

B.3 REFERENCE CONDITIONS

  • The benchmark varies references through object or dial numbering, gray shading, and grid lines, with grid spacing differing for transformation prediction.Transformation panels use lines every 30 px and translations of exactly two grid cells; the other tasks use lines every 15 px.Source: §B.3 · pp. 15–16

C EVALUATION DETAILS

Evaluation uses task-specific prompts and operational criteria for response references and ranking accuracy. It also defines procedures for measuring wrong-side angle readings and grid use.

C.1 PROMPTS

  • Paired reference conditions receive identical task-and-question prompt text, while shape-counting prompts encode category-inclusion rules and require a numerical count.Ranking prompts request ordered labels, while transformation prediction requests a single option letter.Source: §C.1 · pp. 16–17

C.2 REFERENCE-STRATEGY ANNOTATION

  • Responses are classified as systematic when they use object numbers or row-and-column references, and ambiguous when references are vague or absent.The annotation covers 800 shape-counting responses from Gemini-3.1-Flash-Lite and GPT-5.4-mini on plain and numbered images.Source: §C.2 · p. 17

C.3 PAIRWISE ACCURACY

  • Pairwise accuracy is the fraction of item pairs ordered as in the gold ranking, averaged over valid rankings.A ranking is invalid if it cannot be parsed or omits or repeats an item label.Source: §C.3 · p. 18

C.4 WRONG-SIDE READINGS IN ANGLE RANKING

  • Wrong-side angle readings are counted when a predicted position is closer to the angle’s conjugate-angle position than its true position.The procedure excludes angles whose correct and wrong-side readings imply the same rank and uses only scenes with valid plain and shaded rankings.Source: §C.4 · p. 18

C.5 GRID-USE JUDGMENT

  • Grid use is judged from whether responses give positions, distances, or sizes in grid units or coordinates, rather than merely mentioning the grid.The judge sees the question and response, but not the image, and quotes response text supporting positive decisions.Source: §C.5 · pp. 18–19

D OTHER RESULTS

The supplied material identifies two result tables for ranking tasks but does not provide their cell values.

D OTHER RESULTS

  • Table 5 reports pairwise accuracy for angle and size ranking, while Table 6 reports wrong-side readings in angle ranking.The wrong-side table’s comparison includes random rankings on the same scenes as a reference rate.Source: D OTHER RESULTS · p. 19, Table 5 · p. 19, Table 6 · p. 20
Loading 2610.39419v1…