PaperlayerView exact revision on arXiv

Source-linked AI summary

Is Visual Prompting All You Need? Studying VLM Spatial Reasoning under Progressive Visual Scaffolds

Lars Benedikt Kaesberg, Tianyu Yang, Florian Valentin Wunderlich, Terry Ruas, Jan Philip Wahle, Daniel Kurzawe, Bela Gipp

arXiv:2608.21170v1cs.CVcs.AI

TL;DR

It remains unclear how visual presentation shapes VLM performance when the reasoning task is unchanged. The paper tests progressive visual scaffolds on SPaRC, finding gains of up to 34.0 percentage points over the original visual format and more effective GRPO post-training.

  • Problem

    How visual presentation affects performance when the reasoning task is unchanged remains unclear, while end-to-end accuracy conflates perception with reasoning.

    Source: §1 · p. 1, §2.1 · p. 2

  • Method

    The paper introduces a hierarchy of lightweight visual scaffolds for SPaRC that progressively increases perceptual support while preserving the same board layout, rules, and target solution.

    Source: §1 · p. 2, §5 · p. 9

  • Results

    Up to 34.0 percentage points over the original visual board, scaffolds improve accuracy across multiple VLMs and make GRPO post-training more effective, while mainly reducing grounding errors as rule satisfaction remains challenging.

    Source: §5 · p. 9

  • Takeaways & Limitations

    Visual presentation can change both what VLMs solve and what their failures measure.

    Source: §5 · p. 9

    Empirical claims are restricted to SPaRC, and the hand-designed scaffolds are diagnostic interventions rather than an established general prompting strategy.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Vision-language models (VLMs) have advanced rapidly in multimodal reasoning, yet recent work shows that their failures often reflect an interaction between visual grounding and downstream reasoning. What remains less clear is how the visual presentation of a task shapes model performance and failure modes when the underlying reasoning problem is unchanged. We study this question in SPaRC, a benchmark for grid-based visual spatial planning, by introducing lightweight input-side scaffolds that preserve the visual modality while making spatial structure more accessible. Across multiple VLMs, these scaffolds improve task accuracy over the original visual setting by up to 34.0 percentage points and further complement GRPO-based training, yielding up to 4.6 additional accuracy points compared with near-zero gains on the original visual input. Analyses on both end-to-end task solving and object detection show that these gains are closely tied to reductions in grounding-related errors, while rule reasoning remains comparatively challenging. We find that visual presentation is a central factor that determines whether VLM benchmarks measure grounded perception, downstream reasoning, or a mixture of both.

1 Introduction

With the planning problem held fixed, visual presentation changes SPaRC accuracy and failure modes, showing that input-side perceptual support materially affects measured VLM planning performance.

1 Introduction

  • The study introduces progressively stronger visual scaffolds for SPaRC while keeping the board layout, rules, and target solution fixed.Scaffolds range from the original puzzle image to explicit endpoints, coordinate grids, cell-level coordinates, and text-rendered symbols.Source: §1 · p. 2
  • End-to-end accuracy, board-element detection, and path-error analyses show that scaffolding separates grounding errors from residual reasoning challenges.The evaluated path errors include endpoint grounding, connectivity, self-intersection, and rule satisfaction; visual presentation changes the distribution of failures with the planning problem held fixed.Source: §1 · p. 2
  • Up to 34.0 percentage points of accuracy gains over the original visual format came from stronger input-side perceptual support without fine-tuning.The gains show that the input representation accounts for a substantial share of SPaRC difficulty, including for frontier-scale models.Source: §1 · p. 2
  • GRPO fine-tuning yields larger gains with scaffolded visual inputs than with original visual input or the text-formulation baseline.Endpoint localization, coordinate grounding, and invalid path construction improve substantially, while rule satisfaction remains comparatively challenging.Source: §1 · p. 2

2 Related Work

VLM benchmark accuracy can conflate visual perception with reasoning, especially in spatial planning. This work varies the visual representation while keeping the planning task fixed to examine that ambiguity.

2.1 Multimodal Reasoning in VLMs

  • VLM benchmark accuracy conflates perception with reasoning, a problem especially acute in visual spatial planning when failures may arise from either stage (Wang et al., 2026).The study addresses this ambiguity by changing only visual representation while holding the planning task fixed.Source: §2.1 · p. 2

2.2 Visual Spatial Planning and Representation Design

  • Image-based spatial planning underperforms text descriptions; o4-mini scores 5.6% on SPaRC visual input versus 15.8% on text (Kaesberg et al., 2025).Prior mitigations include fine-tuning or merging, inference-time visual tools, and visual scaffolds; this work is closest to scaffolding but uses single grounding scaffolds to examine which perceptual factors support reasoning.Source: §2.2 · p. 3
  • The scaffolds progressively expose endpoint identity, spatial indexing, and symbol semantics while preserving the planning task, allowing analysis of what closes the vision-text gap.Unlike task-agnostic CoordConv coordinate channels, the Axis Labels and Cell Coordinates scaffolds render coordinates into images and use SPaRC’s known structure (§3.2).Source: §2.2 · p. 3

3 Methodology

The methodology uses SPaRC to test how visual scaffolds and image degradations affect grid-based spatial planning, evaluating both path solving and rule-symbol recognition. The task stays fixed while the board’s presentation changes.

3.1 SPaRC as a Testbed

  • SPaRC contains 1,000 grid-planning examples (500 training, 500 testing) across five difficulty levels; models must produce a valid continuous coordinate path from a designated start to exit.The benchmark uses 2D grids populated with rule cells, a start node, and an end node.Source: §3.1 · p. 3

3.2 Input-Side Visual Scaffolding

  • A progressive visual-scaffold hierarchy runs from the original board through endpoint markers, axis labels, cell coordinates, and text rule labels, while keeping the planning task unchanged.The cumulative, task-specific variants are used to examine how error profiles change as perceptual burden falls; text labels shift some grounding burden to reading in-image text.Source: §3.2 · pp. 3–4, Figure 2 · p. 4

3.3 Visual Degradation and Recovery

  • Low Contrast, Low Resolution, and board rotation degrade the original image; overlaying Cell Coordinates tests whether spatial cues recover performance without textualizing rule symbols.The degradations reduce symbol distinctiveness, fine pixel detail, or canonical orientation, respectively.Source: §3.3 · p. 4

3.4 Evaluation Tasks and Metrics

  • Evaluation pairs path solving with independent rule-symbol detection to measure planning behavior and the visual grounding that supports it.Path metrics include Final Task Accuracy, Correct Start/End, Connected Path, Non-Intersecting, No Rule Violation, and Fully Valid. Detection metrics are Exact-Match Board Accuracy, Average Rule Detection Accuracy, and Per-Type Detection Accuracy; comparing Cell Coordinates with Text Symbols separates local spatial-grounding gains from explicit symbol semantics.Source: §3.4 · pp. 4–5

4 Results and Analysis

Across SPaRC results, visual scaffolds improve task-solving accuracy and perceptual grounding, but harder planning and orientation-sensitive alignment remain challenging. Text Symbols also enables larger GRPO gains than naive visual input.

4.1 Experimental Setup

  • The evaluation covers seven open-weight VLMs spanning 24B to 397B parameters and tests GRPO on Qwen 3 VL 4B and 8B Thinking, using a 500-puzzle test split.Inference uses greedy decoding unless stated otherwise.Source: §4.1 · p. 5

4.2 Effect of Visual Scaffolding on Task Solving

  • Qwen 3.5 397B reaches 48.4% accuracy with Text Symbols versus 14.4% on the Original board, and exceeds the text baseline by +10.4 pp.Scaffolding also leaves the two accuracy-floor models, Gemma 3 27B and Mistral Small 3.2, below 1% across visual conditions, with no resolvable gain.Source: §4.2 · pp. 5–6, Figure 3 · p. 6

4.3 Effect of Visual Scaffolding on Perceptual Recognition

  • Gemma 4 31B’s average rule accuracy rises from 85.4% on the Original board to 97.0% with Text Symbols, while Qwen 3.5 397B reaches 100% detection for Start, End, Path, and Gap.Scaffolding is not uniformly beneficial: some coordinate-based variants reduce detection accuracy, consistent with added visual clutter or distribution shifts.Source: §4.3 · p. 6, Figure 5 · p. 7, Figure 6 · p. 7
  • Object-detection performance strongly correlates with task-solving accuracy across models and board representations (r = 0.96, R2 = 0.92, p < 0.001), though this does not show perception alone determines success.The correlation using Average Rule Detection Accuracy is weaker but remains significant (r = 0.51).Source: Figure 7 · p. 7, §4.3 · p. 7

4.4 Path-Level Error Analysis

  • Fully Valid paths rise from 8.9% on the Original board to roughly 48% with Cell Coordinates or Text Symbols, chiefly through improved endpoint correctness and connectivity.Correct Start/End increases from 11.8% to around 90%, while Connected Path moves from 68.2% to 76.3%; non-intersection and rule-violation metrics show no clear scaffold-related improvement.Source: §4.4 · pp. 7–8, Figure 8 · p. 8

4.5 Recovery from Visual Degradation

  • Under Cell Coordinates, low contrast and low resolution have limited effects, but rotation causes larger accuracy drops: −9.0 points for Qwen 3.5 397B and −3.8 for Gemma 4 31B.Rule detection shows a similar pattern, supporting recovery of impaired visual grounding while leaving orientation-sensitive alignment errors unresolved.Source: Figure 9 · p. 8, §4.5 · p. 8

4.6 Effect on GRPO Post-Training

  • On Text Symbols, GRPO improves accuracy by +3.8 points for 4B (1.4% →5.2%) and +4.6 points for 8B (1.2% →5.8%), exceeding the corresponding text-baseline gains.On the unannotated Original image, accuracy instead changes from 1.4% to 0.8% for 4B and 0.2% to 0.4% for 8B. Only the Text Symbols gains have disjoint pre- and post-training intervals.Source: Figure 10 · p. 8, §4.6 · p. 9

5 Conclusion

On SPaRC, lightweight visual scaffolds improve VLM task-solving accuracy while primarily reducing grounding errors; the findings motivate evaluating both the reasoning task and the perceptual interface.

5 Conclusion

  • Up to 34.0 percentage points, lightweight visual scaffolds improve SPaRC task-solving accuracy over the original visual board and make GRPO post-training more effective.The scaffolds preserve the board layout, rule configuration, and target solution while progressively increasing perceptual support.Source: §5 · p. 9
  • Object-detection and path-level validity analyses indicate scaffolding mainly reduces grounding-related errors, while rule satisfaction remains comparatively challenging.Visual scaffolding separates grounding-sensitive errors from residual reasoning challenges.Source: §5 · p. 9
  • Visual presentation can change both what VLMs can solve and what their failures measure, so multimodal-planning benchmarks should control the reasoning problem and its perceptual interface.Source: §5 · p. 9
  • The empirical claims are restricted to SPaRC; transfer of the gains or scaffolds to natural images, robotics, or document understanding is not established.Source: §5 · p. 9
  • The portable experimental logic is to intervene on the input while holding the task fixed; residual errors motivate layout-aware encoders and planning-aware post-training or search-based decoding.Encoders target grounding-sensitive errors, while post-training or search-based decoding target persistent rule-satisfaction errors.Source: §5 · p. 9

Limitations … B.1 Rules

The appendix bounds the study’s claims to SPaRC and its evaluated open-weight VLMs, while documenting limitations in scaffold design, model coverage, training scale, and statistical resolution. It also describes the hardware, dataset, and puzzle rules underlying the evaluation.

Limitations

  • The scaffolds encode known SPaRC requirements, so their gains are partly by construction; the authors treat them as diagnostic interventions and restrict empirical claims to SPaRC.Because scaffolds are cumulative rather than orthogonal, the hierarchy does not separate perception from planning; the study also leaves minimal sufficiency and automatic construction open.Source: Limitations · p. 9
  • The seven-model pool spans 24B–397B open-weight VLMs, but excludes closed frontier systems; 4-bit quantization may lower absolute accuracy for Qwen 3.5 397B and GLM 4.6V.Quantization is held constant within each model, so it cannot explain the reported 34.0-percentage-point scaffolding jump on Qwen 3.5 397B.Source: Limitations · pp. 9–10
  • GRPO was tested only on Qwen 3 VL 4B and 8B Thinking, leaving scaling to larger reasoning models uncharacterized; gains were near zero on Original versus up to 4.6 points on Text Symbols.The comparison is central to the claim that perceptual access constrains what RL post-training can teach.Source: Limitations · p. 10
  • Accuracies use 500 puzzles; bootstrap intervals resolve the main scaffolding gains but not smaller effects, including GRPO gains on the text baseline or gains at the accuracy floor.The unresolved effects are described in Appendix E.1 and §4.6.Source: Limitations · p. 10

A Model and Hardware Details

  • Evaluation used seven VLMs on 4 NVIDIA A100 80GB GPUs, with quantization for the two largest models; Qwen 3.5 397B consumed about 720 A100-hours versus about 4 for Llama 4 Scout.The inference-cost figures exclude GRPO training, which is accounted for separately.Source: A Model and Hardware Details · pp. 12–13

B SPaRC Dataset

  • SPaRC contains 1,000 grid pathfinding puzzles, split into 500 training and 500 test puzzles, with a continuous non-self-intersecting path required to satisfy each puzzle’s rules.The grid uses top-left origin coordinates, and each puzzle has one start and one end point.Source: B SPaRC Dataset · p. 13

B.1 Rules

  • SPaRC rules require paths to collect every dot, avoid gaps, separate differently colored stones, and pair each star with exactly one same-color symbol.Source: §B.1 · p. 13
  • Other rules require touching the triangle-specified number of cell edges, enclosing exact non-rotated or mirrored polyomino shapes, and subtracting ylop shapes from polys in the same region.Multiple polys may share a region if they fit without overlap; an exactly canceled poly and ylop impose no constraint.Source: §B.1 · p. 13

B.2 Dataset Creation

The dataset uses randomly generated grid puzzles and a validation loop that adjusts rule density to control the number of solutions. Puzzle difficulty is estimated on a five-level scale using rule and grid characteristics.

B.2 Dataset Creation

  • Puzzles use random 2–6 by 2–6 grids, roughly half-filled with rules, and random start/end points; brute-force validation adjusts density until puzzles have 1–50 solutions.If a puzzle has no solution, rule density is decreased; if it has more than 50 solutions, density is increased and the puzzle regenerated.Source: §B.2 · pp. 13–14
  • Difficulty combines rule types, rule cells, rule density, grid size, and estimated rule interactions into levels 1–5; the test set has 86, 118, 121, 86, and 89 puzzles at levels 1–5, respectively.Source: §B.2 · p. 14

C Comparison with the SPaRC Text Baseline

On SPaRC, scaffolded visual inputs outperform text baselines for several stronger VLMs, while the text baseline remains difficult for models overall compared with human solvers.

C Comparison with the SPaRC Text Baseline

  • Humans reach 98% on the text baseline, versus 38.0% for the strongest VLM, Qwen 3.5 397B; four of seven VLMs score below 10%.The human-model gap shows that SPaRC remains non-trivial under purely textual presentation.Source: C Comparison with the SPaRC Text… · p. 14
  • Scaffolded visuals beat text baselines by +10.4 pp for Qwen 3.5 397B (38.0% to 48.4%) and +9.6 pp for Gemma 4 31B (16.6% to 26.2%).Qwen 3.5 27B improves from 33.9% to 40.0% (+6.1 pp), and GLM 4.6V from 8.4% to 12.8% (+4.4 pp).Source: C Comparison with the SPaRC Text… · p. 14
  • The three weakest models stay within ±1 pp of their text baselines near the floor, while visual-over-text gains scale with model capability.This matches the scaffolding-gain pattern reported in §4.2.Source: C Comparison with the SPaRC Text… · p. 14

D Recovery Variants · E Additional Experiments · E.1 Bootstrap Confidence Intervals

Recovery variants add explicit Cell Coordinates to degraded boards while leaving the task and degradation parameters unchanged; bootstrap intervals show resolvable scaffold gains for five models above the accuracy floor, but not for two models below 1% on every visual condition.

E Additional Experiments

  • The text-formulation baseline reports accuracy for human solvers from Kaesberg et al. (2025) and the seven evaluated VLMs.Figure 11 identifies the evaluated groups but provides no accuracy values in the supplied caption.Source: Figure 11 · p. 14

D Recovery Variants

  • Recovery variants overlay Cell Coordinates on degraded boards while keeping the planning task and degradation parameters unchanged.This isolates explicit spatial indexing from changes to the visual signal.Source: D Recovery Variants · p. 14

E.1 Bootstrap Confidence Intervals

  • Scaffold gains are resolvable for five models above the accuracy floor, while Gemma 3 27B and Mistral Small 3.2 remain below 1% on every visual condition with no resolvable gain.The intervals use 500 test puzzles and 200,000 bootstrap replicates.Source: §E.1 · p. 14
  • Accuracies are computed over 500 test puzzles, with 95% bootstrap percentile intervals from resampling puzzle IDs with replacement for 200,000 replicates using seed 20260821.Table 2 covers every model and condition, and Table 4 covers GRPO runs.Source: §E.1 · p. 14
  • Differences to each model’s best scaffold use paired resamples, evaluating both conditions on the same resampled puzzle IDs.Table 3 reports these differences.Source: §E.1 · p. 14

D Recovery Variants

  • Figure 12 presents low-contrast, low-resolution, and rotation degradations alongside their Cell Coordinates recovery counterparts.The recovery boards retain the degradation and add the coordinate overlay.Source: D Recovery Variants · p. 14, Figure 12 · p. 15

E.2 Grid Lines versus Coordinate Labels

The ablation shows that explicit coordinate labels, rather than generic visual modification alone, support task-solving gains. A blank label margin disrupts cell indexing, while labels act as spatial anchors.

E.2 Grid Lines versus Coordinate Labels

  • On task-solving accuracy, grid lines alone add +1.4, +0.2, and +2.6 pp over Original, while full Axis Labels exceeds Original by +2.4 to +7.6 pp.The comparison separates grid lines alone, grid lines with a blank label margin, and the full Axis Labels scaffold across the same 500 test puzzles.Source: §E.2 · p. 15
  • A reserved blank margin falls below Original for all three models and disrupts cell indexing, while explicit labels anchor coordinates, especially for weaker models.The proposed explanation is that models treat the blank band as part of the board and offset their indices accordingly.Source: §E.2 · p. 15
  • ±1 coordinate-offset rates are 4–5% with explicit labels, 17–46% with a blank margin, and 6–14% for Qwen grid-only variants, while Gemma 4 31B remains at 35%.Source: §E.2 · p. 15

E.3 GRPO Post-Training · E.4 Averaged Confusion Matrix · E.5 Correlation with Average Rule Detection Accuracy

Average Rule Detection Accuracy correlates positively with task accuracy, but less strongly than Exact-Match Board Accuracy; across models, Text Symbols sharpen symbol-class detection, and GRPO is evaluated under matched visual and text input conditions.

E.3 GRPO Post-Training

  • GRPO was compared across the SPaRC text baseline, Original visual board, and Text Symbols scaffold, with each model trained and evaluated using the same representation.This matched-input design tests how perceptual access affects the training signal.Source: §E.3 · p. 15
  • GRPO trains Qwen 3 VL 4B and 8B Thinking on 500 SPaRC training examples, rewards paths satisfying all validity conditions, and evaluates on the held-out test split.The study uses these open-weight reasoning models because they fit the available training hardware budget.Source: §E.3 · p. 15

E.4 Averaged Confusion Matrix

  • On the Original board, diagonal correctness falls to 0.23 for End, 0.29 for Negative Polyshape, and 0.37 for Dot, with most errors assigned to Path.These per-type confusion patterns are averaged across all seven evaluated VLMs.Source: §E.4 · p. 16, Figure 13 · p. 17
  • Text Symbols produces a sharper detection diagonal across models, although several rule types retain 0.20–0.34 of cells labeled Path or Empty.The reported dominant gain comes from disambiguating symbol classes, while weaker models can still default to background classes.Source: §E.4 · p. 16, Figure 13 · p. 17

E.5 Correlation with Average Rule Detection Accuracy

  • Average Rule Detection Accuracy correlates with end-to-end task accuracy (r = 0.51, R2 = 0.26, p < 0.001), weaker than Exact-Match Board Accuracy (r = 0.96).The comparison uses Average Rule Detection Accuracy as the perception metric alongside the exact-match correlation reported in §4.3.Source: §E.5 · pp. 16–17
  • Exact board-level perception aligns more tightly with task success because solving requires every task-relevant element, whereas per-rule averages can stay high despite critical symbol errors.Average Rule Detection Accuracy is dominated by the many Empty and Path cells.Source: §E.5 · p. 17

E.6 Token Usage

Completion-token budgets vary widely across models and scale with difficulty only for some; scaffolding benefits both long- and short-reasoning models, indicating that improved perceptual access helps but does not replace reasoning capacity.

E.6 Token Usage

  • Scaffolding gains were +31.8 pp for Qwen 3.5 27B and +34.0 pp for Qwen 3.5 397B, while Gemma 4 31B gained +24.0 pp below ∼4k tokens.The largest-budget models still struggle to overcome grounding difficulty on the Original board; short responses can also benefit from improved visual representations.Source: §E.6 · p. 17
  • Perceptual access acts upstream of inference-time compute: scaffolding helps long- and short-reasoning models, but does not replace reasoning capacity.These patterns are drawn from scaffolding results across models with different completion budgets.Source: §E.6 · pp. 17–18
  • Qwen 3.5 completion lengths rise with difficulty (27B: ∼31k–∼50k; 397B: ∼24k–∼36k), while GLM 4.6V stays ∼14k–17k and four models remain below ∼6k tokens.Figure 15 reports average completion tokens per puzzle by difficulty on the Original visual input.Source: §E.6 · p. 17, Figure 15 · p. 18

E.7 Object Detection under Visual Degradation … CITATION SHEET

This section covers visual-prompting evaluations, including rotation-related rule detection with overlaid cell coordinates and prompts ranging from image-grounded solving to perception-only evaluation, alongside administrative AI-usage and citation materials.

E.7 Object Detection under Visual Degradation

  • Full Board Rule Detection Accuracy fell 33.4 pp for Qwen 3.5 397B and 16.2 pp for Gemma 4 31B with Cell Coordinates under rotation; original-board low-contrast and low-resolution changes were all |∆| ≤6.2 percentage points.The Qwen 3.5 397B rule-detection drop under rotation (−33.4 pp) exceeded its end-to-end drop (−9.0 pp), suggesting geometric transformations particularly disrupt scaffolded rule perception; the overlay can become misaligned and add clutter.Source: Figure 16 · p. 18, §E.7 · p. 18

F.1 Default Prompt

  • The default prompt is a compact image-grounded baseline that describes the goal, coordinate system, and rules, then requests the solution as a list of (x,y) coordinates.A textual puzzle rendering is included only when the textual-representation flag is enabled.Source: F Prompts · p. 18, §F.1 · p. 19

F.2 Improved Prompt

  • The improved prompt adds explicit path-cell, rule-cell, and region definitions, clarifies coordinate parity, and includes a worked region example.It also distinguishes positive and negative polyshapes and specifies the output format.Source: F Prompts · p. 18, §F.2 · p. 19

F.3 Object-Detection Prompt

  • The object-detection prompt replaces solving with reconstructing a symbolic grid from the image, enabling perception accuracy to be measured separately from path-finding.It requests a 2D symbol array and embeds a symbol legend and color codes.Source: F Prompts · p. 18, §F.3 · p. 20

G AI-Usage Card

  • The AI-usage card asks about harm-minimization measures and states that corresponding authors verify and agree with modifications or generations of AI-generated content.The card is identified as AI Usage Card v2.6 (Wahle et al., 2023).Source: G AI-Usage Card · p. 22

CITATION SHEET

  • The citation sheet lists the paper’s authors, title, EMNLP 2026 venue, publisher, and August 24, 2026 generation date.It states that the sheet was generated with citeassist.uni-goettingen.de.Source: CITATION SHEET · p. 23

Related papers

Loading 2608.21170v1…