Source-linked AI summary
Is Visual Prompting All You Need? Studying VLM Spatial Reasoning under Progressive Visual Scaffolds
Lars Benedikt Kaesberg, Tianyu Yang, Florian Valentin Wunderlich, Terry Ruas, Jan Philip Wahle, Daniel Kurzawe, Bela Gipp
TL;DR
The paper examines how visual presentation affects VLM performance and failure modes when the underlying spatial-planning problem is fixed. It introduces progressive visual scaffolds for SPaRC and finds that they improve accuracy by up to 34.0 percentage points, reduce grounding-related errors, and make GRPO post-training more effective.
Problem
It remains unclear how visual presentation changes VLM performance and failure modes when the underlying reasoning problem is unchanged.
Method
The paper evaluates progressive lightweight visual scaffolds on SPaRC while preserving board layout, rules, and target solution, using task-solving and object-detection analyses.
Results
Up to 34.0 percentage points of accuracy improvement over the original visual board accompanies reduced grounding-related errors, while rule satisfaction remains comparatively challenging.
Takeaways & Limitations
Visual presentation can change what VLM benchmarks measure, separating grounding-sensitive errors from residual reasoning challenges.
Takeaways & Limitations
The empirical claims are restricted to SPaRC, and the hand-designed cumulative scaffolds are diagnostic interventions rather than a general prompting strategy.
Abstract
from arXiv · showhide
Vision-language models (VLMs) have advanced rapidly in multimodal reasoning, yet recent work shows that their failures often reflect an interaction between visual grounding and downstream reasoning. What remains less clear is how the visual presentation of a task shapes model performance and failure modes when the underlying reasoning problem is unchanged. We study this question in SPaRC, a benchmark for grid-based visual spatial planning, by introducing lightweight input-side scaffolds that preserve the visual modality while making spatial structure more accessible. Across multiple VLMs, these scaffolds improve task accuracy over the original visual setting by up to 34.0 percentage points and further complement GRPO-based training, yielding up to 4.6 additional accuracy points compared with near-zero gains on the original visual input. Analyses on both end-to-end task solving and object detection show that these gains are closely tied to reductions in grounding-related errors, while rule reasoning remains comparatively challenging. We find that visual presentation is a central factor that determines whether VLM benchmarks measure grounded perception, downstream reasoning, or a mixture of both.
1 Introduction
The paper asks whether VLM planning failures arise from weak planning or from difficulty grounding visual structure. Using SPaRC and progressively stronger visual scaffolds, it shows that presentation changes both accuracy and failure modes.
- The central question is whether VLMs fail to plan or fail to see when solving visual spatial-planning tasks.
- SPaRC provides grid-based puzzles requiring models to perceive board elements and construct valid paths under explicit constraints.
- The study introduces lightweight scaffolds that progressively expose endpoints, coordinates, and rule-symbol semantics while preserving the underlying planning problem.
- Up to 34.0 percentage points of zero-shot improvement over the original visual format shows that input representation accounts for substantial benchmark difficulty.
- Scaffolding separates grounding errors from residual reasoning challenges, with visual presentation changing both performance and failure-mode distributions.
2 Related Work
Prior work shows that visual spatial-planning accuracy can conflate perception with reasoning. This paper addresses that ambiguity by progressively making spatial structure explicit while keeping the planning task fixed.
- VLM benchmark accuracy conflates perceiving visual input with reasoning over what was perceived, especially in visual spatial planning.
- Image-based SPaRC inputs underperform textual descriptions of the same environment, a gap often interpreted as a modality limitation.
- The paper intervenes on input representation by progressively exposing endpoint identity, spatial indexing, and symbol semantics without changing the planning task.
3 Methodology
The methodology uses SPaRC to test how progressively stronger visual scaffolds affect spatial planning while preserving the underlying task. It combines scaffolded and degraded board representations with end-to-end solving and object-detection evaluations to examine planning and grounding errors.
- Task Setting: SPaRC is a grid-based visual spatial-planning benchmark in which models generate valid coordinate paths between designated start and exit cells under rule constraints.The dataset contains 1,000 examples spanning five difficulty levels.
- Input Representations: The scaffold hierarchy progressively exposes endpoint identity, spatial indexing, cell localization, and rule-symbol semantics without changing the board layout or planning objective.The variants are cumulative, and later representations may affect multiple error types simultaneously.
- Input Representations: S/E Markers make endpoints explicit, while Axis Labels and Axis Labels + S/E add global coordinate references and combined endpoint support.These variants target endpoint-localization, coordinate-referencing, spatial-alignment, and related path-structure errors.
- Input Representations: Cell Coordinates print coordinates inside each cell, strengthening local spatial grounding for localization, coordinate binding, connectivity, and complete-path validity.Text Symbols extends this representation by replacing graphical rule symbols with text labels.
- Input Representations: Text Symbols makes symbol identity explicit and is intended to expose residual rule-grounding and constraint-reasoning errors after visual information becomes more accessible.This intervention shifts part of the grounding burden toward reading in-image text.
- Input Conditions: Low Contrast, Low Resolution, and Rotated settings degrade the original board image while preserving the same planning problem, probing sensitivity to visual grounding.The degradations respectively reduce color distinctiveness, fine-grained pixel detail, or canonical orientation.
- Evaluation: Evaluation combines end-to-end path generation with object detection to measure both final planning behavior and visual recognition supporting it.Task solving uses Final Task Accuracy and path-level validity conditions, while detection reports board-level, average rule, and per-type accuracy.
- Evaluation: Object detection is evaluated independently by parsing a fixed-schema 2D array of symbol codes and comparing predictions cell by cell with ground truth.Confusion matrices also track offdiagonal predictions for ground-truth Start cells.
4 Results and Analysis
Across SPaRC experiments, progressively scaffolded visual inputs improve task solving and object recognition, while harder planning and some representation-specific failures remain. The gains also extend to GRPO post-training, especially when visual inputs expose task-relevant structure.
- Task-solving: Input-side scaffolding consistently improves task-solving accuracy over the Original board for models above the accuracy floor.Qwen 3.5 397B rises from 14.4% to 48.4% with Text Symbols, while Gemma 3 27B and Mistral Small 3.2 remain below 1% across conditions.
- Task-solving: 48.4% with Text Symbols exceeds Qwen 3.5 397B’s 38.0% text baseline by +10.4 percentage points.Changing the input format alone produces this comparison, so the text formulation is not an upper bound on task reasoning.
- Task-solving: Accuracy decreases monotonically with puzzle difficulty under Text Symbols, with sharp drops at levels 4–5.Scaffolding helps most when the remaining planning problem is within model capacity; harder instances retain long-horizon planning challenges.
- Perceptual recognition: Scaffolded inputs generally improve rule recognition, although gains are not monotonic across every model and representation.Gemma 4 31B reaches 97.0% average rule accuracy with Text Symbols, versus 85.4% on the Original board; some axis-based variants reduce recognition.
- Perceptual recognition: Text Symbols sharply reduces Qwen 3.5 397B’s symbol confusions, raising Start, End, Path, and Gap detection to 100%.On the Original board, Gap is correctly detected only 21% of the time and is misclassified as Path in 65% of cases.
- Perceptual recognition: Object detection and end-to-end accuracy are strongly correlated across models and representations (r = 0.96, R2 = 0.92, p < 0.001).The authors caution that this correlation does not prove perception alone determines success; both measures may depend on model capacity.
- Path-level analysis: Scaffolding mainly improves endpoint and connectivity validity, while Non-Intersecting and No Rule Violation show little clear change.Fully Valid paths rise from 8.9% on the Original board to roughly 48% with Cell Coordinates and Text Symbols, whereas Connected Path moves from 68.2% to 76.3%.
- GRPO post-training: +4.6 points is the largest GRPO gain, achieved with scaffolded visual input on the 8B model versus near-zero gain on unannotated visual input.Scaffolding yields +3.8 points on 4B and +4.6 points on 8B, while unannotated visual input changes by −0.6 and +0.2 points, respectively.
5 Conclusion
Visual scaffolding changes both VLM performance and the failure modes measured in SPaRC, while persistent rule-satisfaction errors remain. The experimental claims are limited to SPaRC, but the intervention logic isolates how perceptual presentation affects evaluation.
- Scaffolds improve SPaRC task-solving accuracy by up to 34.0 percentage points over the original visual board and make GRPO post-training more effective.The comparison spans multiple VLMs and preserves the underlying planning problem.
- Object-detection and path-level validity analyses show that scaffolding mainly reduces grounding-related errors, while rule satisfaction remains comparatively challenging.The scaffold hierarchy separates grounding-sensitive failures from residual reasoning challenges.
- Visual presentation can change both what VLMs solve and what their failures measure.This means benchmark outcomes may reflect grounded perception, downstream reasoning, or their interaction.
- Future multimodal-planning benchmarks should control both the underlying reasoning problem and the perceptual interface through which it is presented.
- The empirical claims are restricted to SPaRC and do not establish transfer of the gains or scaffolds to natural images, robotics, or document understanding.The portable contribution is the experimental logic of intervening on input while holding the task fixed and observing error shifts.
Limitations
The study’s limitations constrain interpretation through hand-designed, cumulative scaffolds, benchmark scope, model coverage, quantization, training scale, and finite evaluation size.
- Scaffold design: Hand-designed scaffolds make improvements partly by construction and are diagnostic interventions rather than a general prompting strategy.The study does not test minimal sufficiency or automatic scaffold construction.
- Interpretation: Because scaffold steps are cumulative rather than orthogonal, the hierarchy cannot cleanly separate perception from planning.The study also counts in-image text reading as perception.
- Scope: All empirical results come from SPaRC, whose grid regularity enables controlled presentation changes but limits claims to this benchmark.The board layout, rule configuration, and target solution remain fixed while presentation varies.
- Model coverage: The model pool contains seven open-weight VLMs but excludes closed frontier systems that could respond differently to the scaffolding hierarchy.The evaluated models span 24B to 397B parameters and dense and Mixture-of-Experts architectures.
- Training scope: GRPO experiments cover only Qwen 3 VL 4B and 8B Thinking models, preventing direct characterization of reinforcement-learning scaling for the largest models.The observed pattern is consistent across the two trained sizes, with near-zero Original-board gains versus up to 4.6 points on Text Symbols.
- Statistical resolution: Accuracies use 500 puzzles, so smaller effects are not resolvable, including GRPO gains on the text baseline and gains for two floor-level models.The main scaffolding gains are resolvable against bootstrap intervals.
B.2 Dataset Creation
SPaRC puzzles are generated on small grids, validated for solvability and controlled solution counts, and assigned difficulty using structural complexity features.
- Puzzle generation: Puzzles are generated on m × n grids with m,n ∈ [2,6], approximately half the cells filled with rules, and random start and end points.A generation-validation loop checks candidate puzzles by brute-force search over valid paths.
- Puzzle validation: The validation loop decreases rule density when no solution exists and increases it when more than k=50 solutions exist.Brute-force validation is needed because many puzzles fall into NP or NP-Complete complexity classes.
- Difficulty estimation: Test difficulty is scored from rule diversity, rule-cell count, density, grid size, and estimated rule interactions on a 1-to-5 scale.The test set distributes 86, 118, 121, 86, and 89 puzzles across levels 1 through 5, respectively.
C Comparison with the SPaRC Text Baseline
Compared with SPaRC’s textual presentation, the strongest scaffolded visual inputs improve several capable models, while the weakest models remain near the accuracy floor.
- Text baseline: Humans reach 98% on the SPaRC text baseline, while the strongest VLM reaches 38.0% and four of seven models score below 10%.The human-model gap shows that the textual presentation remains non-trivial independently of visual encoding.
- Model comparisons: Qwen 3.5 397B improves from 38.0% to 48.4% (+10.4 pp, Text Symbols) relative to the text baseline.This is the largest reported visual-over-text gain among the listed capable models.
- Model comparisons: Gemma 4 31B improves from 16.6% to 26.2% (+9.6 pp, Text Symbols), while Qwen 3.5 27B improves from 33.9% to 40.0% (+6.1 pp, Cell Coordinates).GLM 4.6V improves from 8.4% to 12.8% (+4.4 pp, Cell Coordinates).
- Floor-level models: Llama 4 Scout, Gemma 3 27B, and Mistral Small 3.2 stay within ±1 pp of their text baselines, all near the accuracy floor.The visual-over-text gain scales with model capability in the reported comparison.
D Recovery Variants
Recovery variants overlay Cell Coordinates on degraded boards, isolating explicit spatial indexing while preserving the planning task and degradation parameters.
- Controlled comparison: Each recovery variant keeps the planning task and degradation parameters unchanged, isolating the contribution of explicit spatial indexing.The comparison changes the representation rather than the underlying puzzle or degradation setup.
- Evaluation: Accuracies are evaluated over 500 test puzzles with 95% bootstrap percentile intervals from 200,000 puzzle-ID resamples.Paired differences resample puzzle IDs once and evaluate both conditions on the same resample; gains are resolvable for five models above the floor.
- Recovery design: Figure 12 presents low-contrast, low-resolution, and rotated boards alongside recovery versions with Cell Coordinates overlaid.The recovery counterparts retain the degraded visual conditions while adding spatial indexing.
E.2 Grid Lines versus Coordinate Labels
The Axis Labels ablation shows that explicit coordinate labels, rather than generic visual changes, provide the main benefit for spatial grounding. Blank reserved margins disrupt indexing, whereas labels anchor coordinates and improve task solving.
- +1.4, +0.2, and +2.6 pp: Grid lines alone provide only small gains over the Original board for the three strongest models.
- Below the Original board for all three models: reserving a blank label margin harms task-solving accuracy.
- +2.4 to +7.6 pp: Printing coordinate labels in the reserved margin recovers and exceeds Original-board performance.
- 4–5% with explicit labels versus 17–46% with a blank margin: ±1 coordinate offsets rise sharply when the reserved margin is uninformative.
- Explicit labels act as coordinate anchors, while a blank band appears to be treated as part of the board and offsets cell indices.
E.3 GRPO Post-Training
The section evaluates GRPO post-training alongside visual representations and examines whether perception measurements align with end-to-end planning success. Results emphasize representation-dependent gains and stronger alignment for strict board-level perception.
- GRPO is applied to Qwen 3 VL 4B Thinking and 8B Thinking using 500 SPaRC training examples and validity-based path rewards.
- Approximately 30 h and 60 h: GRPO training requires these wall-clock times for the 4B and 8B models, respectively.
- +34.0 pp: The strongest reported scaffold gain over the Original visual format is achieved by Qwen 397B.
- Across seven evaluated VLMs, Original-board symbols are heavily confused with the dominant Path class, motivating board-level perception analysis.
- r = 0.51, R2 = 0.26, p < 0.001: Average Rule Detection Accuracy remains positively associated with end-to-end task accuracy.
- r = 0.96: Exact-Match Board Accuracy aligns more tightly with downstream task accuracy than per-rule averages.
E.6 Token Usage
Inference-time compute varies substantially across model families and difficulty levels. Large reasoning budgets do not eliminate Original-board grounding difficulty, while visual scaffolding benefits both long- and short-reasoning models.
- ∼31k to ∼50k tokens: Qwen 3.5 27B increases completion length with puzzle difficulty.
- ∼24k to ∼36k tokens: Qwen 3.5 397B also scales completion length with difficulty.
- ∼14k–17k tokens: GLM 4.6V uses a moderate, roughly constant completion budget across difficulties.
- Below ∼6k tokens: Mistral Small 3.2, Gemma 3 27B, Gemma 4 31B, and Llama 4 Scout remain low-budget and show no visible difficulty scaling.
- +31.8 pp and +34.0 pp: Qwen 3.5 27B and 397B have the largest scaffolding gains despite already using the largest compute budgets.
- Visual access benefits both long-reasoning and short-reasoning models but does not substitute for reasoning capacity.
E.7 Object Detection under Visual Degradation
Object-detection analyses test how visual degradation affects board-element perception with and without cell-coordinate scaffolding. Rotation particularly undermines scaffold-supported rule detection, even when end-to-end reasoning partly compensates.
- All |∆| ≤6.2 percentage points: Low contrast and low resolution have only marginal effects on Original-board rule detection.
- −9.0 pp end-to-end versus −33.4 pp rule detection: Rotation damages Cell Coord performance much more at the rule-detection level for Qwen 3.5 397B.
- The Cell Coord overlay’s contribution to rule-level perception is especially sensitive to geometric transformations.
- Cell-level scaffolding primarily supports spatial grounding, but the scaffold becomes a liability when the board is rotated.
- The object-detection prompt replaces solving with symbolic-grid reconstruction, isolating visual grounding errors from path-finding errors.
- The improved prompt adds explicit path-cell, rule-cell, and region definitions, plus a worked example and precise output formatting.