iVISPAR -- An Interactive Visual-Spatial Reasoning Benchmark for VLMs

• Existing benchmarks often simplify spatial reasoning through static text or images, 2D environments, and limited interactivity, missing dynamic multi-step complexity.Source: §1 · pp. 1–2
• iVISPAR introduces an interactive benchmark for evaluating VLM visual-spatial reasoning as agents in dynamic 3D environments.Source: §1 · p. 2
• The Sliding Geom Puzzle replaces numbered tiles with color- and shape-defined geometric objects, requiring logical planning, spatial awareness, and multi-step problem-solving.Source: §1 · p. 2
• The benchmark scales difficulty through board size, object count, and solution-path length, including configurations described as NP-complete challenges beyond baseline human performance.Source: §1 · p. 2
• VLMs handle basic spatial reasoning but struggle with complex scenarios, particularly in 3D environments, and remain below optimal and human baselines.Source: §1 · p. 2

2.1 Spatial Reasoning Benchmarks

• Existing spatial-reasoning benchmarks often lack language interfaces, high-fidelity interaction, or demanding 3D generation, while visual datasets generally do not support interactive planning and action execution.Source: §2.1 · p. 2

2.2 Spatial Reasoning in LLMs

• LLMs have been studied for spatial planning, path generation, symbolic geometric reasoning, and spatial question answering in textual or symbolic environments.Source: §2.2 · p. 2

2.3 Spatial Reasoning in VLMs

• VLM research evaluates spatial relationships, object configurations, and visual-language alignment, increasingly extending from static visual question answering to interactive simulated agents.Source: §2.3 · pp. 2–3
• iVISPAR contributes an interactive multimodal benchmark for evaluating VLMs as agents in spatially grounded environments.Source: §2.3 · p. 3

3 The iVISPAR Benchmark

• iVISPAR presents board states as 3D images, 2D top-down views, or text and lets agents issue natural-language actions through a text-based API.Source: §3 · p. 3

3.1 Sliding Geom Puzzle

• The Sliding Geom Puzzle replaces numbered tiles with objects identified by color and shape, increasing visual-spatial complexity and requiring feature-based object matching.Source: §3.1 · p. 3

3.2 Game dynamics

• Agents rearrange geoms toward a goal configuration by issuing cardinal moves that reference each object’s color-shape combination.Source: §3.2 · p. 3
• After each command, the simulator validates the action and returns an updated board state; effective moves shorten the path, ineffective moves lengthen it, and invalid moves leave the state unchanged.Source: §3.2 · p. 4

3.3 Observation Spaces

• 3D observations use angled views that may occlude objects, whereas 2D layouts keep elements visible; text representations encode board states directly in the prompt.Source: §3.3 · p. 4

3.4 Complexity Scalability

• Puzzle difficulty scales through board size, object count, object variability, shortest-path length, and interference between objects’ optimal paths.Source: §3.4 · p. 4

4.1 Dataset Generation

• The benchmark uses 4 × 4 boards with 2–11 objects and shortest paths of 2–11 steps, yielding 300 configurations and 900 episodes per model across modalities.Geom identities combine four shapes and four colors; configurations use zero geom interference so shortest paths equal cumulative Manhattan distance.Source: §4.1 · p. 5

4.2 Baselines

• Human participants and AI agents provide comparison bounds through human interaction, A*-computed optimal solutions, and uniformly sampled valid random moves.The human baseline uses 30 participants interacting through the same command-line prompting format as VLM agents.Source: §4.2 · p. 5

4.3 Models

• The study evaluates six open- and closed-source VLMs using official APIs or public checkpoints, with shared decoding settings for open-source models.The evaluated systems include Sonnet-3.5, Gemini-2.0-flash, GPT-4o, InternVL2.5-78B, LLaVA-OneVision-72B, and Qwen2-72B.Source: §4.3 · p. 5

4.4 Context-Aware Zero-Shot Reasoning

• Models use context-aware zero-shot chain-of-thought reasoning with only the previous two steps retained, limiting experience while preserving sequential coherence.The approach combines state representations, task instructions, pretrained knowledge, and no examples.Source: §4.4 · p. 5

4.5 Instruction Prompts

• A single minimal prompt template is shared across models and modalities, avoiding model-specific prompt engineering and keeping comparisons reproducible.Visual and text prompts are isomorphic except for the image placeholder.Source: §4.5 · p. 5

4.6 Evaluation

• Evaluation combines solved-environment fraction with mean step-deviation from the A*-derived optimal path, where zero regret indicates optimal behavior and positive regret indicates detours.The deviation metric remains informative even when environments are unsolved.Source: §4.6 · p. 6

4.7 Auxiliary Task

• The auxiliary task evaluates visual board-state reconstruction by matching predicted and true geoms with the Hungarian algorithm and categorizing misses, hallucinations, attribute mismatches, and format errors.A match requires sharing at least color or shape; unmatched elements are classified as missed or hallucinated.Source: §4.7 · p. 6

5 Results

• The study compares 3D, 2D, and text-based agents over 300 episodes per modality with a 20-action cap, measuring completion and deviation from optimal paths.Figure 4 also includes human and random-action baselines.Source: §5 · p. 7
• Action analysis separates effective and ineffective legal moves from invalid moves and illegal commands, distinguishing progress toward the goal from unchanged board states.Effective moves reduce path length, ineffective moves increase it, while invalid or illegal actions leave the state unchanged.Source: §5 · p. 7
• Board-state inference counts correctly identified, missed, hallucinated, and mismatched geoms, while also tracking formatting failures.Mismatches are assessed in coordinates, color, and shape.Source: §5 · p. 7
• Complexity is analyzed along shortest-path length and geom count because longer paths demand broader planning while more geoms increase local rearrangement demands.Figure 7 summarizes performance across 100 combinations of these complexity factors using minimal distance to the goal after 20 steps.Source: §5 · p. 7

6.1 Model Performance

• Sonnet-3.5 achieves the highest reported success rate, 89.7% in 2D vision, while open-source models perform near the random baseline and humans solve tasks perfectly with near-optimal paths.Gemini-2.0-Flash and GPT-4o follow Sonnet-3.5 among the reported closed-source comparisons.Source: §6.1 · p. 7, Figure 6 · p. 8
• Even models solving fewer than 1% of tasks often produce more efficient paths than random actions, indicating limited goal-directed behavior despite low completion.This behavioral pattern is consistent with the accompanying error and board-state analyses.Source: §6.1 · pp. 7–8

6.2 Error Patterns

• Models rarely issue illegal commands or format errors, but board-state inference falls from 4.2 correctly identified objects in 2D to 1.4 in 3D.The 3D decline is driven mainly by coordinate errors, with additional color, shape, and missed-detection increases while hallucinations and format issues remain stable.Source: §6.2 · p. 8
• Localization errors help explain weaker 3D performance by producing more ineffective moves, including attempts to place objects out of bounds or on occupied cells.The connection is supported by the joint action and board-state error analysis.Source: §6.2 · p. 8

6.3 Modality Impact

• Closed-source models perform best on 2D vision, followed by text and then 3D, whereas open-source models remain near-random on visual inputs but relatively stronger on text.Out-of-bounds errors are nearly twice as common in text as in 2D vision, while board-state inference is more accurate in 2D than 3D.Source: §6.3 · p. 8

6.4 Complexity Scaling

• Performance declines as shortest-path length and geom count increase, with geom count especially damaging in 3D while text performance depends more on path length.The pattern highlights difficulty with sequential visual planning and spatial reference from language alone.Source: §6.4 · p. 8, Figure 7 · p. 9
• As geom count rises, coordinate errors increase, while hallucinations, format errors, color and shape mismatches, and missed objects remain comparatively stable.The auxiliary-task pattern isolates localization as the error type most sensitive to board density.Source: §6.4 · p. 9

7 Conclusion

• iVISPAR evaluates VLM agents’ logical planning, spatial awareness, and multi-step problem-solving in 3D vision using the Sliding Geom Puzzle.The evaluation spans board configurations with two complexity levels and compares open-source and closed-source models against human, optimal, and random baselines.Source: §7 · p. 9
• VLMs understand instructions and outperform random agents on simple tasks but struggle with complex configurations and intricate problem properties.The findings report substantial performance differences among tested VLMs.Source: §7 · p. 9
• VLMs perform better with 2D vision than with 3D or text-based tasks, while 3D inputs produce more frequent errors in detecting, placing, and matching geometric objects.These board-state inference errors indicate that visual alignment remains especially challenging for 3D spatial reasoning.Source: §7 · p. 9
• The benchmark is planned for expansion to scene understanding, rotation, and transformation challenges.These additions are identified as future work rather than current benchmark capabilities.Source: §7 · p. 9

Limitations

• The restricted context window limits the number of images VLMs can process, unlike humans’ ability to recall each episode step and use past experience.Extended image inputs are described as disruptive to sequential coherence and costly to process.Source: Limitations · p. 10
• Long-context or deep-thinking models may be ill-suited to step-wise interactive simulations because their per-frame API costs are disproportionately high.This limitation also constrains direct comparisons with human participants, who integrate episodic knowledge more efficiently.Source: Limitations · p. 10

Impact Statement

• The paper identifies no specific immediate ethical or societal risks beyond those already associated with the field.It attributes this assessment to the work’s largely theoretical nature and limited scale.Source: Impact Statement · p. 10

A.1.1 System Prompt Instructions

• The prompt requires agents to transform a 4×4 board from its active state to the goal state using valid actions along the shortest path.Agents must analyze the current state, compare it with the goal, check past actions, and propose the next move while respecting non-overlap and boundary constraints.Source: §A.1.1 · p. 13
• The action interface restricts outputs to move <color> <shape> <direction>, using four specified colors, four shapes, and four directions.Actions must change the state, cannot use coordinates, and must end with the required action string without extra characters.Source: §A.1.1 · p. 13
• The auxiliary board-state task asks agents to locate every colored geometric object on a 4×4 image board and report chess-style coordinates.Columns a–d run left to right, rows 1–4 bottom to top, and outputs list only occupied coordinates with valid color-shape labels.Source: §A.1.1 · p. 13

A.1.2 Observations of Scaling Episode Complexity

• Figure 8 illustrates observations across vision 3D, vision 2D, and text modalities as episode complexity varies from 2 to 11 geometric objects.The examples include optimal path lengths of 8, 4, 2, and 6 on 4×4 boards.Source: Figure 8 · p. 14

A.1.3 Interaction Example

• The interaction example instructs agents to compare active, goal, and past states, including previous actions, to select valid moves and recover from failed attempts.Invalid actions include overlap, blocked or out-of-bounds moves, and actions that leave the board unchanged.Source: §A.1.3 · p. 15
• The worked interaction clears a blocked pathway by moving the blue cube right after comparing object positions between the current and goal states.The example identifies the green sphere as blocking lower movements and the yellow sphere as needing to move from B1 toward B2.Source: §A.1.3 · p. 15

A.1.4 Episode Progression Example

• Figure 9 presents a vision-3D episode progression over steps 1–12 for an optimal path length of 9, including three mistakes.Other modalities are described as progressing analogously.Source: Figure 9 · p. 16

A.3 Sliding Tile Puzzle

• The sequential generalized sliding-tile puzzle formalizes feasible tile paths with adjacent movement, goal achievement, exclusive occupancy, and one-tile-at-a-time motion.Because moves are sequential, head-on collision and corner-following constraints are omitted.Source: §A.3 · p. 17

A.7.1 Optimal Agent

• The optimal agent requires a start and goal, and ensures either a path from start to goal or failure.Source: §A.7.1 · p. 25
• The optimal agent iteratively processes states while the open set is nonempty, removing the current state and examining its neighbors.Source: §A.7.1 · p. 25
• After reaching the goal, the optimal agent returns a path reconstructed from the recorded predecessors.Source: §A.7.1 · p. 25
• For each neighbor, it updates the predecessor and cost when the tentative score improves or the neighbor lacks a recorded score.Source: §A.7.1 · p. 25
• The optimal agent ranks neighbors with fScore[neighbor] = gScore[neighbor] + heuristic(neighbor, goal) and adds unseen neighbors to the open set.Source: §A.7.1 · p. 25

A.7.2 Random Agent

• The random agent requires board size n, an initial state, and a maximum step count, ensuring a path from the initial to final state.Source: §A.7.2 · p. 25
• During generation, the random agent appends each current state to the path and returns the completed path after the loop.Source: §A.7.2 · p. 25