PaperlayerView exact revision on arXiv

Source-linked AI summary

RoTA-Bench: Evaluating 3D Spatial Reasoning through Interactive Projection Alignment

Qingchuan Ma, Yuexiao Ma, Xiang Meng, Yixuan Zou, Tianyu Xie, Wang Chen, Yuhui Zeng, Yuhang Wu, Xiawu Zheng, Rongrong Ji

arXiv:2610.17665v2cs.AI

TL;DR

Can vision-language models infer 3D understanding from 2D observations well enough to act precisely? RoTA-Bench tests this through interactive projection alignment using icon-derived, solvable 3D tasks. The results show a substantial human–model gap, divergence between image-space and pose-space progress, and sensitivity of fine-grained control to interaction history.

  • Problem

    Existing evaluations leave limited evidence on whether vision-language models can turn 2D observations into 3D understanding that supports precise action.

    Source: §1 · pp. 1–2

  • Method

    RoTA-Bench uses interactive projection alignment: models rotate solvable 3D objects constructed from 2D icons to match a target silhouette through visual feedback.

    Source: §1 · p. 2

  • Results

    GPT-5.6-Sol succeeds on 17% of tasks, while all 400 human task runs succeed; image-space and pose-space progress also diverge, and fine-grained control is sensitive to interaction history.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The findings highlight the challenge of converting visual evidence into precise 3D control.

    Source: §6 · p. 9

    Objects extruded from a limited set of 2D icons constrain shape diversity and cannot fully represent complex 3D geometries.

    Source: §5.5 · p. 9

Abstract

from arXiv · show

Can vision-language models turn 2D observations into 3D understanding that supports precise action? This requires inferring object geometry, anticipating the effects of rotation, and refining actions through visual feedback. We introduce RoTA-Bench to evaluate these abilities through interactive 3D-to-2D projection alignment. Given a target silhouette, a model observes a rotated object and chooses successive rotations to recover the target projection within a limited action budget. Our automated pipeline constructs silhouette-preserving 3D objects from 2D icons and uses reversible initialization to guarantee solvability, yielding 674 tasks and a stratified 100-task test set. Empirically, we find that (i) a substantial human-model gap persists: GPT-5.6-Sol succeeds on only 17% of tasks, whereas 20 undergraduate participants collectively complete the test set four times with a 100% success rate; (ii) image-space and pose-space progress can diverge: at matched states, the model more often improves silhouette overlap while worsening pose alignment than humans (19.4% vs. 10.9%); and (iii) fine-grained control is sensitive to interaction history: removing it leaves 96.8% of the model's actions at a coarse 30-degree step size and sharply reduces observed task success. These findings highlight the challenge of converting visual evidence into precise 3D control, beyond recognizing or matching individual images.

1 Introduction

RoTA-Bench targets the gap between passive visual assessment and precise action from 2D observations. Its results show that interactive projection alignment remains difficult for models and distinguishes image-space progress from pose alignment.

1 Introduction

  • Spatial perception supports maintaining object constancy across viewpoints and recovering 3D geometry from 2D projections, motivating evaluation of whether models can use it for precise action..Source: §1 · p. 1
  • Static benchmarks omit action and closed-loop feedback, while many interactive benchmarks emphasize planning or pathfinding rather than inferring 3D orientation from visual cues.The latter settings depend largely on coding, reasoning, and planning rather than translating visual evidence into action.Source: §1 · pp. 1–2
  • RoTA-Bench tests whether models can align a rotated object to a target silhouette through successive visual-feedback-guided rotations within a limited action budget.The pipeline constructs objects from 2D icons, uses reversible initialization to guarantee solvability, and provides 674 tasks with a stratified 100-task test set.Source: §1 · p. 2, Figure 1 · p. 2
  • GPT-5.6-Sol succeeds on 17% of tasks, while all 400 human task runs succeed; the model also more often improves overlap while worsening pose alignment (19.4% vs. 10.9%).The model’s 19.4% rate is compared with 10.9% for humans at matched states.Source: §1 · p. 2

2 Related Work

Prior work covers visual-language capabilities and spatial reasoning in static and interactive settings, but the cited benchmarks leave gaps in evaluating visual 3D orientation control.

2.1 Multimodal Large Language Models and Spatial Reasoning

  • Vision-language and spatial models have added visual instruction following, explicit 3D representations, and spatial cues, yet models still struggle with camera-pose estimation, orthographic projection, and shape recognition.RoTA-Bench instead examines whether 2D observations can support 3D understanding and precise action.Source: §2.1 · pp. 2–3
  • Figure 2 depicts icon-based object construction, reversible initialization, reference-path search, stratified test selection, and closed-loop rotation evaluation.The action space contains 24 discrete rotations, and success requires projection IoU ≥0.90.Source: Figure 2 · p. 3

2.2 Static Benchmarks for Visual and Spatial Reasoning

  • Static visual and spatial benchmarks assess fixed inputs in a single turn, without action or visual feedback.The cited suites cover spatial relations, diagrams, geometry, pose, and action preconditions; A2RBench focuses on symbolic rules rather than grounded 3D perception.Source: §2.2 · p. 3

2.3 Interactive Benchmarks for Visual and Spatial Reasoning

  • Interactive benchmarks cover embodied navigation, digital interfaces, and puzzles, but often emphasize high-level planning, 2D displays, or grid-based pathfinding rather than rotational pose variation.RoTA-Bench evaluates closed-loop projection alignment as a test of 2D-to-3D understanding for precise action.Source: §2.3 · pp. 3–4

3 RoTA-Bench

RoTA-Bench isolates 3D orientation alignment through visual projection matching and closed-loop rotation. Its construction produces solvable icon-derived tasks, while its metrics distinguish task success from geometric diagnostics.

3.1 Motivation and Design Rationale

  • Restricting interaction to rotation creates nonlinear silhouette changes, aspect changes, and self-occlusion while separating orientation reasoning from translation, grasping, and contact dynamics.The design is intended to make the challenge visual and resist textual representation.Source: §3.1 · p. 4

3.2 Task Definition

  • The task supplies a target silhouette, current silhouette, and depth-shaded view; agents choose among 24 rotations, receive updated views, and succeed at projection IoU ≥0.90.Episodes can also end by stopping, reaching the action budget, or stagnating.Source: §3.2 · pp. 4–5

3.3 Dataset Construction

  • The pipeline converts filtered 2D icons into sphere-bounded 3D objects, initializes poses with reversible rotations, and uses reference search to build a stratified 100-task test set from 674 tasks.Reversibility guarantees solvability by construction; reference paths use pose-guided beam search and 30°-restricted breadth-first search.Source: §3.3 · p. 5

3.4 Evaluation Metrics

  • Task Success Rate counts episodes reaching IoU ≥0.90, while Mean Final IoU and mean rotation steps average outcomes across tasks, including failures.Episodes ending by STOP, the 48-decision limit, or stagnation before the threshold count as failures.Source: §3.4 · p. 5
  • Pose error and surface coverage diagnose geometry and visibility rather than define task success; a lower pose error need not yield a larger immediate IoU improvement.Direction and angle metrics assess whether rotations choose error-reducing directions and the best legal angles.Source: §3.4 · pp. 5–6

4 Experimental Setup

The evaluation compares vision-language models, agent frameworks, human participants, and random and greedy baselines under a shared task protocol.

4 Experimental Setup

  • The setup evaluates nine vision-language models, two GPT-5.6-Sol-based agent frameworks, 20 undergraduate participants, two stochastic baselines, and two greedy baselines.Evaluations use a common task protocol, action space, decision budget, success criterion, and termination rules.Source: §4 · p. 6
  • Models run at temperature 0; closed-source systems are queried through APIs at a reported total cost of $1,482.07, while SpatioLM runs locally on eight NVIDIA A100 GPUs.The passage directs readers to appendices for prompts, hyperparameters, tools, and evaluation settings.Source: §4 · p. 6

5 Results

Across RoTA-Bench results, humans substantially outperform vision-language models, while diagnostics locate weaknesses in visual exploration, rotation choice, and fine-grained control. Image-space improvements can also diverge from pose alignment, and the benchmark is limited to icon-derived objects and discrete single-target rotations.

5.1 Main Results

  • GPT-5.6-Sol succeeds on 17.0% of tasks versus 100.0% for humans; pose-greedy reaches 100.0%, while IoU-greedy succeeds on 39.0%.Humans average final IoU 0.923 in 16.22 steps; pose-greedy reaches final IoU 0.924 in 8.32 steps, whereas ReAct has the highest final IoU at 0.628.Source: §5.1 · p. 6, Table 1 · p. 7

5.2 Diagnosing Model Failures

  • GPT-5.6-Sol’s surface coverage is 79.58%, close to humans’ 83.07%, but effective-direction rate is 58.75% versus 83.91%.Conditioned on effective directions, its angle gap is 6.49° versus humans’ 2.99°, and its angle loss is 2.25° versus 0.96° per step.Source: §5.2 · pp. 6–7, Table 1 · p. 7

5.3 Ablation: Auxiliary Views and Interaction History

  • Removing the auxiliary depth view reduces success from 16.67% to 3.33%, while removing interaction history reduces observed success from 5/30 tasks to 0/30.The history-removal effect is larger than the auxiliary-view effect.Source: §5.3 · pp. 7–8, Table 2 · p. 7
  • Without interaction history, 30° rotations rise to 96.76% from 62.75%, while 1° adjustments fall to 0.16% from 12.18%.Effective-direction rates remain nearly unchanged across ablation settings (54.85%–56.52%), while the IoU–angle correlation weakens from ρ = −0.472 to ρ = −0.242 without history.Source: §5.3 · p. 8

5.4 Action Association with Projection and Pose Discrepancies

  • GPT-5.6-Sol’s selected angles correlate more strongly with pre-action IoU (ρ = −0.536) than with pose error (ρ = 0.368), unlike humans’ strong associations with both.Human correlations are ρ = −0.696 for IoU and ρ = 0.751 for pose error; most evaluated models show the same relative pattern as GPT-5.6-Sol.Source: Figure 4 · p. 8, §5.4 · p. 9
  • Across 211 matched states, GPT-5.6-Sol performs more UF actions than humans (19.43% vs. 10.90%), while humans perform more DC actions (33.18% vs. 23.70%).Humans also achieve more UC actions (50.24% vs. 34.60%) and fewer DF actions (5.69% vs. 22.27%).Source: Figure 5 · p. 8, §5.4 · p. 9

5.5 Limitations and Future Work

  • The benchmark’s objects are extruded from a limited set of 2D icons, and each task has one target with discrete rotation actions.The authors identify these choices as limiting shape diversity and simplifying interaction dynamics relative to continuous physical environments.Source: §5.5 · p. 9

6 Conclusion

RoTA-Bench evaluates interactive 3D-to-2D projection alignment, with findings that expose challenges in converting visual evidence into precise 3D control. The conclusion also documents the benchmark’s construction, release, and scope.

6 Conclusion

  • RoTA-Bench evaluates whether vision-language models can use 2D observations to support precise action through interactive projection alignment.The conclusion identifies challenges in converting visual evidence into precise 3D control.Source: §6 · p. 9
  • The pipeline constructs silhouette-preserving 3D objects and uses reversible initialization to guarantee solvability, producing 674 tasks and a stratified 100-task test set.Source: §6 · p. 9

AI use statement

  • The authors state that generative AI aided writing and related-work discovery, while evaluated vision-language models served only as experimental subjects.The authors reviewed AI-assisted materials and take responsibility for the final content.Source: AI use statement · p. 9

Ethics Statement

  • All 20 undergraduate participants gave informed consent and received compensation; the platform collected no personally identifiable or sensitive data.Task geometries came from open-source icons under CC BY 3.0 or CC0, with attributions preserved.Source: Ethics Statement · p. 9

Reproducibility Statement

B Icon Sources and Licensing

  • The 674 source icons come from the Game Icons repository snapshot and are used under CC BY 3.0 or CC0 with attribution preserved.The collection contains 641 CC BY 3.0 icons and 33 CC0 icons.Source: B Icon Sources and Licensing · p. 13

C Human Evaluation

Human evaluations used the same observations, actions, and task rules as model evaluations. The section details symmetry estimation, target and coverage definitions, icon-to-observation processing, and verified search references.

C Human Evaluation

  • Twenty undergraduates each completed one consecutive 20-task block; across four passes, all 400 human task runs succeeded.Humans and models received the same observations, action vocabulary, task rules, and pose-transition function; human reasoning was not recorded.Source: C Human Evaluation · pp. 13–14

D Computation of Object Symmetries

  • Object symmetries are estimated from normalized 2D points by testing rotations and reflections, then retaining transforms that map the point set onto itself within sampling tolerance.The retained planar symmetries are converted into 3D rotations that form the symmetry set used for pose comparisons.Source: D Computation of Object Symmetries · pp. 15–16

E Target Orientation and Icon Details

  • The target pose is the identity because depth-axis projection preserves the source icon’s details, while other orientations may obscure them and need not yield a unique match.Source: E Target Orientation and Icon Details · p. 16

F Surface Coverage

  • Surface coverage is the area-weighted fraction of reconstructed surface observed at least once during an episode, so revisiting a point adds no coverage.Visibility is determined from surface-point orientation and ray obstruction at each observed pose.Source: F Surface Coverage · p. 16

G Icon Processing and Observation Generation

  • Foreground pixels must pass opacity and background-distance tests; deterministic point sampling and shape filters remove fragmented or degenerate icons.Source: G Icon Processing and Observation… · p. 16
  • The pipeline gives 2D outlines finite 3D thickness with deterministic depth samples, then renders orthographic projections and an auxiliary depth-cue image.The auxiliary intensity depends on depth and horizontal position and is clipped to [40, 235].Source: G Icon Processing and Observation… · p. 17

H Search Reference Construction

  • Inverse initialization supplies a feasible solution; two searches generate candidate paths, replay them against the evaluation IoU threshold, and retain the shorter verified path, which is not necessarily optimal.Source: H Search Reference Construction · p. 17

I Dataset Composition and Test Selection

Test selection draws from tasks with longer search-reference paths and allocates places proportionally across path-length groups.

I Dataset Composition and Test Selection

J Additional Results

Additional analyses compare model and human actions, audit outcomes and response validity, and describe the task-level resampling used for uncertainty estimates. They also define the trajectory metrics reported across the test set.

J Additional Results

  • The reported trajectory statistics include final pose error, pose-improving steps, mean per-step IoU change, and mean per-step pose reduction.The shared-state analysis categorizes actions by whether IoU and pose error improve or worsen; Table 8 reports context and outcomes across 211 matched pairs.Source: Table 9 · p. 19, Table 8 · p. 19

K.1 Success Counts and Confidence Intervals

  • GPT-5.6-Sol has the highest observed success rate among the nine models and two agents, but that ranking alone does not establish reliable differences.Table 11 reports success counts and 95% Wilson confidence intervals on the 100-task test set.Source: §K.1 · p. 19, Table 11 · p. 20

K.2 Response Formats and Episode Outcomes

  • The trajectory audit classifies episode endings as stagnation, below-threshold STOP, or step-limit termination, and defines Repeat by consecutive rotations with the same signed axis and angle.Repeated actions can be useful, and stopping reasons alone do not explain why a task failed.Source: §K.2 · p. 19

K.3 Statistical Units and Resampling

  • Uncertainty analyses resample whole tasks, keep each task’s four human runs together, and use paired task samples for method comparisons.The additional bootstrap analyses use 10,000 resamples and percentile 95% confidence intervals.Source: §K.3 · p. 20

K.4 Direction and Angle Diagnostics

  • Human exceeds Sol in Effective Direction Rate by 25.16 percentage points and has a 3.50° smaller Angle Gap; both paired 95% intervals exclude zero.These paired-task action differences do not directly measure the models’ internal pose estimates.Source: §K.4 · p. 20

K.5 Input Ablations and Fine-Grained Control

The ablations associate interaction history with finer rotation choices, while further analyses compare model and human behavior at shared states, decompose action errors, and test performance across object geometries.

K.5 Input Ablations and Fine-Grained Control

  • 96.76% of no-history rotations were 30° (598/618), versus 62.75% with full input (443/706); 1° rotations fell from 12.18% (86/706) to 0.16% (1/618).The ablation removes prior context while retaining the current observation and instructions; it does not isolate a single reasoning mechanism.Source: §K.5 · pp. 20–21

K.6 Comparing Associations and Shared-State Actions

  • For GPT-5.6-Sol, ∆ρ = 0.169 (95% CI: [0.020, 0.330]); for Human, ∆ρ = −0.055 (95% CI: [−0.084, −0.023]).The differences support a stronger association with IoU for Sol and with pose error for Human; matched states control current pose, but earlier histories may differ.Source: §K.6 · pp. 21–22
  • At 211 matched participant–task–state pairs, humans reached shared states after 2.40 rotations on average versus Sol’s 4.18, with direct coverage of 42.01% versus 46.21%.Matching controls current pose, but earlier observations and actions may differ, so the comparison does not establish an internal mechanism.Source: §K.6 · p. 22

K.7 Oracle Action Decomposition

  • Correcting rotation direction yields a larger one-step pose-error improvement than correcting angle, while correcting both yields the lowest expected error.The analysis enumerates legal actions at 2,366 states visited by GPT-5.6-Sol and defines the best action by minimum symmetry-adjusted successor pose error.Source: §K.7 · p. 22, Table 17 · p. 23

K.8 Cross-Geometry Comparison on Matched 30-Task Sets

  • GPT-5.6-Sol achieves higher mean final IoU and a slightly higher success rate on matched-distribution Poly Haven mesh tasks than on icon-derived tasks.The sets are not paired task-by-task; the comparison suggests geometric familiarity may affect performance.Source: §K.8 · pp. 22–23, Table 18 · p. 23

L Implementation and Evaluation Settings

The appendices document evaluation and implementation settings, then illustrate the prompts used to guide the model’s interaction. The prompts present visual cues, action constraints, and chronological history across turns.

L Implementation and Evaluation Settings

  • Implementation settings cover local software, hardware, model hyperparameters, agent configurations, interaction settings, and estimated API costs based on recorded token use and published prices.The cost estimates include the main evaluations and three GPT-5.6-Sol input-ablation conditions.Source: L Implementation and Evaluation Settings · pp. 23–24, Table 20 · p. 24

M Prompt Examples

  • The prompt pairs the target projection with a current oblique view and projection, defines camera-fixed rotation tokens and step sizes, and requires a JSON action response.The oblique view uses the same camera direction and object orientation as the current projection, with depth-dependent shading for additional cues.Source: M Prompt Examples · pp. 24–25
  • Later-turn prompts include earlier observations and action–reasoning records alongside the current observation, with history kept chronological.The example sequence shows the first-turn prompt followed by second- and third-turn prompts.Source: M Prompt Examples · p. 24, Figure 8 · p. 25, Figure 9 · p. 26

N Success and Failure Examples

Two selected GPT-5.6-Sol trajectories illustrate a successful recovery and refinement sequence and an unsuccessful repeated-action sequence; the examples are explicitly non-representative.

N Success and Failure Examples

  • The examples were selected as the shortest executed-rotation runs among successful and unsuccessful tasks, so they are illustrative rather than representative.The selected runs are single_00270 and single_00039.Source: N Success and Failure Examples · p. 26
  • The successful single_00270 trajectory raises IoU from 0.624 to 0.977 after reversing an unhelpful yaw and refining across roll, pitch, and yaw.The selected run takes eight rotations.Source: N Success and Failure Examples · p. 26, Figure 10 · pp. 27–28
  • In single_00039, all 12 actions are 30° rotations in the same yaw direction, returning to the initial alignment at IoU 0.450 without improvement.Repeated proposals to reveal a hidden face do not change the action strategy, and stagnation stopping ends the run.Source: N Success and Failure Examples · p. 26, Figure 11 · pp. 29–31

Related papers

Loading 2610.17665v2…