PaperlayerView exact revision on arXiv

Source-linked AI summary

SpaceAct: Benchmarking 3D Spatial and Temporal Reasoning via Closed-Loop Action and Active Perception

Chenglong Shi, Zekai Yuan, Yue Song, Jon Pu, Will Wu

arXiv:2610.38405v1cs.ROcs.AI

TL;DR

Existing benchmarks often omit active visual grounding, continuous physical constraints, or time-dependent action, leaving important demands of 3D physical reasoning unevaluated. SpaceAct is a closed-loop benchmark paired with SAPA, which supports active perception and physical interaction under controlled conditions. Across frontier models, results show a substantial gap in continuous manipulation and persistent visual-grounding and control bottlenecks even when a correct plan is supplied.

  • Problem

    Many prior benchmarks use static visual questions or simplify interactive tasks with privileged coordinates, abstract actions, or no time-dependent dynamics.

    Source: §1 · p. 2

  • Method

    SpaceAct evaluates closed-loop 3D spatial and temporal reasoning across verified tasks, while SAPA supports active perception, physical action, and controlled diagnostic conditions.

    Source: §1 · pp. 3–5

  • Results

    Across 44 hardest instances, frontier MLLMs scored 4.8–79.1, and 7 of 10 solved at most 27.3% of continuous 6-DoF tasks; for the top three, +state closed 90% of the gap and reduced collisions to 0.5.

    Source: §1 · p. 5

  • Takeaways & Limitations

    Even with a reference plan, visual grounding and closed-loop control remain bottlenecks; with a plan and exact 3D state, the three strongest models solved every diagnostic instance in 5.3 turns on average with 0.5 collisions per episode.

    Source: §7 · p. 21

    The main matrix uses seed 100 and the hardest tier of each family; only four L5 families were rerun with additional seeds, while the other seven models were run once.

    Source: §7 · p. 21

Abstract

from arXiv · show

To solve problems in 3D space rather than merely describe images, multimodal large language models (MLLMs) need 3D spatiotemporal reasoning: reconstructing 3D geometry from occluded 2D views, anticipating physical dynamics, and executing precise, closed-loop actions. Prior studies test passive VQA on static images or symbolic planning, requiring no execution or supplying privileged coordinates while ignoring collision, statics, and timing. We introduce SpaceAct, an interactive benchmark evaluating 3D spatiotemporal reasoning via closed-loop physical action and active perception, with SAPA, a Spatiotemporal Active-Probing Agent framework. Built on interlocking bricks and mechanical puzzles, it has 108 verified instances across 36 task families in six categories, without privileged coordinates, and diagnostic modes (+plan, +state, no-camera, blind, L7 levers). In brick-bridging or aperture-threading, models rotate the camera to resolve occlusion, read in-scene rulers, output 6-DoF poses under swept-collision and statics checks, and self-correct via split-screen rollback feedback (dynamic tasks freeze time during deliberation and give five strobe frames). Across 10 frontier MLLMs on the 44 hardest instances, scores span 4.8-79.1 (seven solve at most 27.3% of 6-DoF tasks); for the top three, a reference plan (+plan) closes 44% of the gap while leaving collisions at 5.2 per episode, whereas adding exact 3D state (+state) closes 90% and cuts collisions to 0.5; for four others, +plan lifts the mean score from 10.9 to 35.3 and +state to 81.8; and 46% of failures are collision or observation loops. SpaceAct and SAPA advance evaluation to closed-loop physical interaction, showing that even with a valid plan, MLLMs remain bottlenecked by 3D visual grounding and action refinement, while offering a testbed for active multi-view exploration and explicit 3D grounding.

1 Introduction

Existing benchmarks often omit active visual grounding, continuous physical constraints, or time-dependent action, limiting evaluation of the capabilities needed for 3D manipulation. SpaceAct and SAPA combine these requirements in a closed-loop benchmark and diagnostic framework, and results show that visual grounding and closed-loop control remain bottlenecks even when models receive a correct plan.

1 Introduction

  • Prior benchmarks often use static images or videos, privileged coordinates, collision-free motion planning, or tasks without time-dependent dynamics, omitting key demands of physical manipulation.SpaceAct instead requires visual metric localization, continuous collision and statics constraints, and action in an evolving world.Source: §1 · pp. 2–3
  • SpaceAct combines ruler-based localization, continuous 6-DoF manipulation, calibrated statics, mechanism synthesis, and strobe dynamics across 36 task families and 108 verified instances in six categories.Instances are validated by a deterministic solver and filtered for known shortcuts.Source: §1 · p. 4
  • 96.6 score with 0.5 collisions: for the three strongest models, adding exact 3D state to a reference plan closed 90% of the remaining gap, while the plan alone closed 44% and left collisions at 5.2.For four other models, +plan raised mean score from 10.9 to 35.3, and +state raised it to 81.8.Source: §1 · p. 5
  • Across 44 hardest instances, 10 frontier MLLMs scored 4.8–79.1, while 7 of 10 solved at most 27.3% of continuous 6-DoF manipulation tasks.Models also struggled to turn rollback feedback into geometric corrections: 46% of unsolved episodes involved collision or observation loops.Source: §1 · p. 5
  • SAPA provides active camera control and physical probing, dynamic memory, five-frame strobe observations, split-screen rollback feedback, and controlled diagnostic modes.Its modes target components of the perception–control loop, including visual grounding, physical identification, planning, and frame-aware control.Source: §1 · pp. 3, 5

2 Related Work

Prior work evaluates spatial reasoning through passive visual questions, interactive tasks with abstract actions, embodied environments that simplify 3D physics, or physical prediction and construction tasks. SpaceAct combines active metric grounding, continuous manipulation under physical constraints, and time-dependent interaction; SAPA standardizes feedback for closed-loop evaluation.

3D spatial reasoning benchmarks

  • Static spatial VQA benchmarks test depth, orientation, perspective, and metric estimation, but models remain passive and do not issue continuous 3D actions or experience collision or collapse.Multi-step suites also use multiple-choice questions to test mental rotation and assembly-step prediction.Source: §2 · pp. 5–6

Interactive spatial reasoning benchmarks

  • Interactive spatial benchmarks use discrete or semantic actions, while VABench delegates collision checking and trajectory generation to a fixed execution layer.SpaceAct requires ruler-grounded metric 6-DoF control, enforces swept collision and statics, and introduces a moving world.Source: §2 · pp. 6–7

Interactive embodied and puzzle benchmarks

  • Embodied and puzzle benchmarks often rely on semantic priors, externally supplied object coordinates or motion planners, and tasks where time does not affect outcomes.SpaceAct removes these three shortcuts.Source: §2 · p. 7

Physical construction and intuitive physics

  • Physical construction and intuitive-physics benchmarks emphasize passive prediction, 2D trial and error, or stepwise assembly and load testing; SpaceAct combines these with 3D mechanisms and strobe dynamics.Its scope also includes ruler-based localization, continuous manipulation, and calibrated statics.Source: §2 · p. 7

Feedback-driven LLM agents

  • SAPA unifies first-violation truncation and split-screen rollback views to provide models with identical visual feedback conditions during evaluation.The framework builds on reasoning–acting and environment-feedback self-correction agents.Source: §2 · p. 7

3 Problem Formulation and the SAPA Framework

SpaceAct represents vision-driven physical problem solving as a partially observable process and gives SAPA a three-tier action space for perception, physical control, and discrete or timed actions. SAPA combines these capabilities in a per-turn policy, using dynamic memory and execution feedback to support active perception and self-correction.

3.1 Action Space

  • The action space combines information-gathering perception, metric translation and rotation or brick placement, and discrete, timed, or final-commit actions.Camera actions preserve world time; physical motions are checked continuously and stop at the first collision, topology, or statics violation.Source: §3.1 · pp. 7–8

3.2 A Deliberation-Frozen Spatiotemporal POMDP

  • The framework models the task as a partially observable process with hidden physical parameters and an explicit clock that advances only during action execution.Observations may be single images or five strobe frames, and the policy conditions on task instructions, bounded history, and a persistent note.Source: §3.2 · p. 8

3.3 Diagnostic Decomposition

  • The diagnostic decomposition separates visual 3D grounding, physical identification, multi-step planning, and frame-aware control, while treating them as capabilities within one policy.Controlled interventions replace or constrain one capability; +state supplies true state alongside a reference plan, while other modes and levers stress grounding or control.Source: §3.3 · p. 8
  • Figure 3 contrasts diagnostic inputs: +plan adds a high-level reference plan, while +state also supplies coarse layout and exact payload poses at every turn.The model still generates every action, and conditions share observations, task rules, history, notes, and execution protocol.Source: Figure 3 · p. 9

3.4 SAPA: An Active-Perception Agent

  • SAPA decides at each turn where to look and what to do, carrying dynamic memory across turns.The model returns a rationale, an updated note, and 1–5 tool calls; execution halts at a violation and returns split-screen rollback feedback.Source: §3.4 · p. 9

3.5 Scoring

  • Episodes receive a tiered completion-quality score from 0 to 100, with solved episodes scoring 40–100 and unsolved episodes 0–35.Construction tasks B1, B2, and B3 that end in a load test are pass/fail and contribute only to success rate.Source: §3.5 · p. 10
  • Figure 4 summarizes the benchmark, SAPA turn inputs, execution checks, observation feedback, and diagnostic conditions.The framework uses RGB observations with rulers and no coordinates; dynamic tasks provide five same-view frames, and a violation truncates execution and returns a split-screen rollback view.Source: Figure 4 · p. 10

4 The SpaceAct Benchmark

SpaceAct comprises six categories spanning construction, puzzles, manipulation, assembly, perception, and temporal or mechanism reasoning. Its difficulty tiers add interacting constraints and, at L7, perception or coordinate-frame challenges; capability tags and programmatic verification characterize and validate the instances.

4.1 Six Task Categories

  • SpaceAct comprises 36 task families and 108 instances, with three instances per family, across six categories: structural construction, combinatorial puzzles, continuous manipulation, assembly, multi-view perception, and temporal/mechanism reasoning.Source: §4.1 · p. 11

4.2 Difficulty Tiers and L7 Levers

  • L1 uses few constraints and generous margins, while L3 and L5 add coupled constraints, decoys, and tighter tolerances; L7 keeps the L5 task fixed and adds perception or coordinate-frame levers.Temporal L7 tiers instead introduce undisclosed physical changes, such as tunnel speeds or ice tilt.Source: §4.2 · p. 12

4.3 Capability Annotation

  • Fine-grained family tags aggregate into six capabilities: 3D grounding, mental rotation, topological constraints, non-monotone sequencing, quantitative statics, and temporal estimation with mechanism synthesis.Source: §4.3 · p. 12

4.4 Verification

  • Every instance is program-verified as solvable, with a reference solution defining its action-count score; generators also reject instances with known shortcuts.Discrete puzzles use search for optimal solutions and, when required, uniqueness certification; continuous tasks use replayed scripts or load checkers.Source: §4.4 · p. 12

5 Experimental Setup

The experiments evaluate ten frontier MLLMs on a 44-instance hardest-setting matrix, reporting tiered scores, success rates, and episode-level efficiency and collision measures. Baselines and paired controlled diagnostics compare model performance under specified interventions, with incomplete API runs handled separately from scored outcomes.

Experimental Setup

  • The main matrix covers 44 instances across all 36 families, using the hardest tier available, one run per model-instance pair, a 100-turn budget, and up to five actions per turn.The evaluation uses ten frontier MLLMs through public APIs.Source: §5 · pp. 12–13
  • The primary metric is mean tiered score over 41 graded instances, alongside success rate, turns, collisions or rollbacks, tokens, and category-level success rates.Source: §5 · p. 13
  • Baselines comprise a random valid agent, a myopic greedy agent covering 29 instances, and a reference solver covering 38 instances.Source: §5 · p. 13
  • Controlled diagnostics pair conditions against the same model-instance episodes, with plan and state tests on eight L5 instances and other interventions on their specified subsets.Pairs with abandoned runs are dropped from paired diagnostics; abandoned main-matrix runs count as unsolved with score 0.Source: §5 · p. 13

6 Results

Across the main matrix, model scores span 4.8–79.1, with the greatest difficulty in metric 3D geometry tasks. Spatial L7 difficulty, controlled diagnostics, trajectory analyses, and robustness checks further locate where performance breaks down and how consistently the findings hold.

6.1 Main Results

  • Scores span 4.8 (Step 5 Preview) to 79.1 (Claude Opus 5.5), separating three model tiers; the opaque 4×4×4 gravity maze is solved by none of nine completed runs.The tiered score averages over 41 graded instances; success rate covers all 44.Source: §6.1 · pp. 13–14, Table 3 · p. 14

6.2 Category Profiles

  • Mean success rates are lowest for multi-view perception (23.3%) and 6-DoF manipulation (29.1%), where seven models solve at most 27.3% of manipulation tasks and seven solve no perception task.Category rankings vary, so a single aggregate score hides capability differences.Source: §6.2 · p. 15, Figure 5 · p. 15

6.3 Difficulty Levers

  • Spatial L7 lowers mean scores from 26.2 to 13.7, while temporal L7 scores remain 46.5 versus 46.6; spatial L7 keeps the underlying task plan unchanged.The spatial drop is concentrated outside the leading tier; the seven models below the top three solve 0 of 42 L7 spatial episodes.Source: Figure 6 · p. 15, §6.3 · p. 16

6.4 Controlled Diagnostics

  • For the top three, +plan raises mean score from 65.1 to 80.3 but leaves collisions at 5.2 per episode; +state reaches 96.6 and cuts collisions to 0.5.For the four further models, +plan raises mean score from 10.9 to 35.3 and +state to 81.8; +plan collisions rise from 21.5 to 28.0, while +state cuts them to 4.6.Source: Table 4 · p. 16, §6.4 · p. 17, Figure 7 · p. 17
  • Removing images lowers score from 73.9 to 17.7 and raises collisions from 3.6 to 33.5 per episode; locking the camera lowers heavily occluded-instance scores from 58.3 to 37.8.Removing the contact snapshot leaves mean score nearly unchanged (31.8 to 30.3) but raises collisions from 14.6 to 23.2 per episode.Source: §6.4 · pp. 17–18

6.5 Behavioral Analysis

  • For eight of nine models, camera actions are more common in unsolved than solved episodes, and 44%–71% of collisions are followed by another collision on the next turn.Batching yields 1.29–2.52 actions per response and reduces model calls by 23%–60% (median 50%).Source: §6.5 · pp. 18–19

6.6 Failure-Mode Analysis

  • Collision and observation loops account for 46% of 227 unsolved episodes; loops reach 60% in continuous 6-DoF manipulation, while 86% of unsolved mechanism-synthesis episodes are stalled without loops.In the helical-wire case, Claude solves in 34 turns with 15 collisions, whereas Qwen records 93 collisions and stops at progress 0.46 after grounding errors.Source: §6.6 · pp. 19–20, Figure 10 · p. 20

6.7 Robustness

  • Across three seeds, model rankings are identical and mean-score standard errors are at most 2.0 points; puzzle-rule perturbations leave solve counts essentially unchanged but cannot rule out memorization.The perturbation comparison is P2: 7/10 versus P3: 6/10.Source: §6.7 · p. 20

7 Discussion and Limitations

Across task categories and controlled diagnostics, current frontier MLLMs handle temporal probing comparatively well but struggle to turn visual grounding into continuous 3D action. Exact 3D state substantially improves diagnostic performance, while the evaluation scope is concentrated on the hardest tier and mostly one seed.

7 Discussion and Limitations

  • With a reference plan and exact 3D state, the three strongest models solve every diagnostic instance in 5.3 turns on average with 0.5 collisions per episode.A plan alone leaves their collision rate unchanged; collision and observation loops account for 46% of unsolved episodes.Source: §7 · p. 21
  • Temporal physical probing holds up comparatively well, but the steepest drops come from hidden geometry or rotated working frames, and 46% of unsolved episodes are observation or collision loops.Without images, models degrade into blind probing by collision.Source: §7 · p. 21
  • The main matrix uses seed 100 and the hardest tier of each family; only the three diagnostic models have reruns on seeds 101 and 102, while the other seven models were run once.State injection improves 48 of 54 pairs and worsens 2; image removal lowers all 12 graded pairs.Source: §7 · p. 21

8 Conclusion

SpaceAct evaluates 3D spatial and temporal reasoning through closed-loop physical action and active perception, with SAPA providing a framework for that evaluation. Results across frontier MLLMs indicate that visual grounding and closed-loop control remain bottlenecks even with a correct high-level plan.

8 Conclusion

  • Across 10 frontier MLLMs, a reference plan closes 44% of the remaining score gap for the three strongest models but leaves collisions unchanged, while exact 3D state closes 90% and reduces collisions from 5.2 to 0.5.For four further models, the mean score rises from 10.9 to 35.3 with a plan and to 81.8 with the plan and exact 3D state.Source: §8 · p. 21
  • SpaceAct contains 36 task families and 108 verified instances across construction, puzzles, continuous 6-DoF manipulation, assembly, multi-view perception, and temporal and mechanism reasoning.The benchmark uses closed-loop physical action and active perception.Source: §8 · p. 21
  • SAPA provides diagnostic modes targeting visual grounding, physical identification, multi-step planning, and closed-loop control, while enforcing continuous collision and rigid-body statics constraints without privileged coordinates.The clock is frozen during model deliberation.Source: §8 · p. 21

Appendix A Task Family Catalog and Verification Details

Appendix A catalogs SpaceAct’s 36 task families and 108 verified instances across six categories, alongside their action primitives, constraints, and deterministic verification methods. The examples include occluded assembly and perception tasks, continuous manipulation puzzles, and timed dynamic mechanisms.

Appendix A Task Family Catalog and Verification Details

  • Table A.1 catalogs 36 task families and 108 verified instances across six categories, listing tiers, action primitives, physical or geometric constraints, and deterministic verification methods.Figure A.1 shows one initial observation per family as rendered to the model.Source: Appendix A · p. 23, Table A.1 · p. 23
  • Continuous manipulation examples include chiral peg-in-hole, corridor rod, wire-ring, and unlinking tasks, with swept-volume or SE(3) planning-based reference verification.Dynamic examples include strobe-observed timing tasks and mechanisms checked with phase-window, kinematic, collision-free, or trajectory solvers.Source: Appendix A · p. 24
  • Catalog examples include occluded polycube packing and low-elevation block counting, with exact-cover and visibility-based verification methods.The catalog also includes multi-wall and oblique-shadow construction and occluded chiral or rotated-copy perception tasks.Source: Appendix A · p. 24
  • Figure A.2 presents a five-frame strobe observation and a split-screen rollback view at full resolution.The strobe frames share a viewpoint at 0.25-second intervals while the deliberation clock is frozen; the rollback image shows a forbidden-zone collision and the valid state after rollback.Source: Appendix A · p. 25, Figure A.2 · p. 25

Appendix B Evaluated Model Configurations

Appendix B reports configurations for 10 frontier MLLMs accessed through public APIs. Its configuration table organizes reasoning settings alongside maximum output-token settings.

Appendix B Evaluated Model Configurations

  • The evaluated set contains 10 frontier general and reasoning MLLMs, all accessed via public APIs.Table B.1 is the configuration table for these models.Source: Appendix B · p. 25, Table B.1 · p. 25
  • The table headings identify reasoning configuration and maximum output tokens as configuration fields.Reported settings include native long-CoT thinking and default thinking, each with 131,072 maximum output tokens.Source: Appendix B · p. 25
  • A Preview configuration uses high reasoning effort and 65,536 maximum output tokens.Source: Appendix B · p. 25

Appendix C Supplementary Diagnostic and Ablation Studies

Appendix C tests individual L7 difficulty levers, control-loop batching, and diagnostic interventions to characterize their effects on performance.

Appendix C

  • −10.2 to −12.5 points: +rot, +blueprint, and +occ produce the largest score drops among single-factor L7 levers.Single-factor tool-frame control (+tool) increases mean score on M5-L5 for GPT-6 Astra and Gemini 4 Argon by aligning rotations with the heart’s symmetry axis.Source: Appendix C · p. 26
  • Table C.2 compares batched open-loop control (cap5) with single-action closed-loop control (cap1) across L5 spatial and temporal tasks.The supplied passage identifies the comparison but does not report its outcomes.Source: Table C.2 · p. 26
  • Table C.3 breaks down +plan, +state, nonote, and nosplit diagnostics by model, excluding pairs with abandoned runs.Claude Opus 5.5 has six tasks for main, +plan, and +state, and two tasks for nonote and nosplit.Source: Table C.3 · p. 26

Appendix D Per-Model Trajectory and Behavioral Signatures

Appendix D defines behavioral metrics for batched-protocol trajectories and notes a model-specific exclusion in the reported table.

Appendix D

  • Table D.1 summarizes per-model behavioral metrics across 38 batched-protocol tasks, including action batching, camera use, collision responses, and persistent-note updates.It also tracks repeated collisions, camera inspections after collisions, and mean batch size before and after a collision.Source: Appendix D · p. 26, Table D.1 · p. 27
  • Claude Opus 5.5 has no unsolved trajectory in Table D.1 because its three unsolved runs were abandoned.The table’s metrics include the percentage of collision turns immediately followed by another collision and the share followed by camera inspection.Source: Appendix D · p. 26, Table D.1 · p. 27

Appendix E Robustness Across Random Seeds and Rule Perturbations

Appendix E examines cross-seed stability and rule-perturbed puzzle variants, with the reported seed analysis limited to families completed by every run.

Appendix E

  • Standard errors across seeds are at most 1.98 points, and model rankings remain invariant across seeds 100, 101, and 102 on four L5 families.The four families are M1, M3, A1, and B4, the families on which every run completed.Source: Appendix E · p. 27, Table E.1 · p. 27
  • Table E.2 compares standard and rule-perturbed puzzle variants across all 10 frontier MLLMs.The supplied passages identify the comparison but do not report its outcomes.Source: Table E.2 · p. 27, Appendix E · p. 27
  • Tower of Hanoi standard and restricted-disk variants are labeled as auxiliary and outside the benchmark.Source: Appendix E · p. 27

Related papers

Loading 2610.38405v1…