PaperlayerView exact revision on arXiv

Source-linked AI summary

From Perception to Action: An Interactive Benchmark for Vision Reasoning

Yuhao Wu, Maojia Song, Yihuai Lan, Lei Wang, Zhiqiang Hu, Yao Xiao, Heng Zhou, Weihua Zheng, Dylan Raharja, Soujanya Poria, Roy Ka-Wei Lee

arXiv:2602.21015v1cs.CV

TL;DR

Existing VLM evaluations largely use static, single-turn formats that do not test interactive physical reasoning. CHAIN introduces a closed-loop 3D physics benchmark, revealing that models often fail to maintain coherent multi-step strategies under structural constraints.

  • Problem

    Current VLM evaluations largely use static, single-turn formats, leaving agents’ ability to plan and adapt actions as physical constraints evolve untested.

  • Method

    CHAIN evaluates structured physical reasoning through closed-loop 3D tasks requiring models to observe outcomes, choose feasible interactions, and revise plans.

  • Results

    Models’ performance degrades sharply as structural constraints tighten, with many failing to maintain coherent multi-step strategies and translate perceived structure into effective actions.

  • Takeaways & Limitations

    CHAIN exposes a persistent gap between perceiving physical structure and acting reliably when early actions reshape future feasible actions.

  • Takeaways & Limitations

    CHAIN’s scale is limited, especially for interlocking puzzles, because high-fidelity environments require substantial manual modeling and debugging.

Abstract

from arXiv · show

Understanding the physical structure is essential for real-world applications such as embodied agents, interactive design, and long-horizon manipulation. Yet, prevailing Vision-Language Model (VLM) evaluations still center on structure-agnostic, single-turn setups (e.g., VQA), which fail to assess agents' ability to reason about how geometry, contact, and support relations jointly constrain what actions are possible in a dynamic environment. To address this gap, we introduce the Causal Hierarchy of Actions and Interactions (CHAIN) benchmark, an interactive 3D, physics-driven testbed designed to evaluate whether models can understand, plan, and execute structured action sequences grounded in physical constraints. CHAIN shifts evaluation from passive perception to active problem solving, spanning tasks such as interlocking mechanical puzzles and 3D stacking and packing. We conduct a comprehensive study of state-of-the-art VLMs and diffusion-based models under unified interactive settings. Our results show that top-performing models still struggle to internalize physical structure and causal constraints, often failing to produce reliable long-horizon plans and cannot robustly translate perceived structure into effective actions. The project is available at https://social-ai-studio.github.io/CHAIN/.

1. Introduction

CHAIN addresses the limits of static, single-turn VLM evaluations by introducing an interactive 3D benchmark for closed-loop physical problem solving. It evaluates whether models can understand, plan, and execute structured action sequences under geometric, contact, and support constraints, while showing that current models struggle to do so reliably.

  • Motivation: Structured 3D physical problem solving requires coupling perception, action, and iterative feedback while anticipating feasible actions under hidden geometric and contact-driven constraints.Objects must be manipulated, configurations explored, and outcomes assessed under feasibility constraints.
  • Motivation: Current VLM evaluations largely use static, single-turn formats such as VQA, leaving multi-step planning and adaptation under evolving constraints untested.These evaluations primarily measure the correctness of a final textual answer.
  • CHAIN benchmark: CHAIN is an interactive 3D, physics-driven benchmark that evaluates models by requiring them to understand, plan, and execute structured action sequences grounded in physical constraints.The benchmark shifts evaluation from passive perception to active problem solving in a physics-engine-driven environment.
  • Findings: Current VLMs and diffusion-based models struggle to internalize physical structure and constraint dependencies, maintain coherent multi-step strategies, and translate perceived structure into effective extended-interaction actions.The study evaluates state-of-the-art VLMs and diffusion-based models under a unified interactive setting.
  • CHAIN benchmark: CHAIN comprises 109 distinct interactive levels with clear difficulty separation, testing hidden geometric constraints, contact-induced dependencies, and multi-body support relations over multi-step interaction.Agents must infer and exploit physical structure to select feasible actions.

2. CHAIN Benchmark

CHAIN is an interactive 3D benchmark for evaluating VLM physical reasoning through multi-step puzzles and spatial packing tasks. Its controllable environments, closed-loop interaction protocol, and success, plan-efficiency, and cost-efficiency metrics assess both correctness and realistic execution efficiency.

  • Task Families: CHAIN evaluates two task families: interlocking mechanical puzzles requiring ordered fine-grained manipulation and 3D stacking tasks requiring geometric packing into fixed containers.The puzzle suite contains 32 instances, while the stacking suite contains 77 instances with easy, medium, and hard difficulty levels.
  • Benchmark Construction: The benchmark construction pipeline filters for sequential causal dependencies, builds standardized Unity or Python physical environments, and supports controlled, reproducible evaluation.Unity handles complex interlocking mechanics, while the lightweight 3D Python engine supports other environments.
  • Evaluation Protocol: Each episode initializes a predefined environment and uses closed-loop interaction, providing task instructions, interaction history, and current multi-view observations before each action.Agents select actions from a predefined action set while interacting over multiple steps.
  • Evaluation Metrics: CHAIN measures task success, plan efficiency, and cost efficiency to evaluate whether agents solve tasks, avoid redundant actions, and limit token and dollar expenditure.Plan efficiency is conditioned on solved tasks, and cost accounting includes billable prompts, tool interactions, and model replies across interaction rounds.

3. Experiments

Experiments evaluate diverse VLMs and world models under unified interactive settings, revealing persistent failures in physical-structure reasoning, long-horizon planning, and constraint-driven action. Interaction helps modestly, but current models remain unreliable on difficult puzzles and complex stacking tasks.

  • Experimental setup: The evaluation uses a shared action API and system-executed atomic actions to isolate operation-level decisions under a unified interaction protocol.All models use temperature 0.6, top-p = 0.95, and a 30–60-step interaction budget per instance.
  • Overall model performance: GPT-5.2 leads overall with Pass@1=22.9 and Succ Task=25, while also achieving the highest Stacking score (31.2).Kimi-k2.5 is the strongest open-source model, with Pass@1=13.8 and Succ Task=15.
  • World-model evaluation: World models fail to produce physically valid Luban-lock disassemblies, violating interlocking constraints or hallucinating structural changes, with failures worsening at higher complexity.SORA 2 and WAN 2.6 often directly translate target beams despite infeasible interlocking, while VEO 3.1, KLING 2.6, and HUNYUANVIDEO 1.5 corrupt object structure and identity.
  • Difficulty analysis: Stacking performance declines sharply with difficulty: GPT5.2 and Claude-Sonnet-4.5 reach 100.0% on Easy, while the best Hard result is only 6.3%.Stacking-Mid separates the models, with GPT5.2 at 55.0% versus Claude-Sonnet-4.5 at 20.0%.
  • Difficulty analysis: Puzzle is substantially harder than Stacking, with models peaking at 10.0% on Puzzle-Easy and reaching 0.0% on Puzzle-Mid/Hard.The bottleneck is inference of hidden blocking constraints and feasible multi-step disentanglement from partial observations.
  • Interaction ablation: Interaction improves over one-shot reasoning, but Puzzle accuracy remains only 3.1% interactively versus 0.0% for every evaluated model one-shot.This indicates that even limited success requires iterative constraint discovery rather than a fully inferred initial disassembly plan.
  • Inference strategies: Pass@1 and Avg@4 are nearly identical overall, while stronger selection signals provide limited gains for long-horizon interactive planning.RM reranking improves by +0.6 and a stronger VLM judge by +1.3, but both trail Pass@2; more reliable verifier signals appear necessary.

4. Related Work

Existing VLM and physical-reasoning evaluations largely emphasize static scenes, single-step inference, or snapshot-based perception. CHAIN addresses this gap by evaluating structured reasoning about evolving spatial configurations and multi-round physical interactions.

  • Reasoning of Vision-Language Models: VLMs remain largely limited to static scenes and single-step inference, while embodied extensions frequently fail in multi-round physical interactions.The passage contrasts visual-language alignment with limited temporal dynamics and unreliable performance during repeated physical interaction.
  • Physical Benchmarks: Physical reasoning benchmarks have progressed from visual plausibility and synthetic primitives to commonsense question answering and perception-centered tasks, but remain largely static.These benchmark families evaluate physical reasoning in simplified, controlled, commonsense, or perception-oriented settings without substantial dynamical interaction.
  • 3D Structure Perception: 3D structure perception advances reasoning about depth and relative positioning through multi-view and point-cloud representations, yet benchmarks rarely evaluate evolving spatial configurations.Existing efforts primarily prioritize static scene reconstruction or snapshot reasoning rather than changes in spatial structure over time.

5. Conclusion

CHAIN evaluates vision-language models through closed-loop, multi-step physical reasoning in interactive 3D tasks. Results reveal a persistent gap between perceiving scenes and executing coherent actions under structural and causal constraints.

  • Benchmark and findings: CHAIN shifts vision-language evaluation from passive perception to closed-loop, multi-step physical reasoning across interlocking puzzles and 3D stacking/packing.Models repeatedly observe outcomes and choose actions in interactive 3D environments.
  • Benchmark and findings: Performance degrades sharply as structural constraints tighten, with many models failing to maintain coherent multi-step strategies despite correct scene perception.The benchmark exposes failures in translating perceived structure into sustained action sequences.
  • Benchmark and findings: Interlocking mechanical puzzles cause near-total collapse because hidden geometric constraints challenge current models’ physical reasoning.Stacking tasks additionally reveal brittle stability and long-horizon feasibility reasoning.
  • Benchmark and findings: Current models rarely internalize how early actions reshape the future feasible action space.This limitation captures the persistent gap between seeing and acting in dynamic physical environments.

A. Appendix · A.1. Limitations

CHAIN’s limitations are its finite environment coverage, especially for interlocking puzzles, and its reliance on Pass@1 under costly multi-step interaction. Stacking complexity can scale programmatically, but high-fidelity puzzle construction remains engineering-intensive, while evaluations are capped at 30–60 steps per instance.

  • A.1. Limitations: CHAIN contains 32 Puzzle instances and 77 Stacking instances, limiting the current scale of interactive environments.The benchmark’s finite instance set is identified as a main limitation.
  • A.1. Limitations: The scale limitation is especially pronounced for interlocking puzzles.These puzzles require high-fidelity environment construction to capture kinematic feasibility and contact-rich behavior.
  • A.1. Limitations: Stacking instances are programmatically generated and can scale to near-unlimited complexity.This scalability contrasts with the finite current benchmark inventory.
  • A.1. Limitations: High-fidelity interlocking puzzles require careful environment engineering to faithfully capture kinematic feasibility and contact-rich interactions.The passage identifies environment engineering as the constraint on scaling these puzzles.
  • A.1. Limitations: CHAIN mainly reports Pass@1, defined as single-attempt success, because multi-step interactive evaluation is costly.Each episode involves a non-trivial number of interaction rounds.
  • A.1. Limitations: The evaluation budget is capped at 30–60 steps per instance.The cap is motivated by the cost of closed-loop interaction across multi-round episodes.
  • A.1. Limitations: Interactive tasks can exhibit run-to-run variability, which the Pass@1-focused evaluation does not fully characterize.The authors note that preliminary analysis in Section 3.6 addresses this issue.

A.2. Model list

The benchmark evaluates a broad pool of state-of-the-art multimodal LLMs spanning closed-source APIs and open-source checkpoints, with an optional lightweight Seed1.6-Flash variant.

  • Model list: The closed-source evaluation set includes GPT5.2, o3, Opus4.5, Sonnet4.5, Gemini3-Pro (preview), Gemini3-Flash (preview), and Seed1.6.Seed1.6-Flash is included as an optional lightweight variant.
  • Model list: The open-source evaluation set consists of Qwen3VL-235B (A22B, think), Qwen3VL-30B (A3B, think), Qwen3VL-8B (think), Qwen3VL-2B (think), GLM4.6V, MiMo-v2-Flash, and Kimi-VL-A3.

A.3. Task Examples

The benchmark presents puzzle and stacking examples organized into three difficulty levels. Puzzle difficulty depends on piece count and interlock complexity, while stacking difficulty depends on container size and piece count and shape diversity, which determine the search space.

  • Puzzle Examples: Puzzle difficulty is divided into three levels according to the number of pieces and the complexity of interlocks.This categorization is shown in Fig. 5.
  • Puzzle Examples: Easy puzzles use few pieces, simple interfaces, and weak interlocking, enabling distinctive placement and short, low-risk disassembly.These puzzles involve few steps and a low risk of dead-ends.
  • Puzzle Examples: Middle-level puzzles contain more pieces and complex contact surfaces, requiring careful moving-direction and intermediate-order selection.Interlocking features occur more frequently at this level.
  • Puzzle Examples: Hard puzzles have many mutually interlocked pieces, requiring long-horizon planning and constraint propagation across the entire disassembly.Multiple pieces may jointly constrain removal.
  • Stacking Examples: Stacking difficulty is categorized by container size and the number and shape diversity of pieces, which jointly determine the combinatorial search space.This categorization is shown in Fig. 6.
  • Stacking Examples: Easy stacking uses a small container such as 2×2×3 with few, relatively simple pieces, producing a small solution space.Feasible placements are constrained.
  • Stacking Examples: Middle-level stacking uses a medium container such as 2×3×3, increasing spatial freedom and placement permutations while requiring partial-fill reasoning to avoid unreachable cavities.The agent must reason carefully about partial fills.
  • Stacking Examples: Hard stacking uses a large container such as 3×3×4 and more pieces, making placement order and orientation highly ambiguous and sensitive to early decisions.This setting is highly combinatorial.

A.4. Case Analysis of Stacking Task

The stacking analysis contrasts a successful structure-first trajectory with a failed trajectory caused by an early irreversible commitment. Success comes from building a stable base and systematically filling space, whereas failure produces constrained placements and awkward residual space.

  • Successful trajectory: The successful agent first places compact blocks to form a stable, flat base before stacking upper layers layer by layer.This structure-first sequence spans Steps 1–3.
  • Successful trajectory: After expanding the base, the agent fills the remaining volume systematically without unsupported overhangs or isolated cavities.Subsequent placements occur in Steps 5–7 after base expansion in Step 4.
  • Failed trajectory: The failed agent begins with a tall vertical column before establishing a complete foundation, constraining later placements to wrap around it.Although it rolls back some actions and tries new placements, the partially built structure encloses awkward residual space.

A.5. Polycube Stacking Puzzle Generation Pipeline

The pipeline generates stable, solvable, and de-duplicated 3D polycube stacking and assembly puzzles within an a × b × c voxel box through sample–verify candidate generation and hard validity checks.

  • Generation and validation: Candidate connected-piece partitions are sampled, then filtered for exact-cover solvability, structural constraints, and optionally linear assembly feasibility.The generator uses a sample–verify design that applies these checks as hard validity requirements.
  • Generation and validation: Piece shapes are canonicalized under rigid rotations and hashed to enforce strong de-duplication.This canonicalization supports removing duplicate puzzle configurations while preserving rotational equivalence.
  • Generation and validation: The resulting puzzles are designed to be stable, solvable, and de-duplicated within the voxel box.These are the pipeline’s stated generation objectives.

A.5.1. OBJECTIVES AND CORE CONSTRAINTS … A.6. Prompt details

The benchmark generates voxel puzzles that satisfy exact-cover, connectivity, size, structural, contact, and optional linear-assembly constraints. It varies partition difficulty, verifies candidates with DLX and physical checks, de-duplicates accepted instances, stores structured outputs, and specifies physically valid disassembly-video prompts.

  • A.5.1. OBJECTIVES AND CORE CONSTRAINTS: A solvable puzzle partitions box B into pieces whose placements cover every voxel exactly once, using rotations and translations within the box.Each piece must also be 6-neighbor connected and obey the specified minimum and maximum cell-count bounds.
  • A.5.1. OBJECTIVES AND CORE CONSTRAINTS: Structural constraints exclude filled 2 × 3 axis-aligned faces, isolated pieces, and, optionally, instances without collision-free linear assembly.Optional assembly translates one piece at a time along one of the six axis directions without intersecting the remaining pieces.
  • A.5.2. CANDIDATE PARTITION SAMPLING BY DIFFICULTY: Difficulty modes sample regular cuboids for easy instances, planar connected growth for mid instances, and unconstrained connected growth with rotational shape uniqueness for hard instances.Hard-mode sampling also uses solver visited nodes as a difficulty proxy and enforces a mode-specific threshold.
  • A.5.3. EXACT-COVER SOLVABILITY VIA DLX: DLX validates packability by encoding candidate placements as exact-cover rows over voxel columns and one-use-per-piece columns.The solver caches placements, anchors the fewest-placement piece at the origin, and uses an MRV-style column heuristic.
  • A.5.4. PHYSICAL FEASIBILITY AND LINEAR ASSEMBLY EXTRACTION: Physical verification rejects packed solutions with isolated pieces and, when required, greedily extracts a six-direction linear disassembly sequence.The REMOVABLE predicate rejects a candidate if no remaining piece can translate one voxel step without collision or leaving the box.
  • A.5.5. STRONG DE-DUPLICATION BY SHAPE MULTISET CANONICALIZATION: Puzzle de-duplication canonicalizes each piece across 24 rigid rotations, sorts the canonical shapes into a multiset signature, and hashes that signature.This prevents repeated puzzle templates while treating rotationally equivalent piece shapes as identical.
  • A.5.6. END-TO-END BATCH GENERATION AND OUTPUTS: For every box size and difficulty mode, accepted puzzles are saved with JSON manifests containing pieces, packed solutions, assembly orders, difficulty statistics, and signatures, plus 3D visualizations.The staged sampler can relax the minimum piece size from 4 to 3 and accepts candidates only after structural, solvability, contact, optional assembly, and hard-mode checks.
  • A.6. Prompt details: The world-model prompt targets physically valid disassembly videos of Kongming/Luban locks with rigidity, collision avoidance, continuous motion, and axis-aligned sliding constraints.Figure 9 presents the prompt used for interlocking burr-puzzle disassembly generation.
Loading 2602.21015v1…