PaperlayerView exact revision on arXiv

Source-linked AI summary

SceneActBench: Can Agents Act on the 3D Scenes They See?

Yifei Zhao, Xiangxin Zhou, Wenhao Yang, Jiaqi Tang, Pu Jian, Huanjin Yao, Jiarui Yao, Haowei Lin, Chunchao Guo, Zhuo Chen, Wenkai Lyu, Jianzhu Ma, Xueqian Wang, Wenxi Zhu

arXiv:2607.22393v1cs.AIcs.CV

TL;DR

Existing 3D benchmarks rarely test whether agents can coordinate actions across complete multi-object scenes. SceneActBench evaluates visually conditioned action across five tasks in a unified agent loop, finding that no configuration performs consistently well across tasks, with Overall scores of 38.6-50.2.

  • Problem

    Existing benchmarks mainly score text responses or single-object and static edits, leaving coordinated action on complete multi-object 3D scenes insufficiently evaluated.

  • Method

    SceneActBench evaluates agents acting from images or video frames on 3D scenes across five tasks using hidden geometric ground truth and one fixed agent loop.

  • Results

    No configuration performs consistently well across tasks, while Overall scores range from 38.6 to 50.2.

  • Takeaways & Limitations

    The results indicate that acting on 3D scenes requires several distinct capabilities rather than a single solved skill.

  • Takeaways & Limitations

    The main study uses proprietary VLM configurations and one completed run per configuration–case pair, so Overall scores are not repeated-run estimates.

Abstract

from arXiv · show

Vision-language model (VLM) agents increasingly use tools to act on 3D scenes rather than only describe them. Existing 3D benchmarks score textual responses or single-object operations, leaving agent action on complete multi-object 3D scenes under evaluated. We present SceneActBench, a benchmark for visually conditioned action across five 3D tasks under a unified agent-environment loop. Given PNG images or sampled video frames and, where applicable, supplied 3D assets, an agent acts on a 3D environment. We evaluate each final output against hidden ground truth with task-specific geometric metrics. SceneActBench comprises five tasks built from 210 source instances, yielding 520 task cases including paired input conditions. Every task runs through one fixed agent loop to keep the comparison fair. Across eleven proprietary VLM configurations, Overall scores span 38.6-50.2, and none performs consistently well across tasks. We further analyse where and how failures manifest.

1 INTRODUCTION

SceneActBench evaluates whether VLM agents can act on complete 3D scenes from visual input, addressing benchmarks that mainly score text answers, single-object edits, or task success. It uses a unified executable setup with hidden 3D ground truth and reveals broad variation across eleven VLM configurations.

  • Acting on 3D scenes through tools is a stronger test of 3D understanding than describing them.
  • Existing benchmarks often score text answers without changing scenes or evaluate single-object and static edits, leaving full-scene action untested.
  • SceneActBench evaluates visually conditioned 3D action from images or sampled video frames, matching agent-edited scenes against references.The benchmark draws on 210 source instances: 100 furnished rooms, 100 articulated objects, and 10 multi-object dynamic scenes.
  • Overall scores range from 38.6 to 50.2 across eleven proprietary VLM configurations, whose stacked task contributions show different strengths.

2 RELATED WORK

Prior 3D benchmarks largely score textual understanding, while other systems act in 3D environments, generate content, or solve individual operations with task-specific evaluations. SceneActBench addresses this fragmentation with a shared agent–environment loop and hidden geometric scoring across task-specific outputs.

  • 3D understanding benchmarks: Many 3D benchmarks evaluate questions over scans or rendered images using text answers, despite evidence that fluent descriptions can miss spatial judgments.Examples include ScanQA, SQA3D, Spatial457, SpatialVLM, and BLINK.
  • Agents that act in 3D environments: Prior agents act in interactive 3D worlds, ground robot manipulation, or generate scenes and shapes, but target different outputs with task-specific evaluations.Examples include Voyager, VoxPoser, SceneCraft, 3D-GPT, and CAD-Coder.
  • SceneActBench’s unified evaluation: SceneActBench uses one shared inspect–act–render loop and evaluates final task-specific outputs once against hidden 3D ground truth.Inputs are standardised PNG views or sampled video frames, with anonymised GLB assets when applicable; outputs include GLB scenes, animated GLBs or scene states, and JSON camera poses.
  • Task-specific 3D models: Specialist 3D models separately address layout generation, image-to-3D reconstruction, or camera estimation rather than a unified set of scene operations.Representative systems include ATISS, DiffuScene, LayoutGPT, Holodeck, Zero-1-to-3, LRM, TRELLIS, COLMAP, and DUSt3R.

3 BENCHMARK

SceneActBench evaluates whether agents can convert visual evidence into executable 3D outputs through a shared Blender tool loop, with final outputs compared against hidden 3D ground truth. Its benchmark spans spatial grounding, egocentric camera reasoning, articulated motion reproduction, and 3D shape imagination using standardized assets and visual references.

  • Unified evaluation loop: Agents control Blender headlessly from PNG images or sampled video frames, exporting JSON or GLB outputs evaluated against hidden 3D ground truth.GLB files package geometry, materials, transforms, and animation.
  • Benchmark sources: The benchmark combines 100 furnished rooms, 100 articulated objects, and 10 dynamic scenes standardized from three public sources.Dynamic scenes deliberately vary motion types so no single heuristic covers them all.
  • Layout: Layout agents place standardized furniture at metric world positions and yaw from one or approximately 11 calibrated views, testing spatial grounding with ADD-S surface distance.The same 100 scenes are evaluated under both one-view and approximately 11-view conditions.
  • Camera: Camera agents infer hidden 6-DoF extrinsics from an arranged scene and known field of view, scoring the final pose with Position Error in metres and Angular Error in degrees.Position Error measures camera-centre distance, while Angular Error measures viewing-direction difference and does not penalize roll.
  • Motion and shape: Articulated-object agents recover movable parts, joints, and motion from 32 ordered open–close frames, while shape agents reconstruct and texture complete furniture meshes from multiple room views.Articulated outputs are evaluated with Maximum Part Error, which penalizes geometry error and unreproduced motion range across movable parts.

4 EXPERIMENTS

Experiments evaluate eleven proprietary VLM configurations in a shared Blender-based agent harness across five normalized tasks. Results show narrow aggregate leadership but substantial task-, input-condition-, failure-stage-, and interaction-regime differences.

  • Experimental setup: Eleven configurations from ten proprietary VLMs run through one shared harness driving headless Blender via the same MCP interface.Budgets are 30 steps for Layout and Camera, 60 for Articulated, 35 for Reconstruction, and 80 for Dynamic.
  • Aggregate results: 50.2 Overall makes Doubao the top configuration, followed by Claude Opus at 48.9 and GPT 5.4 Medium at 48.7.The fixed Overall averages normalized task scores; GPT 5.4 High trails Medium by only 0.02 points.
  • Aggregate results: Doubao leads Layout (77.4), Camera (34.5), and Dynamic (70.7), while Claude Opus leads Articulated (63.7) and GPT 5.4 Medium leads Reconstruction (10.4).The three leaders differ by only 1.5 Overall points, motivating case-level analysis.
  • Input-condition sensitivity: 12.1 points is Sonnet’s multi-view Layout gain, compared with 9.4 for Gemini and 8.5 for Claude Opus; Step and MiMo instead lose 1.4 and 0.6 points.Multi-view Layout improved nine of eleven configurations while scenes, target outputs, and evaluators remained fixed.
  • Input-condition sensitivity: 17.3 points is GPT 5.4 High’s gain under photo-realistic Dynamic, whereas Step and Sonnet lose 10.3 and 10.2 points.Four configurations improved, six declined, and Claude Opus was unchanged at one-decimal precision; alternate conditions were excluded from Overall.
  • Failure analysis: 13/391 ground-truth parts are moved by Doubao in Articulated, versus 255/391 for Claude Opus and 132/391 for GPT 5.4 Medium.Although the top-three MPE values are close at 0.375–0.404, later type- and direction-correctness rates differ substantially.
  • Interaction analysis: 39.1 task-balanced calls per run are used by one configuration versus 19.9 by GPT 5.4 Medium and 11.4 by Doubao, despite Overall scores of 48.9, 48.7, and 50.2.Across eleven configurations, interaction volume does not positively track Overall; these values describe different regimes rather than causal effects.

5 LIMITATIONS

SceneActBench is a geometric stress test for multi-object 3D action, but its evaluation has limitations involving proprietary configurations, single-run estimates, and limited dynamic-task coverage.

  • Evaluation scope: The main study evaluates proprietary VLM configurations and uses one completed run per configuration–case pair, so Overall scores are not repeated-run estimates.This design limits interpretation of reported Overall scores as measures of run-to-run performance.
  • Evaluation scope: The Dynamic task contains ten scenes and should be viewed as a targeted stress test rather than a broad estimate of dynamic-scene performance.Its scene count limits the breadth of conclusions about dynamic scenes.
  • Evaluation scope: Overall scores depend on fixed normalisation settings.The supplied passage identifies fixed normalisation as another limitation but truncates the specific setting after “c”.

6 CONCLUSION

SceneActBench evaluates 3D understanding as executable action: agents act on environments and must match hidden ground-truth geometry. Across five tasks and eleven configurations, performance varies by task, indicating that 3D-scene action requires several distinct capabilities.

  • 6 CONCLUSION: SceneActBench tests whether agents can act on 3D environments and match hidden ground-truth geometry.It provides a common basis for measuring executable 3D understanding.
  • 6 CONCLUSION: Across five tasks and eleven configurations, no configuration performs consistently well.Configurations with similar Overall scores can still succeed on different tasks.
  • 6 CONCLUSION: The differing task outcomes suggest that acting on 3D scenes requires several distinct capabilities rather than one solved skill.SceneActBench is positioned as a common measurement basis for future expansion.

A BENCHMARK DETAILS · A.1 3D TERMINOLOGY

This section defines the 3D-specific terminology used throughout SceneActBench. The definitions are scoped to the benchmark’s interface and outputs.

  • A.1 3D TERMINOLOGY: Table 4 summarizes the 3D-specific terms used throughout SceneActBench.The terminology supports consistent interpretation of the benchmark.
  • A.1 3D TERMINOLOGY: The benchmark’s terminology definitions are limited to its interface and outputs.This scope keeps the definitions tied to how SceneActBench operates and is evaluated.

A.2 DATASET CONSTRUCTION … B.2 OVERALL SCORE NORMALISATION

SceneActBench constructs standardized indoor, articulated, dynamic, and reconstructed-scene cases, then evaluates agents through a shared Blender interface using task-specific geometric metrics. Scores normalize required task components into five task means, while selected audits and ablations remain outside Overall.

  • A.2 DATASET CONSTRUCTION: 100 indoor rooms contain 3–7 furniture objects from 27 categories, while 100 articulated objects use anonymized assets without joint metadata and 32-frame open–close sequences.Indoor assets are standardized by centroid, yaw, metric scale, and anonymous naming; articulated objects retain joint information only for evaluation.
  • A.2 DATASET CONSTRUCTION: 10 dynamic scenes combine independent movers with 144-frame, 24-fps videos, asset libraries, animated ground truth, mover coordinates, layouts, and cameras.The scenes cover traffic, racing, rail, boats, conveyors, spinning golf balls, platform jumps, and castle mechanisms.
  • A.3 TASK EVALUATORS: Task evaluators use deterministic point sampling, explicit matching and thresholds, invalid-case rules, and hidden ground truth to compute complete per-case metrics.Surface audits use bidirectional distance, while configuration diagnostics average defined case values arithmetically.
  • A.3.1 LAYOUT; A.3.2 CAMERA; A.3.3 ARTICULATED: Layout scores matched meshes with ADD-S, Camera scores position and angular errors PE and AE, and Articulated scores opening-aligned geometry through MPE.Layout audits include placement and scene success; Camera appearance and FOV quantities are audit-only; Articulated also records fallback part errors and motion diagnostics.
  • A.3.4 RECONSTRUCTION: Reconstruction aligns predicted and ground-truth furniture with fixed-scale ICP, matches objects using DBSCAN and Hungarian assignment, and uses F@5% as its sole Overall component.Region, surface, point-embedding, and visual-render measures are diagnostics rather than Overall inputs.
  • A.3.5 DYNAMIC: Dynamic evaluation aligns and matches tracks, assigns unmatched ground-truth movers error 1, and requires finite MME and LE for valid scoring.The evaluator also reports trajectory, direction, heading, scale, static-layout, and size audits; low-poly Dynamic enters Overall, whereas photo-realistic Dynamic is an ablation.
  • B EVALUATION PROTOCOL; B.1 AGENT INTERFACE AND CONFIGURATION: Each configuration controls one headless Blender through the same Model Context Protocol interface with shared inspection, scripting, rendering, and Dynamic reference-frame tools.The core tools are get scene info, get object info, execute blender code, and render scene view.
  • B.2 OVERALL SCORE NORMALISATION: Overall averages five task scores after q↓ normalization of errors and q↑ normalization of Reconstruction F@5%, with missing required values receiving zero.Camera averages PE and AE, Dynamic averages MME and LE, and reference values are 4 m, 90°, or 1; Multi-view Layout and photo-realistic Dynamic remain outside Overall.

B.3 TASK CORRELATION CALCULATION · C EVIDENCE BEHIND THE MAIN ANALYSIS · C.1 EVIDENCE MAP AND SELECTED DIAGNOSTICS

The paper correlates normalized scores across eleven configurations to characterize task associations, while mapping the main analysis to supporting evidence and denominator-aware diagnostics. These diagnostics distinguish ineligible outputs from zero error and document the conditions required for measurable motion.

  • B.3 TASK CORRELATION CALCULATION: Camera and Dynamic task scores use the same PE/AE and MME/LE averages as Overall, respectively, before task-correlation analysis.The matrix correlates normalized scores, so raw metric units do not affect it.
  • B.3 TASK CORRELATION CALCULATION: The analysis forms Z ∈R11×5 with one five-score row per configuration and computes standard Pearson correlations between task columns.The resulting matrix is symmetric, has diagonal ones, and lies in [−1, 1].
  • B.3 TASK CORRELATION CALCULATION: The task-correlation matrix describes associations among these eleven configurations only, not a population relationship or causal test.This bounds the interpretation of the reported task-score associations.
  • C EVIDENCE BEHIND THE MAIN ANALYSIS: The main Analysis examines ranking determinants, input-condition effects, agent failure locations, and interaction usage through linked supporting evidence.Table 5 links each analysis stage to its supporting evidence.
  • C.1 EVIDENCE MAP AND SELECTED DIAGNOSTICS: Table 6 reports denominator-aware failure funnels as pooled numerator/denominator counts, with complete diagnostic rates retained in Table 7.Table 6 exposes the denominators behind conditional diagnostics for all eleven configurations.
  • C.1 EVIDENCE MAP AND SELECTED DIAGNOSTICS: Articulated type and direction require measurable rigid motion, while Dynamic direction requires matched non-loop movers with target motion.These eligibility rules determine which outputs enter the corresponding diagnostic funnels.
  • C.1 EVIDENCE MAP AND SELECTED DIAGNOSTICS: A 0/0 entry denotes no eligible output rather than zero error, preventing denominator absence from being interpreted as perfect performance.The distinction is explicit in the denominator-aware failure-funnel definitions.

C.2 TASK-LEVEL QUALITATIVE EVIDENCE

The qualitative evidence illustrates how measured failures appear in rendered motion and static outputs. Motion errors affect moving parts and trajectories, while static errors involve layout, camera alignment, and reconstruction detail.

  • Scope: Figures 12 and 13 provide illustrative rendered-error examples rather than population statistics.The main analysis reports the measured failures.
  • Motion tasks: Motion outputs can miss moving panels, move static parts, or recover scenes with incorrect trajectory scale or shape.Figure 12 compares reference frames with agent outputs at matched times for Claude Opus 4.6 High.
  • Static tasks: Static layout failures include collapsed arrangements and object overlap.On DiningRoom-13034, most outputs recover the main arrangement, but MiniMax M3 collapses it.
  • Static tasks: Static camera failures produce wrong crops or headings, while reconstruction often replaces detailed furniture with simple proxy shapes.Bedroom-6995 re-renders vary in crop and heading; Figure 13 shows reconstruction by Doubao Seed 2.0 Pro High.

C.3 COMPLETE METRIC TABLE · C.4 AGENT PROCESS AND BUDGET SENSITIVITY · C.4.1 STEP-BUDGET SENSITIVITY

The complete metrics table records means and sample standard deviations for all eleven configurations, while process analysis examines realised execution, tool calls, and shared-budget behaviour. Increasing MaxSteps improves Overall by 12.0–27.7 points, but sensitivity varies substantially by configuration and task.

  • C.3 COMPLETE METRIC TABLE: Table 7 reports task, primary, diagnostic, and audit metrics for all eleven configurations, with each cell giving the mean and sample standard deviation across benchmark instances.Task-score deviations use per-instance normalised scores; undefined diagnostic values are omitted.
  • C.3 COMPLETE METRIC TABLE: Layout and Dynamic include paired input conditions, while FOV and audit metrics are unscored and N/A denotes unavailable values.Bold identifies the best mean per row.
  • C.4 AGENT PROCESS AND BUDGET SENSITIVITY: The process analysis combines aggregate realised steps, tool calls, shared-budget curves, and a separate full Claude Sonnet 5 High comparison of two complete agent stacks.These analyses evaluate both execution behaviour and agent-stack differences.
  • C.4.1 STEP-BUDGET SENSITIVITY: Checkpoints are retrospectively scored under nominal budgets from 10 to 150 steps, using the same MaxSteps value for every task at each curve point.Each checkpoint applies the same per-case scoring rule.
  • C.4.1 STEP-BUDGET SENSITIVITY: Overall rises by 12.0–27.7 points between the two endpoint budgets, demonstrating substantial aggregate sensitivity to the step budget.The configuration order changes as MaxSteps grows.
  • C.4.1 STEP-BUDGET SENSITIVITY: 51.3 points is Camera’s largest mean gain, although its configuration-level change ranges from 15.7 to 79.2.Camera therefore varies far more across configurations than its mean gain alone suggests.
  • C.4.1 STEP-BUDGET SENSITIVITY: Articulated changes little, ranging from −3.0 to +6.1, whereas Reconstruction gains 2.7–8.8 points across the budget endpoints.These task ranges show that budget effects are not uniform across tasks.
  • C.4.1 STEP-BUDGET SENSITIVITY: Budget sensitivity is configuration- and task-dependent, with configuration rankings changing as MaxSteps increases.Figure 14 presents Overall and task scores across MaxSteps values with direct configuration labels in panel (a).

C.4.2 OFFICIAL CLI COMPARISON

Under identical tasks and scoring, Claude Sonnet 5 High scored 34.6 Overall with the official Claude Code CLI versus 39.5 under the shared harness. The comparison shows task-dependent stack sensitivity rather than a uniform interface advantage, and does not isolate a single causal interface factor.

  • Overall comparison: 34.6 Overall official-CLI score versus 39.5 under the shared harness, a 4.9-point gap.All 520 cases completed and were scored; paired conditions remained outside Overall.
  • Task-level results: The official stack improves Camera by 3.4 points and Articulated by 2.5, while the shared harness leads Reconstruction by 2.2 and Dynamic by 22.9.Five missing official animations contribute to the Dynamic gap.
  • Task dependence: The comparison shows no uniform interface advantage: the official stack performs better on hidden camera state and articulated motion but worse on multi-view Layout and multi-object Dynamic.The larger multi-view and Dynamic differences are consistent with different allocations of visual reads, edits, and checks, but do not isolate one causal mechanism.
  • Effective interaction budget: The same MaxSteps value does not guarantee the same number or sequence of environment actions because the stacks package reading, editing, rendering, context management, and stopping differently.Scores therefore compare complete agent stacks rather than an isolated interface component.
  • Scope: Both runs used Claude Sonnet 5 High but differed in image reading, system prompts, and context policies, so the experiment measures stack sensitivity rather than one causal interface factor.This motivates using one common stack for the main ranking.

C.4.3 REPRESENTATIVE INTERACTION TRACES · D PROMPT TEMPLATES

The section illustrates how agents spend interaction budgets through three representative Blender episodes, then specifies the task prompts and operating constraints for layout, camera, articulation, reconstruction, and dynamic-scene tasks.

  • C.4.3 REPRESENTATIVE INTERACTION TRACES: Three scored traces expose distinct processes: a short successful loop, an unchecked completion, and a long unresolved loop.The traces record input reads, code edits, render checks, tool errors, stopping events, and exact episode totals.
  • C.4.3 REPRESENTATIVE INTERACTION TRACES: Movable recall 1.00 and MPE 0.021 follow one early code error, four render checks, and self-stopping at step 19.The episode is shown as a short closed loop in panel 15a.
  • C.4.3 REPRESENTATIVE INTERACTION TRACES: Direction error rate 1.00 and scale error 0.889 occur after six code edits, no render checks, and self-stopping at step 8.Panel 15b demonstrates a completed but unchecked run.
  • D PROMPT TEMPLATES: The prompt harness sends a system message defining role, world rules, and tools, plus a task message containing scene inputs and instructions with runtime-filled placeholders.The quoted prompts preserve harness wording and distinguish system and task cards.
  • D.1 LAYOUT: Layout prompts require reconstructing every object’s position and Z-axis rotation from one or multiple reference views without scaling, using Blender transforms and render-based checks.Agents inspect geometry, place objects with obj.matrix_world = T @ R, and use extra cameras to expose depth and overlap errors.
  • D.2 CAMERA: Camera prompts start from a correctly arranged scene and require iteratively matching an unknown camera pose to one reference image, while PE and AE alone determine the reported Camera score.LPIPS, CLIP, and PSNR are retained only as audit checks.
  • D.3 ARTICULATED: Articulated prompts provide 32 ordered frames of an unlabeled object opening and closing, requiring inference of moving meshes and one GLB export per frame.The agent must inspect the scene and reproduce the complete closed-to-open-to-closed motion.
  • D.4 RECONSTRUCTION / D.5 DYNAMIC: Reconstruction prompts start from an empty Blender scene and require every furniture piece’s geometry, color, placement, and visible structure, while dynamic prompts import components and animate vehicles into one GLB.Both tasks require reference-driven iteration; dynamic prompts use keyframes for trajectories, and the photo-realistic variant changes reference style but not the task or output.
Loading 2607.22393v1…