PaperlayerView exact revision on arXiv

Source-linked AI summary

SceneAgent: An Agentic Pipeline from 3D and 4D Captures to Simulation Environments

Luke Hollis, Tianxing Fan, Heng Yang

arXiv:2610.34745v1cs.ROcs.AIcs.CV

TL;DR

Visual captures show scene appearance, but converting them into interactive simulations requires object reconstruction, physical modeling, and correction of intermediate errors. SceneAgent coordinates these steps through simulator-based inspection and repair, and reports improved static reconstruction and dynamic-scene IoU, alongside simulated interactions and preliminary real-world deployment.

  • Problem

    Visual reconstruction alone does not provide the object geometry, physical parameters, or articulation required for interaction, and errors between conversion stages can yield unusable objects.

    Source: §1 · p. 2

  • Method

    SceneAgent coordinates reconstruction and physical modeling tools across 3D and 4D captures, inspecting and repairing rendered outputs in the target simulator.

    Source: §1 · pp. 2–3

  • Results

    SceneAgent improves static-scene F1 at 1 cm for every harness with scorable paired outputs and raises Gemini 4 Argon’s mean dynamic IoU from 0.86% to 99.09% on three 4DCodeBench scenes.

    Source: §1 · p. 3, §6 · p. 22

  • Takeaways & Limitations

    The converted environments support simulated interactions and preliminary real-world deployment after policy fine-tuning, while digital sisters and hybrid scenes extend environments for policy training and evaluation.

    Source: §6 · p. 22

    The reconstruction study uses one kitchen scene and one cluttered bin scene, while the 4DCodeBench study covers three synthetic scenes; real-world 4D videos remain unevaluated.

    Source: §5 · p. 21

Abstract

from arXiv · show

Visual captures provide detailed scene appearance, but converting them into interactive simulation environments requires object reconstruction, physical modeling, and correction of intermediate errors. We present SceneAgent, an agentic pipeline that coordinates these steps through simulator-based visual inspection and repair. The pipeline processes 3D Gaussian Splatting (3DGS), photogrammetry, LiDAR captures, AI-generated 3DGS scenes, and 4D videos of dynamic scenes, and combines object segmentation, physical parameter estimation, part decomposition, articulation, and deformable simulation. To support policy learning, SceneAgent generates diverse object variants ("digital sisters") and scripted demonstrations. On a cluttered bin scene from GraspClutter6D, SceneAgent raises F1 at 1 cm for every coding agent with scorable outputs both with and without its skills, and its reconstructions are better than SimFoundry: all agent configurations exceed SimFoundry's F1 score, with a best of 60.75% versus 6.9%. On three dynamic scenes from 4DCodeBench, SceneAgent raises the mean dynamic IoU of Gemini 4 Argon from 0.86% to 99.09%. We further demonstrate diverse simulated interactions and preliminary policy transfer to a real robot.

1 Introduction

SceneAgent addresses the challenge of turning visual captures into usable interactive simulations by coordinating reconstruction and physical modeling with simulator-based inspection and repair. Its introduction reports improvements on static and dynamic reconstruction tasks and describes tools for expanding scenes for policy training.

1 Introduction

  • Visual reconstruction alone lacks the object geometry, physical parameters, and articulation needed for interaction, while errors between conversion stages can make outputs unusable.Examples include an incorrect mask, scale, or transform despite plausible individual model outputs.Source: §1 · p. 2
  • SceneAgent coordinates reconstruction and physical modeling across 3D and 4D captures, using simulator-rendered inspection and repair to correct visible errors.The resulting environments can include segmented objects, estimated physical parameters, decomposed parts, articulation, and soft-body behavior.Source: §1 · pp. 2–3
  • SceneAgent raises F1 at 1 cm for every coding-agent harness with scorable paired outputs and substantially outperforms SimFoundry on the evaluated reconstructions.The evaluation covers a cluttered bin scene and a Messy Kitchen dataset.Source: §1 · p. 2
  • 99.09% mean dynamic IoU, up from 0.86%, for Gemini 4 Argon on three 4DCodeBench scenes with SceneAgent.The accompanying figure compares dynamic IoU against tokens per task on synthetic scenes.Source: §1 · p. 3, Figure 3 · p. 3
  • SceneAgent creates related object variants and hybrid scenes, and its contributions include an automated loop that converts real-robot policy failures into simulated training data.The introduction also describes qualitative interaction examples and preliminary real-world policy transfer.Source: §1 · p. 3

2 Related Work

Related work spans real-to-sim reconstruction, demonstration generation and scene variation, and methods for estimating physical properties or modeling articulation and deformation. SceneAgent focuses on coordinating conversion tools across inputs and inspecting and repairing their outputs.

2 Related Work

  • Real-to-sim systems combine scene appearance with interactive geometry; PolaRiS evaluates simulated policy performance, while SimFoundry reconstructs scenes and supports rigid and articulated objects.GASE and Mirage2Matter also combine Gaussian-splat appearance with interactive object geometry.Source: §2 · pp. 3–4
  • SceneAgent emphasizes coordinating reconstruction and physical-modeling tools across input types and using agents to inspect and repair their outputs throughout conversion.Unlike generated-video evaluation approaches, it constructs explicit geometry, physical parameters, and joints for execution in a physics simulator.Source: §2 · pp. 4–5
  • Demonstration-generation methods use scans, videos, or reconstructed environments to expand training data, while SceneAgent samples layouts, executes a scripted expert, and retains successful episodes.The described workflow settles objects under physics before executing the expert using simulator state.Source: §2 · p. 4
  • Prior work addresses unobserved physical properties through material labels, interaction-based estimates, differentiable simulation, or video-based deformable-dynamics estimation.4DCodeBench evaluates agents that reconstruct dynamic scenes from video as executable simulation and rendering programs.Source: §2 · p. 4
  • SceneAgent uses semantic features and VLM estimates for approximate, physically plausible parameter values rather than aiming to recover accurate physical parameters from 3DGS.These values are intended as starting points for domain randomization.Source: §2 · p. 5

3 SceneAgent

SceneAgent converts captures into interactive simulation environments through agent-guided reconstruction, simulator inspection, and repair. Its workflow spans input preparation, scene assembly, and iterative review of objects and transforms.

3.1 Agent Orchestration and Verification

  • Agents render assembled scenes in the target simulator to find and correct errors in scale, orientation, position, segmentation, or articulation.Review can include object registration overlays, isolated views, and views within the assembled scene; detected problems trigger a proposed correction and another review.Source: §3.1 · pp. 5–6
  • An orchestrator plans conversion, ingest agents prepare inputs, and a runner executes tools with visual review before packaging and inspection in Isaac Lab.Object-, scene-, and task-variation skills then produce digital sisters and alternative layouts.Source: §3.1 · p. 6

3.2 Input Preparation and Scene Representation

  • For RGB sequences, COLMAP estimates camera poses and Nerfstudio trains a 3DGS reconstruction; the pipeline also accepts existing 3DGS, photogrammetry, LiDAR, generated scenes, and dynamic videos.Agents can inspect and correct capture-processing issues such as per-image color calibration.Source: §3.2 · p. 7

3 SceneAgent

  • The conversion produces a completed background and objects represented by Gaussians, meshes, transforms, physical properties, and optional joints.SceneAgent organizes its work into reconstruction and repair, scene augmentation, and demonstration generation, with a separate pathway for dynamic 4D videos.Source: §3 · p. 5, §3.2 · p. 7

3.3 Object Reconstruction and Background Completion

SceneAgent reconstructs objects and completes exposed backgrounds, then estimates physical properties and adds object parts, articulation, or deformable behavior for simulation. The workflow combines learned and VLM-based estimates with simulator review.

3.3 Object Reconstruction and Background Completion

  • Object masks are grouped using shared 3D Gaussians, and SAM3D generates geometry that is aligned to each object's Gaussians.The Gaussian assignment requires projection inside a mask and proximity to the front surface; exposed background regions are completed using detected planes and optionally LaMa inpainting.Source: §3.3 · p. 8

3.4 Physical Properties, Parts, and Articulation

  • A physics prior combines visual features and geometric quantities to estimate material class, body type, collision primitive, and numerical properties, which are treated as approximate priors.Its training targets are VLM-written labels rather than physical measurements; a VLM audit decides material class.Source: §3.4 · p. 9
  • PartSAM decomposes objects, CubePart can complete missing geometry, and Articulate Anything proposes URDF joints and motion ranges for review in simulation.For example, the VLM can flag a marker tip hidden beneath its cap before completion.Source: §3.4 · p. 9
  • Objects requiring deformation are represented as soft bodies, with simulated examples showing a corduroy cap and additional assets at rest and under deformation.The physics model uses a rigidity score to inform deformability; objects become soft bodies at deformability of at least 0.65.Source: §3.4 · pp. 9–10, Figure 10 · p. 10
  • SceneAgent also demonstrates contact and fluid simulation, including a robot pressing a plush toy and milk simulated with MPM as a cup pours.Source: Figure 11 · p. 10

3.5 Fracture Simulation

The fracture examples combine material-specific simulation models with scripted manipulation. The authors report progressive parchment tearing and ceramic fracture, while noting that key material parameters are engineering estimates.

3.5 Fracture Simulation

  • 49 interfaces fail progressively as parchment tears, leaving a detached sheet covering 95% of the bowl opening after the hands withdraw.The parchment uses an orthotropic thin-sheet finite-element model with irreversible cohesive fracture; coverage is reported as the lowest value over post-withdrawal frames.Source: §3.5 · p. 11, Figure 12 · p. 11
  • Ceramic vessels fracture when bonds between surface patches break after wall stress exceeds material strength, producing native PhysX rigid-body fragments.The model infers an inner wall 4.5 mm thick for each patch; cracks follow patch boundaries.Source: §3.5 · p. 11, Figure 12 · p. 11
  • Fracture parameters are engineering assumptions, and these examples are individually authored scenes using scripted controllers rather than automatic export or learned policies.Prediction of parameters from captures and validation against measured break forces remain future work.Source: §3.5 · p. 11

3.6 From 3D to 4D Captures

SceneAgent extends its static-capture workflow to videos, fitting simulations to observed motion and inspecting rendered sequences for discrepancies. The resulting physical parameters reproduce that motion but may not be unique or generalize to other interactions.

3.6 From 3D to 4D Captures

  • For 4D captures, agents use the same plan–execute–render–repair workflow as for 3D, extending the scene representation, parameter estimation, and inspection to frame sequences.A 4D capture is a video of a dynamic scene, such as liquid flowing, gel deforming, or fruit breaking apart.Source: §3.6 · p. 11
  • Each dynamic material has an initial state, physical parameters, and a simulator that advances its state frame by frame, with contact against static geometry and other materials.The scene exports a triangle mesh at each frame and preserves particle or vertex identities to provide persistent trajectories.Source: §3.6 · pp. 11–12
  • Agents select material-specific solvers and fit initial states and physical parameters to the video’s observed motion.The described choices include MPM for gels, elastoplastic solids, and liquids interacting with rigid bodies; FLIP for incompressible liquids; and fracture models for brittle objects.Source: §3.6 · p. 12
  • Agents compare rendered simulations with input frames at early, middle, and late times, then revise the camera, geometry, initial states, or physical parameters to address discrepancies.The resulting build script can regenerate frames, meshes, and trajectories without reading the input video or external assets.Source: §3.6 · p. 12
  • Fitted parameters reproduce the observed motion but are not necessarily unique and have not been tested on other interactions.The illustrated fracture and cutting examples use cohesive finite elements in Newton for the watermelon and jelly.Source: §3.6 · p. 12, Figure 13 · p. 12

3.7 Scene Export and Editing

SceneAgent exports editable, simulator-ready scenes, augments reconstructed objects and hybrid environments, and generates demonstrations for policy fine-tuning. Its policy workflow also supports iterative simulation-based responses to observed failures.

3.7 Scene Export and Editing

  • Scenes export as OpenUSD with movable-object mass, collision geometry, and estimated physical properties; deformable, fracture, and fluid behaviors require engine-specific setup.The scenes were tested in Isaac Lab, MuJoCo, and Unreal Engine, but those behaviors do not carry over through the USD file.Source: §3.7 · p. 13

3.8 Digital Twins and Sisters

  • Digital sisters vary reconstructed objects and scene lighting to expose policies to broader appearances and configurations during training.Variants can come from bounded mesh deformations and procedural textures, or model-guided component edits and retexturing, with checks for graspability, plausibility, or scene fit.Source: §3.8 · p. 13

3.9 Hybrid Scenes from Captures and Digital Art

  • Captured objects can be inserted into artist-authored environments without recapturing the workspace, using the same conversion, physical-modeling, and inspection tools.This supports adding task-specific assets, such as a device component or server-rack part, to an existing scene.Source: §3.9 · p. 13

3.10 Demonstration Generation and Policy Learning

  • In the autoresearch loop, agents use observed failure modes to create task or scene sisters, revise controllers, retain only successful simulated episodes, and retrain the policy.New checkpoints are compared with previous ones on fixed paired validation trials before another real-robot round.Source: §3.10 · pp. 15–16
  • The demonstration workflow randomizes initial conditions, runs a scripted pick-and-place expert, discards unsuccessful episodes, and uses rendered observations with expert actions to fine-tune a pretrained VLA.The randomization includes object placement and size and camera and robot-base poses; the task success check requires the object to end inside the target container.Source: §3.10 · p. 16

Agentic Policy Evaluation and Training

The policy autoresearch loop turns real-task recordings into an iterative cycle of simulation, controller revision, policy fine-tuning, and real-world re-evaluation.

Agentic Policy Evaluation and Training

  • Real-task recordings reveal failures; agents build a digital twin, generate task and scene sisters, and evaluate the current policy.Coding agents revise expert controllers and rerun simulated tasks; task-verified recordings feed policy fine-tuning.Source: Figure 19 · p. 17
  • Fixed paired validation trials compare policy checkpoints before the next real-world re-evaluation.This follows fine-tuning on task-verified recordings from the simulated loop.Source: Figure 19 · p. 17
  • The loop begins with real task recordings that expose failures and returns to the real robot after simulated evaluation and policy revision.The illustrated setup uses a Franka Emika Panda with a Sharpa Wave hand.Source: Figure 19 · p. 17

4 Results

Across reconstruction and dynamic-scene benchmarks, SceneAgent improves reported results; the paper also presents qualitative simulations and preliminary real-robot policy transfer.

4.1 Reconstruction Quality Across Agent Harnesses

  • On the kitchen capture, F1 at 1 cm rises from 16.45% to 41.13% for Codex, 7.07% to 19.29% for Claude Code, and 6.33% to 17.97% for ZCode.Their Chamfer distances decrease by 1.35–2.61 cm; Grok Build’s comparison is on a matched three-object subset because it omits the knife block.Source: §4.1 · p. 18
  • 60.75% F1 at 1 cm for Codex with SceneAgent exceeds SimFoundry’s 6.9%; every SceneAgent configuration improves on SimFoundry on both reported GraspClutter6D metrics.Across the six harnesses, F1 with SceneAgent ranges from 40.45% to 60.75%, versus 2.36% to 9.51% without it.Source: §4.1 · p. 18

4.2 Dynamic Scene Reconstruction on 4DCodeBench

  • Mean dynamic IoU increases for all five harnesses on three 4DCodeBench scenes, including Gemini 4 Argon from 0.86% to 99.09%.The reported values also rise from 11.10% to 66.50% for Grok 4.7, 52.16% to 55.95% for Claude Opus 5.5, 12.31% to 39.28% for GLM-5.3, and 7.79% to 36.38% for GPT-6 Astra.Source: §4.2 · p. 19
  • On these three cases, the highest mean dynamic IoU among the benchmark’s 18 published agents is 30.57%, reported for Claude Opus 5.5 [High].Those agents were run by the benchmark authors with their own settings and harness versions.Source: §4.2 · p. 19

4.3 Qualitative Scene and Object Conversion

  • Qualitative examples show converted scenes and objects supporting articulation, deformation, fracture, and simulated interactions, including a robot slewing an observatory telescope by pulling both guide ropes.The paper reports that real-world rollouts for the diverse simulated tasks are still in progress.Source: §4.3 · p. 19

4.4 Preliminary Real-World Policy Transfer

  • In real-world eggplant-to-basket pick-and-place evaluation, success rates did not improve indefinitely as simulated demonstration data scaled and dropped significantly after scaling past 40 demonstration episodes.The evaluation used 50 real-world layouts.Source: §4.4 · p. 20
  • The reported meta-learning strategy improves performance by sequentially fine-tuning across three datasets: varied eggplant demonstrations, three varied pick-and-place tasks, then eggplant demonstrations with only object-position variation.The final dataset contributes 2,000 demonstrations.Source: §4.4 · p. 20

5 Limitations and Future Work

SceneAgent’s conversion is costly and error-prone, and visual inspection cannot verify hidden geometry or physical parameters. Physical fidelity and evaluation remain bounded by uncalibrated parameters and a small set of scenes and tasks.

Conversion cost and verification

  • Processing every object in a detailed environment remains slow and error-prone, while visual inspection cannot verify hidden geometry or physical parameters.Background completion and generated object variants also depend on predictions of content absent from the capture.Source: §5 · p. 21

Physical and visual fidelity

  • Estimated physical parameters have not been calibrated against measured robot interactions, and captured Gaussian splats can produce inconsistent shadows and reflections when combined with mesh objects.Fracture examples use engineering estimates; future work should validate parameters and assess contact-rich behaviors including deformation, tearing, and adhesive contact.Source: §5 · p. 21

Evaluation scope

  • Evaluation covers one kitchen and one cluttered-bin scene converted once per configuration, three synthetic 4D scenes, and real-world pick-and-place; broader testing remains needed.Real-world 4D videos lack 3D ground truth, randomized factors were not isolated, and paired real and simulated trials are still needed to assess policy-evaluation value.Source: §5 · p. 21

Further automation

  • Future work includes isolating scene and physical-parameter randomization effects, inspecting recorded trajectories, and moving capture processing and policy adaptation onto robot hardware.On-device processing would allow data to remain on the device.Source: §5 · p. 21

6 Conclusion

SceneAgent converts 3D and 4D captures into interactive simulations through coordinated reconstruction and physical modeling, with simulator-based inspection and repair. It improves reported reconstruction metrics and extends converted environments for policy training and evaluation, while quantitative policy evaluation and physical-parameter validation remain ongoing.

6 Conclusion

  • SceneAgent coordinates reconstruction and physical modeling with simulator-based inspection and repair, fitting simulated dynamics to observed motion for dynamic-scene videos.Converted environments include digital sisters, hybrid scenes, and an automated loop that turns real-robot failures into simulated demonstrations for policy training and evaluation.Source: §6 · p. 22
  • On a cluttered GraspClutter6D bin scene, SceneAgent raises F1 at 1 cm for every harness with scorable paired outputs, with best F1 60.75% versus 6.9% for SimFoundry; on three 4DCodeBench scenes, Gemini 4 Argon mean dynamic IoU rises from 0.86% to 99.09%.The paper also reports simulated interactions and preliminary real-world deployment after policy fine-tuning; quantitative policy evaluation and physical-parameter validation remain ongoing.Source: §6 · p. 22

Featured Art and 3D Captures

The featured capture list names objects and scenes, including scans, electronics, household items, and museum objects.

Featured Art and 3D Captures

  • Named captures include a Rudbeckia flower, Chinese statue, earthenware pot, and motherboard.The listed motherboard is an Asus Z170-P.Source: Featured Art and 3D Captures · p. 27
  • Other listed captures include a vintage Sony radio, Totoro, and a composite skeleton at the American Museum of Natural History.Source: Featured Art and 3D Captures · p. 27
  • The list also includes a 1977 Bencini camera, a Cessna cockpit, a motorcycle, a towel rack, a red towel, a firefighter hose, and a hat.It additionally names a black-figure dinos and stand at the British Museum.Source: Featured Art and 3D Captures · p. 27

Related papers

Loading 2610.34745v1…