PaperlayerView exact revision on arXiv

Source-linked AI summary

Fysiverse-3D-SimReady Technical Report: Agentic Physical Simulation for Pragmatic 3D World Reconstruction

Lintao Wang, Mingyang Sun, Yang Liu, Dingkang Yang, Lihua Zhang

arXiv:2609.31715v1cs.CV

TL;DR

Single-image 3D generators can recover plausible assets, but independent meshes do not by themselves yield camera-aligned, gravity-consistent, stable scenes. Fysiverse-3D-SimReady refines generated meshes in a shared frame and uses the resulting scene to build a goal-conditioned physical rollout. Experiments show that visual reconstruction alone can leave hidden instability or view and contact inconsistencies.

  • Problem

    Independently generated meshes may fail to align with the input camera, respect gravity and contacts, or remain stable, limiting their suitability as simulation-ready scenes.

    Source: §1 · pp. 1–2

  • Method

    Fysiverse-3D-SimReady grounds generated meshes in a shared gravity frame and original camera, corrects supports and contacts, and uses an agentic workflow to build a goal-conditioned rollout.

    Source: §1 · p. 2

  • Results

    Experiments show visual reconstruction alone can preserve masks while leaving hidden instability, or remove penetration while scenes drift or lose input-view agreement.

    Source: §6 · p. 14

  • Takeaways & Limitations

    Camera grounding, gravity alignment, contact correction, and simulator execution must be treated as coupled scene-level requirements for physical use.

    Source: §6 · p. 14

    A scene counts as simulation-ready only if it aligns with observed masks, has coherent gravity and plausible contacts, and avoids severe collapse or drift after settling.

    Source: §3 · p. 4

Abstract

from arXiv · show

Agentic recognition requires visual perception to move beyond static scene understanding and produce structured scene representations that support the perception--reasoning--action loop. Existing single-image 3D generation methods, however, mainly produce visually plausible object assets rather than simulation-ready scene states. When independently generated meshes are composed in a shared space, they may fail to align with the input camera, violate gravity, interpenetrate nearby objects, or become unstable under physics simulation. We present Fysiverse-3D-SimReady, a grounded refinement framework for reconstructing simulation-ready multi-object scenes from a single RGB image with instance and ground prompts. The method places generated object meshes into a shared gravity-aligned scene, refines their camera-space poses through differentiable rendering, and corrects scene-level supports and contacts for stable physical execution. The scene is then used by an agentic simulation workflow, which converts a scene-specific task goal into an executable physics rollout rendered from the original camera view. Experiments show that Fysiverse-3D-SimReady improves input-view alignment, contact plausibility, and physical stability over existing single-image reconstruction and scene generation baselines, while enabling goal-conditioned physical interactions from a single image.

1 Introduction

Single-image 3D methods can produce plausible object meshes, but independently generated assets do not by themselves form a camera-aligned, physically coherent scene. Fysiverse-3D-SimReady addresses this gap by grounding and refining assets for stable simulation, then connecting the scene to goal-conditioned physical rollouts.

1 Introduction

  • Single-image meshes remain ambiguous in scale, pose, and contact, so their composition can yield camera misalignment, incorrect support, gravity errors, or unstable physics.These failures affect whether the scene renders back into the source image and supports meaningful, stable simulation.Source: §1 · p. 2
  • Fysiverse-3D-SimReady connects single-image scene reconstruction to agentic physical simulation, producing a simulation-ready scene driven by a scene-specific task goal.Source: §1 · p. 2
  • The reconstruction pipeline grounds independently generated assets in a shared gravity frame and the original camera, then corrects supports, contacts, and penetrations for physics simulation.Source: §1 · p. 2
  • The agentic workflow takes the refined scene and a scene-specific goal to build an executable rollout rendered from the original view.Source: §1 · p. 2

2 Related Work

Related work spans language-based agentic workflows, single-image 3D and scene generation, and systems for physically grounded scene construction. The paper positions its approach as bringing these complementary components together for single-RGB reconstruction and simulation.

2.1 Agentic Workflows and Physical Simulation

  • Fysiverse-3D-SimReady applies agentic-workflow principles to physical simulation by interpreting a scene-specific goal, assigning roles and parameters, validating a plan, and invoking simulation and rendering tools.Source: §2.1 · p. 2

2.2 3D Generation and Scene Construction

  • Single-image asset and scene-generation methods improve fidelity and composition, but incompatible scales, poses, coordinate frames, and contact assumptions can still prevent stable physics execution.Source: §2.2 · p. 3

2.3 Simulation-Ready Scene Generation and Simulation Platforms

  • Prior simulation-ready methods address physical constraints and execution at different stages and input settings; the proposed pipeline combines original-view grounding, contact correction, collision proxies, passive settling, and agentic simulation from a single RGB image.Source: §2.3 · p. 3

3 Problem Formulation

The problem is to recover a scene that remains tied to a single photograph while also being usable in a physics simulator. Its formulation combines prompted image evidence with shared-frame visual and collision representations, and defines readiness through visual grounding and physical stability.

3 Problem Formulation

  • The input is a single RGB photograph with prompted instance masks and a separate ground mask marking the visible support surface.The ground mask supports gravity estimation and definition of the simulator ground plane; it is not a foreground object to reconstruct.Source: §3 · p. 3
  • The output scene state combines the input camera, textured object meshes, shared-world poses and scales, collision proxies, and a physics-rollout pose sequence.The visual mesh preserves generated appearance, while a collision proxy supports contact and collision computation without needing to match that mesh exactly.Source: §3 · pp. 3–4
  • A simulation-ready scene must align rendered objects with observed masks, use a coherent gravity direction, and avoid implausible contacts, interpenetration, or severe collapse and drift after settling.Source: §3 · p. 4
  • The formulation requires one scene state to satisfy both source-image fidelity and usability by a physics simulator.Source: §3 · p. 4

4 Methodology

F3SR separates object generation from scene-level refinement, preserving instance identity while grounding meshes in gravity, the input view, and physical relations. It then uses the refined scene for goal-conditioned simulation and original-view rendering.

4 Methodology

  • The framework links single-image scene reconstruction to agentic physical simulation, tracking each instance from its image mask through mesh, support relations, collision proxy, and simulation actor.This identity tracking keeps later language and physics modules from changing the object set and makes corrections traceable to image regions.Source: §4 · p. 4

4.1 Gravity Consistent Scene Initialization

  • Ground prompts and instance masks support shared gravity-aligned initialization; a whole-scene rotation aligns the ground normal to up, and a shared translation rests the support layer on z=0.Global grounding preserves relative object heights rather than snapping every object to the floor separately.Source: §4.1 · pp. 4–5

4.2 Input View Alignment

  • Differentiable rendering first optimizes camera extrinsics, then independently refines each object against its label mask using bounded camera-plane translation, yaw, and scale.The camera update is retained only if mIoU does not fall below the initial camera's; object-refinement phases also roll back when their silhouette degrades beyond a tolerance.Source: §4.2 · p. 6

4.3 Scene Graph Guided Physical Refinement

  • A mask-scoped scene graph supplies semantic, pairwise, and support priors, while collision proxies perform contact tests and guide support-aware penetration correction.Supported objects move along gravity with their subtree; unrelated overlaps are separated horizontally, preserving image-implied support relations.Source: §4.3 · pp. 6–7

4.4 Agentic Goal Conditioned Simulation

  • Planner, Reasoner, and Builder ground a goal to existing actors, propose physical parameters, validate the plan, and prepare an executable rollout using the reconstruction's collision proxies.The simulated poses are written back to detailed visual meshes and rendered from the optimized original camera; a separate static background is used only for rendering.Source: §4.4 · p. 7

5 Experiments

The experiments evaluate F3SR’s input-view alignment, scene geometry and stability, and goal-conditioned physical interactions. Comparisons, ablations, and simulation rollouts assess how its refinement stages support these outcomes.

5.1 Experimental Setup

  • The evaluation compares image-conditioned scene methods on image agreement, penetration, release energy, and passive drift, alongside agentic workflow conditions.Baselines include SAM3D, TabletopGen, and adapted PAT3D; simulator assistance compares the full Planner–Reasoner–Builder workflow with direct plan prediction.Source: §5 · p. 8, §5.1 · p. 8, §5.2 · p. 9

5.2 Quantitative Comparison

  • F3SR achieves the highest PSNR and lowest release energy, eliminates measured inter-object penetration, and reduces drift versus SAM3D by over two orders of magnitude.SAM3D achieves the highest mIoU; removing penetration alone does not ensure simulation-ready reconstruction.Source: Table 1 · p. 9, §5.2 · p. 9

5.3 Qualitative Comparison

  • Side views expose support, placement, and interpenetration artifacts that may be hidden in the original view; F3SR shows fewer artifacts and more stable object-ground alignment.Examples span room-scale and tabletop scenes; SAM3D can produce recognizable assets that remain interpenetrating, floating, or unbalanced when assembled without scene-level correction.Source: §5.3 · p. 10, Figure 3 · p. 10

5.4 Ablation Studies

  • Removing differentiable rendering reduces PSNR from 11.39 to 11.29 dB and mIoU from 0.71 to 0.67, while increasing maximum release energy from 0.04 J to 0.16 J and drift from 1.09 cm to 1.58 cm.The ablation removes per-object camera-plane translation, yaw, and scale refinement while retaining the other pipeline stages.Source: §5.4 · p. 11, Table 2 · p. 11
  • Damping dissipates residual motion during passive gravity settling, helping released scenes converge to stable configurations while preserving intended support layouts.Instability is measured across 60 trials as the percentage of objects that topple, slide from support, or lose stable contact.Source: §5.4 · pp. 11–12, Figure 5 · p. 12
  • Adding Planner, Reasoner, and Builder roles improves prompt following for both tested backends and reduces average end-to-end time in this task set.GPT-5.5 prompt following rises from 86.2% to 93.1%; Qwen3.7-Plus rises from 0.0% to 66.7%.Source: §5.4 · p. 12, Table 3 · p. 12

5.5 Goal Conditioned Simulation Rollouts

  • The workflow renders executed pose sequences from the original camera and demonstrates impact-driven motion, support changes, tipping, rolling, and collision-induced target motion.The examples cover room-scale and tabletop scenes.Source: §5.5 · pp. 13–14, Figure 6 · p. 13

6 Conclusion

Fysiverse-3D-SimReady extends single-image reconstruction to simulation-ready scenes by coupling camera grounding, gravity alignment, contact correction, and simulator execution. Its experiments show that visual reconstruction alone can leave instability or drift, while the combined system supports goal-conditioned physical interaction.

6 Conclusion

  • Single-image reconstruction becomes simulation-ready when camera grounding, gravity alignment, contact correction, and simulator execution are treated as coupled scene-level requirements.The framework keeps object meshes tied to input masks while refining poses in a shared world frame.Source: §6 · p. 14
  • Strong visual reconstruction alone is insufficient for physical use: methods may preserve masks while leaving hidden instability, or remove penetration while drifting or losing input-view agreement.Source: §6 · p. 14
  • Combining grounded reconstruction with Planner, Reasoner, and Builder roles provides a practical bridge to goal-conditioned physical interaction.Source: §6 · p. 14

Related papers

Loading 2609.31715v1…