Source-linked AI summary
SimRecon: SimReady Compositional Scene Reconstruction from Real Videos
Chong Xia, Kai Zhu, Zizhuo Wang, Fangfu Liu, Zhizheng Zhang, Yueqi Duan
TL;DR
Compositional reconstruction from real videos remains limited by weak generalization, incomplete object geometry, and physically implausible simulation scenes. SimRecon connects perception, single-object generation, and simulation with Active Viewpoint Optimization and Scene Graph Synthesizer bridging modules, and experiments on ScanNet validate superior reconstruction quality and physical adherence.
Problem
Existing compositional reconstruction methods show limited real-world generalization and struggle with complete object geometry and physically plausible simulation-ready scenes.
Method
SimRecon reconstructs scene semantics, generates individual object assets, and assembles them in a simulator using optimized viewpoints and scene-graph-guided physical construction.
Results
SimRecon achieves superior performance in reconstruction quality and physical adherence on the ScanNet dataset.
Takeaways & Limitations
The framework provides an object-centric, simulation-ready reconstruction pipeline for cluttered real-world videos.
Abstract
from arXiv · showhide
Compositional scene reconstruction seeks to create object-centric representations rather than holistic scenes from real-world videos, which is natively applicable for simulation and interaction. Conventional compositional reconstruction approaches primarily emphasize on visual appearance and show limited generalization ability to real-world scenarios. In this paper, we propose SimRecon, a framework that realizes a "Perception-Generation-Simulation" pipeline towards cluttered scene reconstruction, which first conducts scene-level semantic reconstruction from video input, then performs single-object generation, and finally assembles these assets in the simulator. However, naively combining these three stages leads to visual infidelity of generated assets and physical implausibility of the final scene, a problem particularly severe for complex scenes. Thus, we further propose two bridging modules between the three stages to address this problem. To be specific, for the transition from Perception to Generation, critical for visual fidelity, we introduce Active Viewpoint Optimization, which actively searches in 3D space to acquire optimal projected images as conditions for single-object completion. Moreover, for the transition from Generation to Simulation, essential for physical plausibility, we propose a Scene Graph Synthesizer, which guides the construction from scratch in 3D simulators, mirroring the native, constructive principle of the real world. Extensive experiments on the ScanNet dataset validate our method's superior performance over previous state-of-the-art approaches.
1. Introduction
SimRecon targets simulation-ready compositional reconstruction from cluttered videos by connecting perception, object generation, and physical simulation with dedicated bridging modules. It addresses visual infidelity and physical implausibility through optimized viewpoints and constructive scene-graph-guided assembly.
- Holistic reconstructions lack complete object geometry and well-defined boundaries, limiting their suitability for simulation and interaction.
- Compositional methods face heuristic view selection, physically implausible final scenes, and pipeline-specific designs that limit generalization and extensibility.
- SimRecon transforms cluttered video input into a simulation-ready compositional 3D scene through unified object-centric spatial representations.
- The framework designs bridging modules to complete individual-object geometry and appearance while ensuring physically plausible placement.
- Active Viewpoint Optimization searches for high-information 3D views, while Scene Graph Synthesizer supports hierarchical physical assembly through supportive and attached relations.
2. Related Work
Related work spans indoor simulator construction, compositional 3D reconstruction, and 3D scene graphs. These lines of research provide context for SimRecon’s object-centric, simulation-oriented framework.
- 3D Indoor Scene Simulators: Indoor simulators use hand-crafted, generation-based, or scan-based construction, with approaches involving manual annotation, procedural generation, or captured scan data.
- Framework Context: SimRecon’s framework is illustrated as a Perception-Generation-Simulation pipeline using object-centric representations and two core modules, AVO and SGS.
- 3D Indoor Scene Simulators: Simulation-ready reconstruction from raw video input could support generating diverse simulation environments from arbitrary videos.
- Compositional 3D Reconstruction: Compositional reconstruction methods model scenes as object collections for interactive generation and embodied tasks, extending earlier single-view and multi-stage approaches.
- 3D Scene Graphs: A 3D scene graph represents objects or areas as nodes and their spatial or functional connections as edges.
3. Approach
SimRecon represents cluttered scenes as object-centric entities and bridges perception, generation, and simulation with optimized views and graph-guided physical assembly. Its approach targets both complete object appearance and physically plausible scene construction.
- Object-Centric Scene Representation: Each object primitive combines intrinsic attributes with relational attributes encoding its role and interactions within a structured scene graph.Intrinsic attributes include spatial, appearance, and physical properties; relational attributes represent supportive, spatial, and functional relationships.
- Overall Architecture: SimRecon uses a Perception-Generation-Simulation pipeline that progressively transforms segmented video observations into simulation-ready compositional entities.Semantic reconstruction provides object attributes, generation completes geometry and appearance, and graph construction supplies physical relations.
- Active Viewpoint Optimization: Active Viewpoint Optimization searches 3D space for informative object projections, replacing heuristic views that can miss geometry under severe occlusion.It formulates viewpoint selection as information gain maximization and uses accumulated opacity as a differentiable proxy.
- Active Viewpoint Optimization: The viewpoint objective maximizes accumulated object opacity and optimizes camera pose through differentiable Gaussian rendering, with depth regularization preventing overly close viewpoints.The method transfers camera transformations to differentiable Gaussian parameters and constrains rendered depth according to object size.
- Scene Graph Synthesizer: Scene Graph Synthesizer incrementally builds a global graph of support and attachment relations from local observations to scaffold physically plausible simulator construction.The graph addresses the difficulty of global reasoning in cluttered, occluded scenes and supports constructive placement.
- Hierarchical Physical Assembly: Hierarchical assembly initializes floor and wall, then uses breadth-first traversal, gravity, collision, and fixed constraints to settle or anchor objects.Supported objects temporarily simulate as active rigid bodies before stabilization, while attached objects receive fixed constraints.
4. Experiments
Experiments evaluate SimRecon on raw RGB videos from ScanNet against compositional reconstruction and indoor-simulation baselines, using geometric, rendering, runtime, and qualitative assessments. Results highlight improved object reconstruction, viewpoint conditioning, physical assembly, and the value of both bridging modules.
- Experimental Setup: Experiments use 20 ScanNet scenes with only raw RGB videos, excluding depths, normals, and semantics.
- Experimental Setup: SimRecon is compared with DPRecon, InstaScene, Gen3DSR, SceneGen, and MetaScenes across reconstruction and simulation-ready scene construction.The evaluation includes both scene-level and single-view reconstruction baselines, plus an indoor simulator baseline.
- Experimental Setup: Evaluation measures geometric fidelity with CD, F-Score, and NC; rendering with PSNR, SSIM, LPIPS, and MUSIQ; and reports processing time.
- Compositional 3D Reconstruction: SimRecon reconstructs assets with high geometric and visual fidelity by optimizing projections for 3D information gain rather than relying on heuristic views.The qualitative comparison covers final reconstructed scenes, while Table 1 reports geometric fidelity, novel-view rendering quality, and inference time.
- Viewpoint Sampling: Active Viewpoint Optimization captures full target-object structure and appearance across varying scales, unlike maximum-visibility and canonical-view sampling strategies.Maximum 2D coverage can lack required 3D structural information, while canonical views can remain occluded and produce malformed geometry and appearance.
- Physical Scene Construction: Scene Graph Synthesizer enables hierarchical, physics-informed assembly that preserves semantic coherence and physical stability from the outset.The qualitative simulator comparison contrasts this construction with MetaScenes’ retrieval-based assets and posthoc collision-resolution search.
- Ablation Study: Ablations show that maximum-visibility AVO can stop at whole-object coverage, missing depth supervision collapses viewpoints toward the surface, and naive SGS alternatives produce incomplete or incoherent graphs.The ablation study visualizes AVO and SGS variants in Figures 7 and 8.
5. Conclusion
SimRecon addresses visual infidelity and physical implausibility in compositional scene reconstruction with two bridging modules across a perception-generation-simulation pipeline. On ScanNet, it achieves superior reconstruction quality and physical adherence.
- SimRecon reconstructs object-centric, simulation-ready scenes from cluttered real-world videos through a Perception-Generation-Simulation pipeline.
- Active Viewpoint Optimization searches for optimal projections that provide high-fidelity conditions for generative completion.
- Scene Graph Synthesizer guides constructive assembly that mirrors real construction principles to ensure physical plausibility from the outset.
- Experiments on ScanNet show superior performance in both reconstruction quality and physical adherence.