HoloScene: Simulation-Ready Interactive 3D Worlds from a Single Video

1 Introduction

• Existing approaches leave gaps in occluded geometry, inter-object relationships, physical plausibility, and interaction, while some prior methods require full observations or handle only simple interactions.The introduction distinguishes these limitations across Real2Sim, amodal reconstruction, and physically plausible reconstruction methods.Source: §1 · p. 2
• HoloScene represents scenes as hierarchical interactive scene graphs encoding object geometry, appearance, and physical properties.Scene-graph recovery is formulated as energy-based optimization that integrates observational data, physical constraints, and generative priors.Source: §1 · p. 2
• HoloScene uses sampling-based tree-structured search with gradient-based refinement to recover scenes with accurate geometry, stable physical interactions, and realistic novel-view rendering.Source: §1 · p. 2
• Across three challenging benchmarks, HoloScene achieves superior geometry accuracy and physical plausibility, with rendering performance comparable to state-of-the-art amodal and physics-aware reconstruction methods.Interactive gaming, realistic video effects, and real-time digital-twin manipulation illustrate practical applications.Source: §1 · p. 2

2 Related Works

• Many scene-modeling methods render realistic novel views but do not provide assets users can interact with, while object-level methods do not handle large, complex indoor scenes.Other approaches have limited components, lack physical stability, or generate layouts that are not faithful to visual inputs.Source: §2 · p. 2
• HoloScene combines scene-graph object interaction with generative priors and sampling to reconstruct each component’s geometry and appearance in physically plausible, interactable environments.Source: §2 · p. 3
• Against CAST [86], HoloScene uses multi-view video optimization for input fidelity, simulator-in-the-loop optimization for stability, and unified energy-based inference rather than greedy sequential optimization.Source: §2 · p. 3

3 Method

• HoloScene combines observational evidence, generative completion priors, and physical constraints in an energy-minimization framework solved with sampling-based tree search and differentiable optimization.Source: §3 · p. 3, §3.3 · p. 6

3.1 Scene Representation

• Nodes encode geometry, appearance, physical properties, and state using SDF-derived meshes and mesh-attached Gaussian splats; edges capture support, beside, and collision relationships.Source: §3.1 · pp. 3–4

3.2 Problem Formulation

• The objective combines rendered RGB, mask, and monocular geometry discrepancies with completeness, geometry, and physics regularizers.Source: §3.2 · p. 5
• Completeness uses virtual-view observations generated with Wonder3D after LaMa inpaints occluded regions, while geometry penalties discourage SDF and mesh intersections.Source: §3.2 · p. 5
• Physical plausibility penalizes gravity-induced state changes and encourages contact between supporting meshes.Source: §3.2 · pp. 5–6

3.3 Inference

• Stage 1 fits per-object SDF geometry and Gaussian appearance to observations with gradient-based losses, then extracts initial meshes.Source: §3.3 · p. 6
• Because completion and physical terms are multimodal or non-differentiable, Stage 2 combines diverse Wonder3D shape samples with breadth-first support-tree search, selecting low-energy candidates.The selected shapes' states and physical parameters are adjusted to enforce stability and prevent interpenetration.Source: §3.3 · pp. 6–7
• Stage 3 re-minimizes observation terms by gradient descent to fine-tune attached Gaussians for complete, realistic appearance.Source: §3.3 · p. 7

4.1 Experimental Results

• HoloScene achieves the best object-reconstruction and stable-object ratios while maintaining comparable scene-level geometry and appearance quality.Evaluations span Replica, ScanNet++, and iGibson; the study measures geometry, rendering, and physical stability.Source: §4 · p. 7, Table 2 · p. 8, §4.1 · p. 9
• On iGibson, HoloScene outperforms all baselines in reconstructing invisible and occluded object regions.The evaluation uses complete object ground truth and renders six viewpoints per object to assess appearance.Source: §4.1 · p. 9
• Generative priors, physical energy, and scene-graph relationships improve physical stability and object-level reconstruction, with a trade-off against scene-level alignment.The ablation reports that physical-plausibility optimization may occasionally compromise pixel alignment with observations.Source: §4.1 · p. 9, Table 3 · p. 9
• HoloScene has the lowest geometry and physics failure rates among the compared reconstruction-based baselines.Its generative priors and physics-aware constraints reduce invalid shapes and inter-object penetrations.Source: §4.1 · p. 9, Table 4 · p. 10

4.2 Interactive Environment Applications

• In Unreal Engine, reconstructed objects can be physically rearranged and support realistic agent interactions; users can also reposition and rotate objects for scene editing.The applications include a third-person game and interactive editing of object Gaussians with underlying physical mesh geometry.Source: §4.2 · p. 10
• HoloScene supports replaying object interactions from video and adding spatially grounded visual effects that blend with reconstructed scenes.The demonstrated effects include rigid-body simulations, character animations, and particle effects.Source: §4.2 · p. 10, Figure 5 · p. 10

• HoloScene combines scene graphs and energy-based optimization to reconstruct environments with realistic appearance, complete geometry, and interactive physical plausibility.Source: §5 · p. 10
• The framework currently handles only videos of static indoor scenes; dynamic scenes and large outdoor environments remain challenging.Source: §5 · p. 10
• Future work will focus on relightable reconstruction and extending support to articulated and deformable objects.Source: §5 · p. 10

• A third-person game lets an agent run, jump, explore, and interact with scene objects that remain stable and follow physical rules.The game is built in Unreal Engine using reconstructed objects and background texture mesh.Source: A More Interactive Environment… · p. 16, Figure 1 · p. 16
• Users can translate objects along the xy-plane and rotate them around the vertical z-axis for coherent scene edits.The demonstrated example repositions a reconstructed swivel chair.Source: A More Interactive Environment… · p. 16
• From static RGB video, the pipeline estimates an object's 6D pose and replays its motion in a virtual scene.It uses VGGT for camera pose, depth alignment, and FoundationPose with the reconstructed asset for tracking.Source: A More Interactive Environment… · p. 16
• Reconstructed scenes can be augmented with rigid-body simulations, character animations, and particle effects that blend naturally with the scene.The effects are spatially grounded and overlaid with shadows onto the input image.Source: A More Interactive Environment… · p. 16

B More Physical Simulation Results

• Figure 2 shows users changing an object's location and orientation through an interactive control panel.Source: Figure 2 · p. 17
• Figure 3 pairs original recordings with simulated Gaussian renderings and marks estimated object 6D poses with oriented bounding boxes.Source: Figure 3 · p. 17
• Across Replica, ScanNet++, and iGibson, HoloScene achieves the best physical stability and plausibility, as well as realistic rendering quality.The comparisons include physical-simulation results across the three datasets.Source: B More Physical Simulation Results · pp. 16–17

C Method Details

• Breadth-first scene-tree search selects each reconstructed object candidate with the lowest physical energy, then adjusts its state for stability.The procedure regularizes SDF values and mesh vertices, prunes points near parent or neighboring objects, and collision-tests the resulting mesh.Source: C Method Details · p. 17
• Figure 4 shows dynamic visual effects including animations, dropping objects, and setting fires.Source: C Method Details · p. 18
• After optimization, contact surfaces are regularized to avoid inter-object collisions and further improve scene stability.Source: C Method Details · p. 18

D.1 Datasets

• Replica provides textured meshes with instance annotations but no videos or camera trajectories, so the authors render images along manually designed dense trajectories.The trajectories cover several hundred frames per scene, with images rendered at 512 × 512 pixels.Source: §D.1 · p. 18
• ScanNet++ provides posed multi-view images and aligned instance-annotated scans; the authors replace scan-derived masks with cleaner masks from.Images are downsampled by a factor of two.Source: §D.1 · p. 18
• iGibson provides complete object and background geometry and appearance but no camera trajectories, so the authors render images along manually designed dense trajectories.The trajectories cover several hundred frames per scene, with images rendered at 512 × 512 pixels.Source: §D.1 · p. 18

D.2 Baselines

• The benchmark adapts open-source ObjectSDF++, PhyRecon, and DP-Recon code to the evaluation setup.Source: §D.2 · p. 18
• ObjectSDF++ and PhyRecon receive new data loaders for supported image resolutions; PhyRecon skips physical loss for objects that fail reconstruction.DP-Recon is adapted for missing-object bounding boxes and evaluated using its final-stage texture meshes.Source: §D.2 · pp. 18–19

D.3 The inference details of HoloScene

• Figures 5–8 show object geometry and appearance through no-physics, intermediate-physics, and final-physics states across ScanNet++, Replica, and iGibson.Source: Figure 5 · p. 19, Figure 6 · p. 20, Figure 7 · p. 21
• HoloScene's inference uses 200k gradient-optimization iterations, up to five sampled candidates per object, and 30k iterations of global Gaussian refinement.The physical simulator is Isaac Sim with default physical parameters; refinement also uses generated images to optimize instance Gaussians.Source: §D.3 · pp. 19–20
• In the iGibson comparison, DP-Recon's incomplete background mesh leads to an object falling in the figure.Source: Figure 8 · p. 22