DRAWER: Digital Reconstruction and Articulation With Environment Realism

Motivation

• Static-scene digital twins could support safer robotics learning and automated game development, but matching visual, geometric, and physical properties remains unresolved.Digital replicas could enable navigation, object interaction, and policy transfer between simulated and real environments.Source: §1 · p. 1
• Existing approaches trade visual fidelity against geometric precision or provide interaction through simplified primitives at the expense of realism.Neural reconstructions are often static and non-actionable, whereas shape primitives and CAD models sacrifice visual and geometric fidelity.Source: §1 · pp. 1–2

Approach

• DRAWER combines a neural SDF with surface-anchored Gaussian splats, then infers articulation types and hinges while aligning simulatable geometry and appearance.The framework also reconstructs hidden interior geometry and appearance to support realistic interactions.Source: §1 · p. 2, Figure 2 · p. 2

Evaluation

• Across six kitchen environments, DRAWER significantly outperforms prior art in visual realism, articulation accuracy, and simulated-motion precision.The evaluation measures visual realism, articulation accuracy, and precision of simulated motions.Source: §1 · p. 2

Applications

• DRAWER supports automatic interactive-game creation and real-to-sim-to-real robotic-controller training without tedious manual effort or hand specification.The generated environments are used for an interactive game and a robotics loop.Source: §1 · p. 2

Novel view synthesis

• DRAWER’s dual representation combines Gaussian splatting’s appearance and real-time rendering with neural surface models’ detailed geometry for active simulation.The approach extends beyond novel-view synthesis to counterfactual visualization during scene interaction.Source: §2 · p. 2, Figure 3 · p. 3

Data-driven simulation

• Unlike prior scene methods, DRAWER simulates articulated objects, improves visual quality, and transfers drawer- and cabinet-opening skills from simulation to the real world zero-shot.These differences distinguish it from Video2Game [89].Source: §2 · pp. 2–3

Articulation modeling and simulation

• Scene-level articulation methods commonly require dynamic observations or human intervention, while DRAWER reconstructs visually, geometrically, and physically faithful scenes from passive observations.Prior methods often focus on semantics and functionality while neglecting visual and geometric fidelity.Source: §2 · p. 3

Controllable video generation

• Controllable video generation can depict image-space dynamics but lacks internal states and remains difficult to integrate with physical simulation engines.DRAWER instead adheres to physical laws, supports simulation engines, and exposes underlying states beyond rendering.Source: §2 · p. 3

3. DRAWER: Digital Reconstruction and Articulation With Environment Realism

• DRAWER addresses the appearance-versus-geometry trade-off by using distinct but tightly coupled representations for rendering, sensing, and physics simulation.This design targets real-time interactive applications while maintaining object-level compatibility for simulation engines.Source: §3 · p. 3, §3.2 · p. 4, Table 1 · p. 4

3.2. Dual Scene Representation

• The dual representation combines neural SDF geometry with Gaussian-splat appearance, coupling high-fidelity rendering to detailed physical structure.SDF-derived geometry supports collision modeling, while anchored Gaussians preserve visual quality and remain bound to the reconstructed surface.Source: §3.2 · pp. 4–5, Figure 5 · p. 4
• Anchoring Gaussians to the SDF-derived mesh prevents visual-geometry mismatch while retaining controlled flexibility and adaptive detail placement.A straight-through estimator enables training despite non-differentiable clipping constraints.Source: §3.2 · p. 5

3.3. Articulating the Scene

• DRAWER identifies articulated and rigid objects, estimates hinges and articulation axes, reconstructs hidden geometry and materials, and composes completed objects back into the scene.Segmentation is fused across views and refined with foundation models and VLMs; physical reasoning combines 3DOI, GPT4o, and VLM arbitration.Source: §3.3 · pp. 6–7

3.4. Downstream Applications with DRAWER

• The framework transfers reconstructed environments into Unreal Engine and physics simulators for interactive games, robot learning, and real-to-sim-to-real transfer.Applications include movement, shooting, drawer and cabinet interaction, object pick-and-place, and robot-policy training without extensive manual data collection.Source: §3.4 · p. 7, Figure 11 · p. 7, Figure 12 · p. 8

4.2. Experimental Results

• DRAWER significantly outperforms prior art across visual realism, articulation accuracy, and simulated-motion metrics on six kitchen environments.The evaluation uses PSNR, SSIM, LPIPS, precision, recall, EMD, and EA-Score to assess these dimensions.Source: §4.1 · p. 8
• DRAWER balances rendering quality and geometric fidelity while providing more interactive functionality than prior scene representations.Although 3DGS achieves the best rendering results, it compromises geometric fidelity; DRAWER supports interactions beyond simple pick-and-place.Source: §4.2 · p. 8
• DRAWER accurately recalls articulated objects with high precision while maintaining high visual and geometric fidelity against URDFormer and Digital Cousin.The full-pipeline comparison evaluates interactable 3D reconstruction methods using Fig. 6 and Tab. 3.Source: §4.2 · p. 8, Table 3 · p. 9
• 1.41 × 10−5 EMD versus 17.7 × 10−5 for KlingAI demonstrates more accurate physically grounded articulated motion simulation.DRAWER simulates trajectories in physics engines, whereas KlingAI motion is often infeasible despite manually supplied point prompts.Source: §4.2 · pp. 8–9
• Adding Gaussian splatting, movable Gaussians, and straight-through estimation progressively improves reconstruction performance in ablations.The ablation begins with a neural-SDF mesh and sequentially restores the remaining components.Source: §4.2 · p. 9
• DRAWER generalizes beyond kitchens to offices and bathrooms, as shown in Fig. 10.Source: §4.2 · p. 9

4.3. Build Your Own Game

• The Unreal Engine game supports real-time movement, shooting, opening articulated objects, and physically realistic object manipulation.Its assets combine SDF-extracted collision meshes, segmented rigid bodies, detailed articulated interiors, and real-time Gaussian rendering.Source: §4.3 · p. 9

4.4. Real-to-Sim-to-Real

• DRAWER generates simulation data for drawer opening, object placement, and closing, then transfers learned policies to a real Franka Emika Panda robot.The pipeline imports reconstructed geometry and articulation into Isaac Sim, uses motion planning and grasp sampling, and trains behavior-cloning policies.Source: §4.4 · pp. 9–10

• DRAWER converts a single video into an interactive environment with articulated and rigid-body dynamics without prior articulation data.Source: §5 · p. 10
• The method integrates an SDF field and Gaussian splats, then decomposes and articulates the representation into a functional environment.Source: §5 · p. 10
• DRAWER demonstrates strong articulation understanding and rendering while supporting interactive games and real-to-sim-to-real robot learning.Source: §5 · p. 10
• Integration with more sophisticated relightable environment reconstruction is identified as a future improvement.Source: §5 · p. 10