PaperlayerView exact revision on arXiv

Source-linked AI summary

SimFoundry: Modular and Automated Scene Generation for Policy Learning and Evaluation

Nadun Ranawaka, Josiah Wong, Wei-Lin Pai, Wei-Teng Chu, Tianyuan Dai, Masoud Moghani, Hang Yin, Yunfan Jiang, Wesley Durbano, Brandon Huynh, Yu Fang, Danfei Xu, Ruohan Zhang, Li Fei-Fei, Linxi Fan, Bowen Wen, Ajay Mandlekar, Yuke Zhu

arXiv:2606.28276v4cs.RO

TL;DR

Real-world robot policy training and evaluation are costly and difficult to scale, while manually building aligned simulations remains challenging. SimFoundry automatically constructs sim-ready scenes from a single video and generates digital cousins for training and evaluation, with simulation results closely predicting real-world performance across tasks and policies.

  • Problem

    Real-world robot policy data collection and systematic evaluation are costly, while manually constructing simulations aligned with real scenes and tasks remains challenging.

  • Method

    SimFoundry converts a single video into interactive, sim-ready scenes and generates affordance-preserving object, scene, and task variations through a modular pipeline.

  • Results

    Across 7 tasks and 5 policy types, simulation evaluations match real-world performance with mean Pearson correlation 0.911 and MMRV 0.018.

  • Takeaways & Limitations

    SimFoundry-generated data transfers to real-world policies, while digital cousins improve robustness to unseen objects, novel layouts, and related downstream tasks.

  • Takeaways & Limitations

    The pipeline inherits off-the-shelf foundation-model failure modes and is currently limited to tabletop-style layouts.

Abstract

from arXiv · show

Training and evaluating robot policies in the real world is costly and difficult to scale. We introduce SimFoundry, a modular and automated system for zero-shot real-to-sim scene construction from a video. SimFoundry generates sim-ready digital twins and supports object, scene, and task editing, enabling the automated generation of diverse digital cousins: affordance-preserving variations of reconstructed real-world scenes. Policies trained on SimFoundry data transfer zero-shot to challenging real tasks involving multi-step manipulation, articulated object interaction, and bimanual interaction, and its digital cousins (variations of the original scene, objects, and tasks) facilitate generalization to new real-world conditions. Across 7 manipulation tasks and 5 policy architectures, SimFoundry simulation evaluations strongly predict real-world performance, with mean Pearson correlation 0.911 and mean maximum ranking violation 0.018. When evaluating sim-trained policies zero-shot in the real world, policies trained with object, scene, and task cousins in simulation show average task success rate improvements of 17%, 21%, and 40%, respectively. Additional details at https://research.nvidia.com/labs/gear/simfoundry/ .

1. Introduction

SimFoundry addresses the cost and scale barriers of real-world robot-policy data collection and evaluation by turning a single video into modular, interactive simulation environments. It reconstructs digital twins, generates object, scene, and task cousins, and supports policy training and benchmarking with strong sim-to-real predictive performance.

  • Motivation: Large-scale robot manipulation data collection is labor-intensive, while systematic real-world evaluation of trained foundation models is difficult to scale.These bottlenecks often require teleoperation efforts spanning many months or years.
  • Motivation: Simulation offers scalable robot-policy training and evaluation through automated generation of diverse demonstrations and simulation-based performance estimates.Recent work has used automated data generation to reduce human effort and improve real-world agents.
  • SimFoundry: SimFoundry converts a single real-world video into interactive, sim-ready environments for policy evaluation and training through a unified modular pipeline.The system combines digital-twin reconstruction, expansion into diverse training environments, and simulation-based policy benchmarking and training.
  • Capabilities: SimFoundry supports rigid and articulated objects, physics annotations, and automated object, scene, and task cousins that expand one reconstructed scene into diverse training environments.Across 12 reconstruction scenes, it achieves zero-shot F1 scores of 0.81–0.92, improving to 0.93–0.99 with only 3 minutes of additional effort.
  • Results: 0.911 mean Pearson correlation and 0.018 MMRV show that SimFoundry simulation evaluations strongly predict real-world performance, outperforming the state-of-the-art baseline by over 0.59 on Pearson correlation.The system also trains policies that transfer to the real world and generalize beyond the reconstructed twin.

2. Related Work

Related work spans 3D asset reconstruction and generation, including retrieval-based alignment, image-conditioned mesh synthesis, and articulated-object modeling. These advances support automated real-to-sim environment construction for policy training and evaluation.

  • 3D Asset Generation and Alignment: 3D asset methods retrieve CAD models, synthesize meshes from limited images, and infer movable parts for articulated objects.The literature includes both retrieval-based alignment and generative approaches for high-fidelity 3D assets.
  • Real-to-Sim for Simulation Environment Creation and Applications: Real-to-sim systems automate simulation-environment construction from real-world captures using high-quality 3D reconstruction and generative synthesis.These systems broadly target either closing the real-to-sim-to-real loop for agent training or providing reconstructed environments for other applications.
  • Method Overview: SimFoundry combines per-object information extraction, 2D-to-3D mesh generation, physical-parameter annotation, and physics-based scene sanity checking.Its simulated augmentations vary objects, scenes, and tasks to create diverse cousins.

3. Preliminaries

This section defines SimFoundry’s reconstruction setting, policy notation, and evaluation measures, while distinguishing strict digital twins from affordance-preserving digital cousins. It also frames synthetic trajectory generation as related simulation data infrastructure.

  • Scene and Policy Definitions: SimFoundry reconstructs real-world scenes in simulation from video using object meshes M_i, scales s_i, and poses p_i generated with multiple foundation models V*.The reconstructed scene is represented as S_sim from an input real scene S_real, with i ∈ {1, ..., N}.
  • Scene and Policy Definitions: A policy π_θ maps current-timestep observations o_t to actions a_t through a neural network parameterized by θ.The policy is defined as π_θ: O → A.
  • Policy Correlation: Real–simulation policy agreement is measured with Pearson correlation coefficient r and mean maximum rank violation (MMRV), ideally satisfying r → 1 and MMRV → 0.Pearson correlation captures linear agreement between real and simulation task results, while MMRV measures average worst rank violation.
  • Digital Twins and Cousins: Digital twins strictly replicate real-scene geometry and object layouts, whereas digital cousins preserve semantic and geometric affordances without explicitly modeling the original scene.Digital cousins are presented as a form of object instance randomization.

4. SimFoundry: A modular, automated real-to-sim generation pipeline

SimFoundry constructs interactive, sim-ready scenes from raw video through Extraction, Generation, and Augmentation, then creates affordance-preserving digital cousins that vary objects, layouts, and tasks. These variations provide controllable diversity and improve robustness and generalization across corresponding changes in real-world conditions.

  • Pipeline overview: SimFoundry uses three stages: Extraction infers per-object information, Generation creates and stabilizes sim-ready assets, and Augmentation produces object, scene, and task variations.The pipeline exports reconstructed scenes to downstream robotics simulators after composing, annotating, and stabilizing the scene.
  • Extraction: Extraction converts a raw RGB video into an RGB-D observation and scene point cloud, then outputs per-object crops and masks for mesh generation and alignment.The process uses estimated depth, camera intrinsics, object detection, iterative segmentation, and image-depth inpainting.
  • Generation: Generation creates visual meshes, estimates object poses, handles articulated parts, assigns collision and physical properties, and composes a stable sim-ready scene.Objects are aligned to reconstructed scene geometry and exported after penetration resolution.
  • Augmentation: Augmentation creates affordance-preserving digital cousins that vary object instances, scene layouts, and task specifications while retaining task-relevant semantics.Object cousins vary geometry, topology, and appearance; scene cousins use semantic spatial predicates and distractors; task cousins generate simulation-compatible goals.
  • Augmentation: Object cousins improve robustness to unseen instances, scene cousins improve generalization to novel layouts, and task cousins improve zero-shot and few-shot downstream task performance.Together, these mechanisms provide controllable diversity across objects, layouts, and tasks.

5. Experiments

The experiments evaluate SimFoundry for real-to-sim policy benchmarking, zero-shot sim-to-real training, diversity-driven generalization, and reconstruction fidelity. SimFoundry closely predicts real-world policy performance, enables transfer across objects, scenes, and tasks, and produces accurate environments at scalable effort.

  • Experimental scope: Experiments span two robot embodiments and manipulation settings including pick-and-place, bimanual coordination, articulated-object interaction, and long-horizon language following.The evaluation covers DROID with a single Franka arm and a bimanual setup with two YAM arms.
  • Real-to-sim policy evaluation: 0.911 mean Pearson correlation and 0.018 MMRV show that SimFoundry scene evaluations closely match real-world results and preserve policy rankings.The benchmark includes zero-shot and 50-demo finetuned policy groups across less difficult and more challenging tasks.
  • Real-to-sim policy evaluation: 0.90 to 0.95 mean Pearson correlation results from sub-task evaluation, especially improving evaluation of multi-step and long-horizon tasks.The procedure helps target future data collection toward difficult sub-tasks that bottleneck overall success.
  • Real-to-sim policy evaluation: 0.59 higher mean Pearson correlation than PolaRiS demonstrates SimFoundry’s advantage under completely zero-shot simulation evaluation without simulator-specific policy adaptation.PolaRiS improves correlation through shallow policy finetuning on its simulation data, whereas SimFoundry uses the same policies without adaptation.
  • Sim-to-real policy training: 99% success on Pot on Stove and 100% success on Stack Dishware show zero-shot transfer from SimFoundry-generated training data to real scenes.The result is obtained across YAM and DROID, using a flow-matching policy trained from scratch for YAM and a finetuned π0.5 DROID checkpoint for DROID.
  • Data diversity and generalization: 31% simulation improvement and 18% real-world improvement over the base DROID checkpoint show that task cousins support generalist policies and few-shot downstream learning.π0.5-FT reaches 29% success on held-out tasks without task-specific demonstrations.
  • Reconstruction analysis: 0.81–0.92 F1 versus 0.66–0.71 for SAM3D shows that fully automated SimFoundry reconstruction achieves higher geometric fidelity.SimFoundry also obtains lower chamfer distance and position error than SAM3D.

6. Limitations

SimFoundry’s modularity depends on off-the-shelf foundation models and assumptions restricting the current pipeline to tabletop layouts. Future work should support multi-level and non-planar environments.

  • 6. Limitations: The system inherits underlying foundation-model failure modes and currently assumes tabletop-style layouts, excluding multi-level or non-planar environments.Relaxing the layout assumption is identified as a natural direction for future work.

7. Conclusion … B. FAQ

SimFoundry presents an automated pipeline for reconstructing interactive, sim-ready scenes and generating digital cousins, with evidence of usefulness for simulation evaluation and sim-to-real policy transfer. The appendix provides supplementary material, including an FAQ covering automation, robotics validation, and physical stability.

  • 7. Conclusion: SimFoundry reconstructs interactive sim-ready scenes from a single video, supports articulated objects, clutter, and occlusion, and generates object, scene, and task cousins.The conclusion describes these capabilities as part of a fully automated pipeline.
  • 7. Conclusion: SimFoundry simulation task-success measurements correlate with real-world policy performance and outperform prior work in sim-based policy evaluations.The conclusion also states that SimFoundry-generated data can train policies transferring to the real world.
  • A. Overview: The Appendix includes an FAQ, limitations discussion, expanded related work, scene reconstruction details, digital-cousin augmentation details, and detailed experiment-results tables.These materials are listed in the appendix overview.
  • B. FAQ: SimFoundry’s distinguishing features are fully automated programmatic reconstruction of scenes, articulated objects, and backgrounds within one pipeline.The FAQ frames this as feature-complete automation compared with alternative simulation-environment methods.
  • B. FAQ: SimFoundry is empirically validated for downstream robotics applications in both real-to-sim evaluation and sim-to-real training settings.The FAQ identifies robotics-task validation as a key distinguishing advantage.
  • B. FAQ: Reconstructed scenes are spawned in PyBullet and stepped until objects settle, guaranteeing physical stability for subsequent initializations.Objects may drift relative to their original fitted poses after settling.
  • B. FAQ: The FAQ directs readers to Appendix D for related-work discussion and Appendix E.4 for additional details on physical-stability testing.The passage also introduces a question about manual effort required for scene production and robot use, but its answer is not included.

C. Limitations · D. Full Related Work · D.1. Real–to–Sim for Simulation Environment Creation and Applications

SimFoundry is limited by nondeterministic VLM outputs, reconstruction and articulation-quality constraints, tabletop assumptions, and substantial single-GPU runtime. Relative to prior real-to-sim and real-to-render work, it combines sim-to-real transfer, correlated policy evaluation, and broader manipulation-task and scene-variation support.

  • C. Limitations: Nondeterministic remote VLM outputs can produce inconsistent inpainting and degenerate or duplicated extracted objects.The issue arises because identical inputs may yield different outputs across runs.
  • C. Limitations: Monocular point-cloud quality can reduce reconstruction accuracy, while articulation depends on difficult 3D mesh segmentation, especially with occluded internal structure.These constraints affect recovered scale, shape, and object articulation results.
  • C. Limitations: The physics-stability procedure assumes objects rest on one flat reference surface, restricting the pipeline to tabletop-style layouts.The paper identifies more complex and varied scenes as future-work territory.
  • C. Limitations: 90 minutes per scene are required for two-pass video inpainting on a single GPU to produce dense, temporally consistent, clean RGB-D streams for splat training.In multi-GPU settings, background reconstruction can run in parallel with Extraction, Generation, and Variation stages.
  • D.1. Real–to–Sim for Simulation Environment Creation and Applications: Prior real-to-sim-to-real manipulation systems typically target single-step rigid-object pick-and-place, whereas SimFoundry supports bimanual, articulated-object, and multi-step manipulation.This comparison concerns the task regimes emphasized in the related work discussion.
  • D.1. Real–to–Sim for Simulation Environment Creation and Applications: Navigation and locomotion real-to-sim approaches address substantially different physics and contact requirements from dexterous manipulation.These approaches close a related real-to-sim-to-real loop for navigation and locomotion.
  • D.1. Real–to–Sim for Simulation Environment Creation and Applications: SimFoundry combines successful sim-to-real transfer and strongly correlated simulated and physical policy evaluation with bimanual, articulated, multi-step tasks and multiple scene-cousin types.It is positioned among a select group demonstrating both transfer and evaluation correlation, while extending task and cousin diversity.
  • D.1. Real–to–Sim for Simulation Environment Creation and Applications: Real-to-render methods avoid physical simulation and its modeling cost but are difficult to apply to higher-precision, contact-rich tasks requiring accurate dynamics.The reconstructed scene is used purely as a rendering target in this line of work.

D.2. Imitation Learning from Human Demonstrations and Synthetic Data Generation. … E.5.2. Manual Background Alignment

SimFoundry reconstructs real-world environments, generates controlled synthetic variations, and trains manipulation agents that transfer to the real world with minimal human effort. Its modular reconstruction pipeline covers extraction, foundation-model processing, articulated-object generation, physical stabilization, and automatic or manual background alignment.

  • D.2. Imitation Learning from Human Demonstrations and Synthetic Data Generation.: SimFoundry uses teleoperation demonstrations as a conventional source of logged robot sensor streams and actions for behavioral-cloning datasets, while reducing human effort through reconstructed environments and synthetic data.The system reconstructs simulation environments with controlled variations, generates synthetic data, and trains agents for real-world transfer.
  • E. Scene Reconstruction: The extraction stage selects a representative RGB-D frame, builds a scene point cloud, iteratively segments and removes foreground objects, and outputs per-object crops and masks.The current video pipeline defaults to frame 0 and asks users to record a clear, preferably front-facing view capturing the whole scene.
  • E.2. Foundation Model Details: SimFoundry is modular: its foundation-model components can be changed during execution, with supported models covering depth, image and video segmentation, scene and image understanding, and inpainting.The listed implementations include DepthAnything3, FoundationStereo, SAM3, Gemini-Pro-3, Gemini-Pro-3-Image-Preview, PriorDepthAnything, SAM2, and VOID.
  • E.3. Articulated Object Generation: Articulated-object generation renders multiview images, predicts articulable parts and joint types, segments meshes with label propagation, generates joint parameters through an API, and predicts physical parameters.The pipeline uses VLM-guided segmentation and joint placement, then estimates link mass, joint friction, and damping using part and object volumes.
  • E.4. Object Depenetration and Physical Stability: To ensure physical stability, SimFoundry generates collision meshes with CoACD, steps the reconstructed scene in PyBullet until poses settle, and caches the resulting poses.Object velocities are reset after each step to avoid potential de-penetration explosions.
  • E.5. Background Reconstruction and Alignment: SimFoundry reconstructs static backgrounds as registered 3D Gaussian Splats through either a fully automatic route from the raw capture or a manual route using a second foreground-free capture.Both routes emit identical asset structures and differ in how the splat is obtained and aligned.
  • E.5.1. Automatic Background Reconstruction and Alignment: The automatic pipeline prepares and inpaints frames, recovers metric depth and poses, trains a depth-supervised splat, and bridges it rigidly into the simulator world frame.It combines original-stream poses, inpainted-stream depth, photometric and L1 depth losses, per-camera pose optimization, and an anchor-frame transform.
  • E.5.2. Manual Background Alignment: The manual route records a second cleared-scene video, trains a splat on clean RGB frames, and uses interactive scene-editor alignment because the captures lack a shared camera trajectory.COLMAP recovers poses and a sparse SfM cloud; foreground-free training typically produces sharper reconstructions, especially on flat surfaces and silhouettes.

E.5.3. Comparison and Trade-offs … F.3. Task Cousins Augmentation

SimFoundry’s reconstruction pipelines produce comparable assets but trade capture effort and processing costs against background fidelity. Its automated digital-cousins system generates identity-preserving object variants, semantically varied scenes, and executable tasks compatible with reconstructed environments.

  • E.5.3. Comparison and Trade-offs: The two reconstruction pipelines emit identical asset structures with comparable reconstruction quality, differing in capture requirements and inherited failure modes.
  • E.5.3. Comparison and Trade-offs: The automatic pipeline uses one existing video without user interaction, while reproducing the original capture’s exact viewpoint geometry.It is preferred when capture effort must be minimized, only one video is available, or clearing the scene is impractical.
  • E.5.3. Comparison and Trade-offs: 90 minutes of foreground inpainting on one NVIDIA RTX 3090 is required for two-pass denoising, although the process can run independently and in parallel across GPUs.
  • E.5.3. Comparison and Trade-offs: The manual pipeline requires a second capture but avoids inpainting artifacts and produces sharper background reconstructions when higher fidelity is the priority.It also requires physical scene access and a brief manual alignment step.
  • F.1. Object Cousins Augmentation: Object-cousin augmentation combines a VLM and image-generation model to preserve object identity while expanding shape, structure, and appearance distributions.It uses segmented object appearance, an isolated transparent-background object image, the original scene image, and a canonicalized object name.
  • F.2. Scene Cousins Augmentation: Scene-cousin augmentation varies object placements through semantic spatial predicates and adds feasible distractor objects to reconstructed scenes.Predicates such as LeftOf, RightOf, InFrontOf, Behind, OnTopOf, and Inside can be selected relative to an anchor object, including multiple predicates together.
  • F.3. Task Cousins Augmentation: Task-cousin augmentation uses visual context and structured scene metadata to rapidly generate varied, executable manipulation tasks compatible with reconstructed scenes.The pipeline extracts a simulated scene image and interactable objects, then applies robot and optional object-level constraints to ensure realism and executability.

F.3.1. Task Cousins Example … G.1.1. Sub-Task Evaluations improve Real-to-Sim Correlations

SimFoundry generates task cousins from a reconstructed cluttered scene and evaluates policies against real-world performance. Sub-task evaluation improves real-to-sim correlation on fine-tuned long-horizon tasks while exposing failure modes.

  • F.3.1. Task Cousins Example: 13 proposed tasks were generated from a reconstructed cluttered scene, with 10 human demonstrations and 100 MimicGen demonstrations collected per task.The demonstrations were used to fine-tune π0.5 and roll out a single multi-task policy in simulation and the real setup.
  • F.3.1. Task Cousins Example: The task-proposal prompt requires exactly the requested number of distinct robot-arm tasks whose goals change the scene configuration and are expressible through object states.The prompt constrains proposals to use only the listed scene objects and excludes goals already satisfied in the initial configuration.
  • G.1. Detailed Results for Real-to-Sim Policy Evaluation: SimFoundry success rates align more closely with real-world results than PolaRiS, where most policies perform poorly, especially on tasks with real-world fine-tuning.Tables G.1 and G.2 report the detailed real-world and simulation success rates used for this comparison.
  • G.1.1. Sub-Task Evaluations improve Real-to-Sim Correlations: The sub-task protocol evaluates remaining subtasks from simulation states where earlier subtasks are already completed.This arbitrary-state reset protocol enables assessment of later subtasks in long-horizon tasks, such as completing Store Marker after opening the cabinet drawer.
  • G.1.1. Sub-Task Evaluations improve Real-to-Sim Correlations: Sub-task evaluations can reveal actionable failure modes for policy improvement, including later-stage behavior after earlier subtasks have succeeded.The protocol is designed to assess subtasks that occur later in long-horizon tasks rather than only evaluating complete-task outcomes.
  • G.1.1. Sub-Task Evaluations improve Real-to-Sim Correlations: 0.951 mean Pearson correlation was achieved with sub-task evaluations, up from 0.902 on fine-tuned tasks.Sub-task evaluations improve correlations for long-horizon fine-tuned tasks and provide actionable insights into policy failure modes.

G.2. Detailed Results for Sim-to-Real Experiments … J.3.3. Real-to-Sim Policy Evaluations in PolaRiS

Across sim-to-real and real-to-sim studies, SimFoundry’s structured object, scene, and task diversity improves robustness, while simulation evaluations closely support real-world policy assessment. The experiments also detail data generation, evaluation protocols, robot platforms, task rubrics, and PolaRiS reconstruction and evaluation procedures.

  • G.2. Detailed Results for Sim-to-Real Experiments: 40% average improvement comes from task cousins, exceeding scene-cousin gains of ∼13% on twin scenes and ∼29% on cousin scenes.Task cousins add related demonstrations sharing objects, predicates, or intermediate behaviors; scene cousins vary semantic object relations and layouts.
  • G.3. Detailed Object Cousin Ablation: 17% average zero-shot sim-to-real success improvement is achieved with object cousins, with gains reported on original and held-out objects.Increasing the number of object cousins generally improves transfer on both reconstructed twin objects and unseen scene objects.
  • H.1. Data Generation Details: ∼10 −15 human demonstrations are augmented with MimicGen through trajectory, material, camera-pose, and DROID table-height randomization.The resulting synthetic datasets are used to train robot learning policies.
  • H.2. Policy Training Details: 10k gradient steps train DROID-finetuned π0 and π0.5 policies, while YAM flow-matching policies train for 40k steps and select checkpoints using periodic simulation evaluation.Real-to-sim experiments also finetune DROID-pretrained π0, π0.5, and GR00T N1.6 checkpoints for three tasks, with additional policies used for simpler tasks.
  • I.1. Robot Embodiments.: Two embodiments are evaluated: the single-arm DROID platform and the bimanual YAM workcell, both using joint-position control during rollout or evaluation.DROID uses external and wrist-mounted ZED cameras and Oculus teleoperation; YAM uses wrist-mounted RealSense cameras, an external top-down camera, and JoyLo teleoperation.
  • I.2. Task Rubric: Task success is binary and requires completing every rubric sub-task, including multi-step actions such as stacking dishware, storing a marker, and throwing away trash.The task set also includes serving fruits, clearing a table, and placing a pot on a stove.
  • J.1. Evaluation Protocol: 25 rollouts per task sample object placements from a 5-by-5 grid, with matched position ranges but potentially different exact simulation and real-world positions.This standardized protocol uses independent sampling without replacement and seeks distributional correspondence while limiting proprioceptive overfitting.
  • J.2. Metrics; J.3. PolaRiS Real-to-Sim Experiment Details; J.3.1. PolaRiS Custom Environment Creation; J.3.3. Real-to-Sim Policy Evaluations in PolaRiS: Pearson correlation and Mean Maximum Rank Violation quantify whether simulation preserves real-world score trends and policy rankings, respectively.PolaRiS provides a browser-based scene composer, but its experiments require external reconstruction software; generalist policies are then evaluated with matched initializations, checkpoints, and rollout protocols.

J.3.4. PolaRiS Results Analysis … L.2.1. Qualitative Reconstruction Results

The section analyzes PolaRiS, interactive human refinement, and reconstruction evaluation across cluttered scenes, finding weaknesses in PolaRiS and stronger SimFoundry reconstruction fidelity. It also documents reconstruction procedures, qualitative outputs, and artifacts from automatic background reconstruction.

  • J.3.4. PolaRiS Results Analysis: PolaRiS showed low correlation with real-world policy evaluation and consistently underestimated policies’ real-world success rates across the evaluated custom-scene tasks.The evaluated policy was cotrained on 10% PolaRiS simulation data and 90% DROID data at 1000 steps.
  • K.1. Human Intervention Details: The unified GUI lets operators constrain extracted objects and adjust generated mesh poses and scales during scene decomposition and generation.This provides accessible touchpoints for tuning intermediate pipeline outputs.
  • K.2. Interactive Pose Refinement: Interactive pose refinement initializes from automatic estimates, visualizes mesh–point-cloud overlays, supports coarse-to-fine 6D pose and scale adjustments, and preserves prior outputs for iterative rollback.The tool is optional because the automatic pipeline can run unattended; refinement is mainly used for Real-to-Sim localization, while broad simulation initializations can reduce its utility for Sim-to-Real generation.
  • L.1. 3D Reconstruction Evaluation Details: The reconstruction evaluation covers 12 tabletop YCB scenes categorized from low to high difficulty by object occlusion and uses staged quasi-ground-truth object poses.Each object’s 6-DoF pose is inferred while fully visible before later objects introduce occlusion.
  • L.1.2. SAM3D Reconstruction Pipeline: SAM3D outputs are transformed into SimFoundry’s shared metric world frame by converting coordinate conventions, applying the common world transform, and resolving global scale ambiguity.After alignment, residual metric differences reflect reconstruction quality rather than frame or scale mismatch.
  • L.1.3. Quantitative Reconstruction Results: SimFoundry outperformed SAM3D zero-shot on Chamfer Distance, F1-Score at threshold 0.01 meters, and Object Bounding Box Position Error, with further gains after minutes of interactive iteration.The qualitative evaluation visualizes all 12 benchmark scenes across the tested clutter and occlusion levels.
  • L.2.1. Qualitative Reconstruction Results: Across five in-the-wild scenes, the automatic background pipeline sometimes produced floating artifacts around support surfaces that partially occluded foreground objects.The reported cause is hallucinated content from the object-removal model during image inpainting; side-by-side comparisons cover manual and automatic pipelines.

L.2.2. Quantitative Reconstruction Results

Quantitative reconstruction quality is assessed by seven complementary render-versus-real metrics across five scenes and 50 frames per scene. The automated pipeline outperforms manual background reconstruction across all reported metrics, with alignment derived analytically from evaluation camera poses.

  • Metric coverage: Seven metrics cover pixel fidelity, structural similarity, color accuracy, and geometric alignment.They include PSNR, SSIM, MAE, RMSE, NCC, EdgeMAE, and ∆E.
  • Metric coverage: PSNR measures global pixel-wise fidelity, while SSIM captures local luminance, contrast, and structural agreement.PSNR is sensitive to global misalignment; SSIM is comparatively tolerant of uniform intensity shifts.
  • Metric coverage: MAE and RMSE quantify per-pixel intensity deviation, with RMSE penalizing large errors more heavily than MAE.MAE averages absolute render-to-ground-truth differences, whereas RMSE is based on squared error.
  • Evaluation protocol: The metrics are evaluated on 50 uniformly sampled frames per scene across five scenes, with per-scene means and cross-scene averages reported.This evaluation is summarized in Table L.6 for dorm_1 through dorm_5.
  • Comparison: The automated pipeline outperforms manual, hand-curated background reconstruction across all reported metrics.Its background-to-world transform is composed analytically from the camera poses defining the evaluation viewpoints, registering the reconstruction to the ground-truth frame by construction.
Loading 2606.28276v4…