Source-linked AI summary

SimFoundry: Modular and Automated Scene Generation for Policy Learning and Evaluation

Nadun Ranawaka, Josiah Wong, Wei-Lin Pai, Wei-Teng Chu, Tianyuan Dai, Masoud Moghani, Hang Yin, Yunfan Jiang, Wesley Durbano, Brandon Huynh, Yu Fang, Danfei Xu, Ruohan Zhang, Li Fei-Fei, Linxi Fan, Bowen Wen, Ajay Mandlekar, Yuke Zhu

arXiv:2606.28276v4cs.RO

TL;DR

Real-world robot-policy evaluation is costly, and constructing aligned simulation scenes remains challenging. SimFoundry turns a single video into interactive simulations for training and evaluation. Across diverse policies, its simulation evaluations closely match real-world results and preserve rankings.

  • Problem

    Manually constructing simulations aligned with real-world scenes and tasks remains challenging, while real-world policy evaluation is costly and requires thousands of trials for rigorous comparisons.

    Source: §1 · p. 2
  • Method

    SimFoundry turns a single video into interactive simulation environments through extraction, sim-ready asset generation, and augmentation with object, scene, and task variations.

    Source: §1 · p. 2, §4 · p. 5
  • Results

    Across diverse policies, SimFoundry evaluations closely match real-world results and preserve policy rankings, with mean Pearson correlation 0.911 and MMRV 0.018.

    Source: §5.1 · p. 7
  • Takeaways & Limitations

    Simulation-based evaluations can inform model development, while digital cousins support robustness to unseen objects, novel layouts, and related downstream tasks.

    Source: §5.1 · p. 7, §7 · p. 9

    The current pipeline's assumptions limit it to tabletop-style layouts.

    Source: §6 · p. 9

Abstract

from arXiv · show

Training and evaluating robot policies in the real world is costly and difficult to scale. We introduce SimFoundry, a modular and automated system for zero-shot real-to-sim scene construction from a video. SimFoundry generates sim-ready digital twins and supports object, scene, and task editing, enabling the automated generation of diverse digital cousins: affordance-preserving variations of reconstructed real-world scenes. Policies trained on SimFoundry data transfer zero-shot to challenging real tasks involving multi-step manipulation, articulated object interaction, and bimanual interaction, and its digital cousins (variations of the original scene, objects, and tasks) facilitate generalization to new real-world conditions. Across 7 manipulation tasks and 5 policy architectures, SimFoundry simulation evaluations strongly predict real-world performance, with mean Pearson correlation 0.911 and mean maximum ranking violation 0.018. When evaluating sim-trained policies zero-shot in the real world, policies trained with object, scene, and task cousins in simulation show average task success rate improvements of 17%, 21%, and 40%, respectively. Additional details at https://research.nvidia.com/labs/gear/simfoundry/ .

1. Introduction

SimFoundry turns a single real-world video into interactive simulation environments for robot-policy training and evaluation. Its modular system also generates affordance-preserving digital cousins to support policy generalization.

  • Simulation can scale demonstration generation and policy evaluation, but manually aligning environments with real scenes and tasks remains difficult, and existing systems often lack the machinery to close the sim-to-real policy-learning loop.Source: §1 · p. 2
  • SimFoundry converts a single video into interactive, sim-ready digital twins and affordance-preserving object-, scene-, and task-level cousins using interchangeable modules from perception through data generation.Source: §1 · p. 2
  • Across 12 reconstruction scenes, zero-shot F1 scores were 0.81–0.92, rising to 0.93–0.99 with 3 minutes of per-object tuning.Source: §1 · p. 2
  • SimFoundry demonstrates manipulation beyond prior real-to-sim work, including multi-step tasks, articulated-object interaction, and bimanual coordination on DROID and YAM embodiments.Source: §1 · p. 2
  • Across 7 tasks and 5 policy types, mean Pearson correlation was 0.911 and mean MMRV was 0.018, exceeding baseline [32] by over 0.59 on Pearson correlation.Source: §1 · pp. 2–3
  • Multi-task SimFoundry data improved generalist policies by up to 31% in simulation and 18% in the real world, while achieving a 29% success rate on held-out real tasks.Source: §1 · p. 3

2. Related Work

Prior work provides tools for 3D asset reconstruction and alignment and builds simulation environments from real-world captures. SimFoundry composes these primitives modularly and extends real-to-sim systems with broader task support and scene, object, and task variations.

3D Asset Generation and Alignment

  • SimFoundry composes retrieval, generative 3D reconstruction, articulated-object modeling, and foundation-model-based scene alignment primitives, with tools swappable and outputs refinable by human intervention.Alignment draws on depth, segmentation, and 6-DoF pose and scale estimation.Source: §2 · p. 3

Real-to-Sim for Simulation Environment Creation and Applications

  • Among real-to-sim systems [27] [34] [101], SimFoundry supports sim-to-real transfer and correlated policy evaluation while handling bimanual, articulated, and multi-step tasks and multiple scene-cousin types.It supports object, scene, and task cousins to increase the diversity of reconstructed environments.Source: §2 · p. 3, Figure 2 · p. 4
  • Prior real-to-sim systems target either agent training [14] [17] [21] [24] [78] [79] [95] [105] or policy evaluation [3] [27] [32] [33] [34] [47] [101] [104], while rendering-only reconstructions [102] limit contact-rich robot tasks.Automated simulation construction from real-world captures has been enabled by high-quality 3D reconstruction and generative 3D synthesis.Source: §2 · p. 3

Method Overview

  • SimFoundry extracts per-object information, generates 3D meshes, and assembles physics-checked scenes; its cousins vary objects, object configurations, or viable scene interactions.Object cousins modify input objects in image space before regenerating meshes; scene cousins change object configuration, and task cousins propose viable interactions.Source: Figure 2 · p. 4

3. Preliminaries

SimFoundry reconstructs real-world scenes from video for simulation and supports real-to-sim policy evaluation and sim-to-real training. The section defines policy-correlation measures and distinguishes digital twins from digital cousins and synthetic trajectories.

3. Preliminaries

  • SimFoundry reconstructs scenes from video into object meshes, scales, and poses, then supports real-to-sim policy evaluation and sim-to-real training.A policy maps current observations to actions through a neural network parameterized by θ.Source: §3 · p. 4
  • Ideal real-simulation policy correlation has Pearson correlation coefficient r→1 and mean maximum rank violation (MMRV)→0; task success requires completing all criteria.Task success is measured end-to-end as a discrete 0 or 1.Source: §3 · p. 4
  • Digital twins replicate real-scene geometry and object layouts, while digital cousins [17] [55] preserve semantic and geometric affordances; MimicGen [37] splices demonstrated subtasks into synthetic trajectories.Digital cousins act as object-instance randomization without explicitly modeling the real scene.Source: §3 · p. 4

4. SimFoundry: A modular, automated real-to-sim generation pipeline

SimFoundry turns video into a physically grounded, sim-ready scene and expands it into affordance-preserving object, layout, and task variations. The paper reports that these variations improve policy generalization across corresponding changes.

Extraction

  • From raw RGB video, Extraction estimates depth, builds a point cloud, aligns the scene, then detects and iteratively segments foreground objects to produce RGB-D crops and masks.Depth is estimated with off-the-shelf models [50] [90], and the ground plane is extracted for simulator-frame alignment.Source: §4 · p. 5

Generation

  • Generation creates visual meshes, aligns and refines object poses, handles articulated parts, assigns collision geometry and physical properties, then stabilizes and exports the composed scene.Articulated objects receive segmented meshes and joint parameters; object penetrations are resolved before export to downstream simulators.Source: §4 · p. 5

Augmentation

  • Object cousins vary geometry, topology, and appearance; scene cousins use semantic spatial relations and optional distractors; task cousins propose feasible goals grounded in scene objects and affordances.Task goals become simulation-compatible specifications for procedural demonstrations, and cousin categories can combine rather than represent mutually exclusive variation types.Source: §4 · pp. 5–6
  • Object cousins improve robustness to unseen instances, scene cousins improve generalization to novel layouts, and task cousins improve zero-shot and few-shot downstream performance.These are the reported effects of the three augmentation mechanisms.Source: §4 · p. 6

Background Reconstruction and Alignment

  • Background reconstruction can fuse foreground meshes with a 3D Gaussian Splat [38], using either an automatic single-video pipeline or a manual second-video capture and interactive alignment.The two routes produce identical asset structures and comparable reconstruction quality, trading capture effort against compute and fidelity on texture-less surfaces and silhouettes.Source: §4 · p. 6

5. Experiments

Experiments evaluate SimFoundry as a zero-shot policy-evaluation benchmark and a source of sim-to-real training data, and analyze reconstruction fidelity and pipeline effort.

5.1. Real-to-Sim Policy Evaluation

  • Mean Pearson correlation is 0.911 and mean maximum ranking violation is 0.018; SimFoundry’s mean Pearson correlation exceeds PolaRiS by over 0.59.These correlations hold across policy types, and the comparison evaluates policies zero-shot in simulation without adapting them to either framework.Source: §5.1 · pp. 7–8, Figure 4 · p. 7
  • Sub-task evaluation raises mean Pearson correlation from 0.90 to 0.95, especially helping evaluations of long-horizon tasks.The procedure can help users target future data collection toward difficult sub-tasks that bottleneck task success.Source: §5.1 · p. 7

5.2. Sim-to-Real Policy Training

  • Sim-trained policies reach 99% success on Pot on Stove with YAM and 100% on Stack Dishware with DROID in zero-shot real-world transfer.The YAM policy is trained from scratch with flow matching; the DROID policy finetunes a π0.5 DROID checkpoint on simulated demonstrations.Source: §5.2 · p. 8
  • On DROID, π0.5 real-world success on Store Marker rises from 60% to 92% with co-training, while π0 gains 36% sim success on Throw Away Trash.The authors report that adding small amounts of real data further boosts performance.Source: §5.2 · p. 8
  • Object cousins yield a 50-point real-world gain on held-out Pot on Stove objects and gains up to 20 points on DROID Throw Away Trash.The reported improvements span both embodiments and include DROID performance in simulation and the real world.Source: §5.2 · p. 8, Figure 5 · p. 8
  • Scene cousins boost DROID Throw Away Trash simulation success by up to 28 points and achieve 16% on Store Marker cousin scenes versus 0% for twin-only training.The latter result is on novel layouts.Source: Figure 5 · p. 8, §5.2 · p. 9
  • SimFoundry-finetuned policies outperform the base DROID checkpoint by up to 31% in simulation and 18% in the real world, while π0.5-FT reaches 29% on held-out tasks without task-specific demonstrations.With the total number of demonstrations fixed, 13 task cousins increase simulated success by 60% on Throw Away Trash and 40% on Store Marker.Source: Figure 5 · p. 8, §5.2 · p. 9, Table 2 · p. 9

5.3. System Analysis

  • Fully automated SimFoundry reconstruction achieves F1 scores of 0.81–0.92 versus SAM3D’s 0.66–0.71, and roughly 5 minutes per object; 3 minutes of tuning raises F1 to 0.93–0.99.The passage also reports lower chamfer distance and position error than SAM3D, with tuning gains across every metric.Source: §5.3 · p. 9

6. Limitations

SimFoundry’s reliance on off-the-shelf foundation models enables modularity but inherits their failure modes, while assumptions in the current pipeline limit it to tabletop-style layouts.

6. Limitations

  • The system inherits failure modes from its off-the-shelf foundation models and is limited by assumptions to tabletop-style layouts; supporting multi-level or non-planar environments is future work.Using replaceable foundation-model components enables broad modularity, but relaxing the layout assumption is a stated direction for future work.Source: §6 · p. 9

7. Conclusion … B. FAQ

SimFoundry automates video-based reconstruction of sim-ready scenes and generates object, scene, and task cousins; its simulations support policy evaluation and training that transfer to real-world robotics tasks.

7. Conclusion

  • SimFoundry reconstructs interactive sim-ready scenes from a single video, generates object, scene, and task cousins, and supports real-to-sim evaluation and sim-to-real policy training.The paper reports that simulation success correlates with real-world policy performance and outperforms prior work in sim-based evaluations; cousins improve robustness to unseen objects, novel layouts, and related tasks.Source: §7 · p. 9

A. Overview

  • The appendix covers limitations, related work, reconstruction and cousin-generation methods, detailed experiments, policies, robot platforms, user interaction, and system analysis.It also provides additional real-to-sim policy evaluation details.Source: A. Overview · p. 18

B. FAQ

  • SimFoundry combines automated programmatic scene, articulated-object, and background reconstruction in one pipeline, with validation in real-to-sim evaluation and sim-to-real training.Source: B. FAQ · p. 19
  • Reconstructed scenes are settled in PyBullet for physical stability, and experimental environments require a few minutes of human interactive tuning after initial smartphone scans.Settling can drift objects from their fitted poses; the pipeline itself generates the environments automatically before operator tuning.Source: B. FAQ · p. 19
  • Roughly 5 minutes per object is the average scene-reconstruction time; transparent-object support depends on the mesh model, with TRELLIS.2 supporting transparency unlike default Hunyuan 2.1.Source: B. FAQ · p. 19
  • On Store Marker and Throw Away Trash, both DROID-pretrained checkpoints achieve 0% success; on Stack Dishware, π0-DROID gets 52% and π0.5-DROID 48%, rising to 100% with sim-only finetuning.The reported results indicate that the pretrained checkpoints perform poorly without task-specific finetuning data, while SimFoundry data helps finetune them on difficult tasks.Source: B. FAQ · p. 19
  • Binary end-to-end success is used because it requires completing the full task under reconstructed dynamics, visuals, and geometry, making strong correlations a stricter test of real-to-sim fidelity.Normalized rewards may give partial credit for intermediate subtasks; success-rate correlations are particularly difficult for long-horizon tasks and real-world-finetuned policies when reconstruction errors cause failures.Source: B. FAQ · p. 19

C. Limitations · D. Full Related Work · D.1. Real–to–Sim for Simulation Environment Creation and Applications

SimFoundry’s limitations include nondeterministic VLM outputs, reconstruction and articulation constraints, tabletop-only stability assumptions, and costly background processing. Related work positions it among real-to-sim systems that support diverse manipulation tasks, scene cousins, sim-to-real transfer, and correlated policy evaluation.

C. Limitations

  • Remote third-party VLMs can produce nondeterministic outputs from identical inputs, including inconsistent inpainting that yields degenerate or duplicated objects.Source: C. Limitations · p. 20
  • Monocular point-cloud quality can limit reconstructed scene scale and shape, while articulation depends on accurate 3D mesh segmentation that can be difficult with generated meshes or occluded interiors.Source: C. Limitations · p. 20
  • SimFoundry’s physics-stability procedure assumes a single flat reference surface, restricting the pipeline to tabletop-style scenes.Future work could extend it to more complex and varied scenes.Source: C. Limitations · p. 20
  • Roughly 90 minutes per scene on a single GPU are needed for two-pass video inpainting, although multi-GPU parallelism can hide much of this runtime.Some robotics experiments use background meshes because generated 3DGS can suffer near-field clipping and increase rendering latency.Source: C. Limitations · p. 20

D.1. Real–to–Sim for Simulation Environment Creation and Applications

  • Prior real-to-sim work also targets navigation [14] and locomotion [21], whose physics and contact requirements differ substantially from dexterous manipulation.Source: §D.1 · p. 21
  • Prior reconstructed-simulation evaluation systems [3] [27] [32] [33] [34] [47] [101] [104] correlate with real-world policy evaluations, but several do not demonstrate training real-world agents and none, to the authors’ knowledge, support general articulated objects.Source: §D.1 · p. 21
  • Real-to-render methods [102] avoid physical simulation, but are difficult to apply to high-precision, contact-rich tasks requiring accurate dynamics.Source: §D.1 · p. 21
  • Unlike prior manipulation systems [17] [24] [78] [79] [95] [105], SimFoundry supports bimanual, articulated-object, and multi-step tasks and generates object, scene, and task cousins to improve transfer.It extends the digital-cousins formulation of [17]; SimFoundry also belongs to systems [27] [34] [101] demonstrating both sim-to-real transfer and strong correlation between simulated and physical policy evaluations.Source: §D.1 · p. 21

D.2. Imitation Learning from Human Demonstrations and Synthetic Data Generation. … E.5.2. Manual Background Alignment

SimFoundry uses reconstructed scenes and controlled variations to generate simulation data for robot policies that transfer to the real world. Its reconstruction pipeline combines modular foundation models with automated and user-guided methods for building articulated objects and aligning scene backgrounds.

D.2. Imitation Learning from Human Demonstrations and Synthetic Data Generation.

  • SimFoundry reconstructs real environments and controlled variations, generates synthetic simulation data, and trains manipulation policies that transfer to the real world with minimal human effort.The paper frames synthetic data generation as a way to ease the time and expense of collecting real-world demonstrations.Source: §D.2 · p. 21

E.3. Articulated Object Generation

  • Articulated-object generation segments meshes into semantic parts, predicts joint parameters, and iteratively refines simulated motion with critic feedback before assigning physical parameters.A VLM identifies articulatable parts and joints; over-segmented meshes are merged by part, optionally refined in a GUI, and joint placements are evaluated in a simulator.Source: Figure E.1 · p. 22, §E.3 · p. 23

E.1. Extraction Details

  • Video extraction selects a representative RGB-D frame, builds a point cloud, and iteratively segments and inpaints foreground objects to produce per-object crops and masks.Frame 0 is the default; users are advised to record a clear, ideally front-facing view that captures the whole scene while minimizing occlusion.Source: §E.1 · p. 22

E.2. Foundation Model Details

  • The modular pipeline supports task-specific foundation models, including DepthAnything3 or FoundationStereo for depth, SAM3 for image segmentation, Gemini for VLM tasks, and multiple mesh and pose models.It also supports SAM2 for video-mask propagation and VOID for video inpainting; mesh segmentation options include P3-SAM, Segment Any Mesh, and Partfield.Source: §E.2 · pp. 22–23

E.4. Object Depenetration and Physical Stability

  • To prevent interpenetration, SimFoundry settles reconstructed objects in PyBullet using collision meshes, zeroing velocities between simulation steps and caching the final poses for stable initialization.Collision meshes are generated with CoACD.Source: §E.4 · p. 23

E.5. Background Reconstruction and Alignment; E.5.1. Automatic Background Reconstruction and Alignment

  • Background reconstruction offers automatic alignment from the original capture or manual alignment from a clean recapture; the automatic route trains a depth-supervised splat and registers it to simulator coordinates.It uses original-stream camera poses and inpainted-stream depth; a per-camera pose optimizer corrects residual misalignment that otherwise makes splats blurry.Source: §E.5 · p. 24, §E.5.1 · pp. 24–25

E.5.2. Manual Background Alignment

  • When users can clear the environment, a second foreground-free capture enables a sharper splat, which they align to the reconstructed scene through interactive translation, rotation, and scale controls.Because the clean capture lacks a shared camera trajectory with the original, its COLMAP world cannot be aligned automatically; the editor serializes the resulting transform for later launches.Source: §E.5.2 · p. 25

E.5.3. Comparison and Trade-offs … F.3. Task Cousins Augmentation

SimFoundry’s reconstruction pipelines trade capture convenience against background fidelity and alignment, while its digital-cousin methods automate object, scene, and task variation for reconstructed scenes.

E.5.3. Comparison and Trade-offs

  • The two reconstruction pipelines produce identical asset structures and comparable reconstruction quality, differing in capture requirements and inherited failure modes.The automatic pipeline suits limited capture effort or a need for reproducible viewpoint geometry; the manual pipeline suits cases where background fidelity is the priority.Source: §E.5.3 · pp. 25–26
  • The automatic reconstruction pipeline uses one video without interaction and preserves exact capture-viewpoint geometry, but inpainting can hallucinate details and takes about 90 minutes on one RTX 3090.Its foreground-inpainting step can run independently and in parallel in a multi-GPU setting.Source: §E.5.3 · pp. 25–26
  • The manual pipeline avoids inpainting artifacts and sharpens difficult background surfaces, but requires a second cleared-scene capture and manual alignment that is less precise.It is preferable when reconstruction quality dominates and a second capture is feasible.Source: §E.5.3 · p. 26

F.1. Object Cousins Augmentation

  • Object cousins are generated with a VLM and image-generation model to preserve object identity while extending shape, structure, and appearance.The pipeline canonicalizes the object and extracts its scene context, decomposes it into grasp-relevant functional components, and proposes realistic geometry, topology, or visual changes applied component-wise; implausible or scene-inconsistent outputs are discarded.Source: §F.1 · p. 27

F.2. Scene Cousins Augmentation

  • Scene cousins vary object placements using task-dependent spatial predicates and add feasible distractors selected from BEHAVIOR while avoiding collisions.An anchor object is selected automatically from the task by a VLM or by the user; distractors can be filtered by mass, volume, density, and category.Source: §F.2 · pp. 27, 29

F.3. Task Cousins Augmentation

  • Task proposals use scene context, available objects, robot constraints, and allowable object-state predicates to produce structured tasks compiled into standardized files.The VLM is instructed to propose distinct tasks that require meaningful state changes from the initial scene configuration.Source: §F.3 · p. 29
  • The task-cousin pipeline rapidly generates varied, executable task distributions compatible with reconstructed scenes, avoiding manual task engineering.It uses scene images, interactable-object metadata, robot and object constraints, and VLM-generated logical goal conditions that require a meaningful state change.Source: §F.3 · p. 29

F.3.1. Task Cousins Example … G.1.1. Sub-Task Evaluations improve Real-to-Sim Correlations

The appendix shows how task cousins can scale multi-task policy training and reports that SimFoundry evaluations align closely with real-world performance. Evaluating from completed sub-task states further improves correlations on fine-tuned tasks.

F.3.1. Task Cousins Example

  • The task-cousins example uses 13 proposed tasks to train one multi-task π0.5 policy for simulation and real-world rollout.The pipeline collected 10 teleoperated demonstrations per task, then used MimicGen [58] to generate 100 demonstrations per task.Source: §F.3.1 · p. 29

G. Detailed Experiment Results

G.1. Detailed Results for Real-to-Sim Policy Evaluation

  • SimFoundry success rates align more closely with real-world performance than PolaRiS, where most policies perform poorly, especially with real-world finetuning.Table G.1 reports per-task Pearson r and MMRV across evaluated policies; Table G.2 reports PolaRiS success rates.Source: Table G.1 · p. 34, Table G.2 · p. 34, §G.1 · p. 34

G.1.1. Sub-Task Evaluations improve Real-to-Sim Correlations

  • The sub-task protocol starts evaluation from simulated states where initial sub-tasks are already complete, testing the policy on the remaining steps.It compares success from those states with full end-to-end task success in the real world; Store Marker evaluations begin with the cabinet drawer open.Source: §G.1.1 · p. 34
  • Starting Store Marker with its drawer open shows π0.5 can almost always finish the remaining task in both simulation and reality.Sub-task evaluations can expose failure modes and provide actionable insights for improving policies on long-horizon fine-tuned tasks.Source: §G.1.1 · p. 34
  • Mean Pearson correlation on fine-tuned tasks rises from 0.902 to 0.951 with sub-task evaluations.The reported improvement applies to fine-tuned tasks.Source: §G.1.1 · p. 34, Table G.3 · p. 35

G.2. Detailed Results for Sim-to-Real Experiments … J.3.3. Real-to-Sim Policy Evaluations in PolaRiS

SimFoundry’s structured object, scene, and task variations improve sim-to-real policy transfer, while its real-to-sim evaluations test whether simulation predicts real performance and compare SimFoundry with PolaRiS. The appendices detail data and policy training, evaluation protocols, and PolaRiS scene construction and evaluation.

G.3. Detailed Object Cousin Ablation

  • Increasing object cousins generally improves zero-shot sim-to-real transfer on both original twin-scene objects and held-out unseen objects.The bimanual ablation compared training with the twin alone or 1, 3, or 9 cousins, using 1,000 demonstrations and 25 evaluation trials.Source: §G.3 · p. 36

G.2. Detailed Results for Sim-to-Real Experiments

  • Figure G.2 reports average zero-shot sim-to-real success gains of 17% for object cousins, 21% for scene cousins, and 40% for task cousins.Object variants preserve affordances; scene variants change layouts; task cousins add related-task demonstrations. The prose separately reports ∼13% improvement on twin scenes and ∼29% on cousin scenes.Source: §G.2 · pp. 35–36, Figure G.2 · p. 37
  • Figure G.2 reports that scene cousins improve performance by an average 21%, including a 28% gain on Throw Away Trash in the twin scene and 16% success on held-out Store Marker cousin layouts versus 0% for twin-only.Source: Figure G.2 · p. 37

G.3. Detailed Object Cousin Ablation

  • Figure G.2 reports that co-training raises π0.5 real-world Store Marker success from 60% to 92% and improves π0 simulated Throw Away Trash success by 36%.The prose says adding real demonstrations further improves performance in most DROID settings.Source: §G.2 · p. 36, Figure G.2 · p. 37

H.1. Data Generation Details

  • SimFoundry datasets begin with ∼10–15 human-controlled demonstrations, then use MimicGen and visual randomization to augment trajectory and appearance diversity.Randomization includes materials and camera poses, plus table height in the DROID setup; the resulting datasets train policies for zero-shot real-world deployment.Source: §H.1 · p. 39

H.2. Policy Training Details

  • DROID experiments fine-tune pretrained π0 and π0.5 checkpoints, while YAM tasks use flow-matching policies; the study covers single-arm DROID and bimanual YAM robots.DROID policies train for 10k steps and YAM policies for 40k; task success is binary and requires completing all subtasks.Source: §H.2 · p. 39, §I.1 · p. 40, §I.2 · p. 40

J.1. Evaluation Protocol

  • Real-to-sim evaluation uses 25 rollouts per task with fixed, independently sampled object positions from 5-by-5 grids, and reports Pearson correlation and Mean Maximum Rank Violation.Simulation and real-world position ranges match, but exact positions may differ; task success is the main metric, and the two reported measures assess score agreement and ranking errors.Source: §J.1 · p. 42, §J.2 · pp. 42–43

J.3. PolaRiS Real-to-Sim Experiment Details

  • SimFoundry retains significantly higher simulation-to-real success correlation than PolaRiS in the reported comparison; PolaRiS evaluations use the same initializations, checkpoints, and rollout protocol.PolaRiS scenes were built using COLMAP and 2DGS reconstructions, then modified with an invisible collider plane and matched camera viewpoint; five generalist policies were evaluated.Source: §J.3.1 · p. 43, §J.3.3 · p. 44, §J.3.2 · p. 44

J.3.4. PolaRiS Results Analysis … L.2.1. Qualitative Reconstruction Results

SimFoundry’s reconstruction outperforms SAM3D in zero-shot fidelity and improves further with brief human refinement, while the appendices describe how reconstructions are evaluated, refined, and compared. PolaRiS evaluations correlate poorly with real-world performance, and automatic background reconstruction can introduce floating artifacts.

J.3.4. PolaRiS Results Analysis

  • PolaRiS evaluations correlate poorly with real-world policy evaluation, consistently underestimating success; shallow simulation finetuning is crucial for improving correlation [32].The reported π0.5 checkpoint was cotrained on 10% PolaRiS simulation data and 90% DROID data for 1000 steps, but it was excluded from the zero-shot correlation comparison.Source: §J.3.4 · p. 44

K.1. Human Intervention Details

  • The unified GUI lets operators constrain extracted objects and quickly adjust generated mesh poses and scales within an otherwise automated pipeline.This human intervention provides accessible touchpoints for tuning intermediate outputs.Source: §K.1 · p. 45

K.2. Interactive Pose Refinement

  • Interactive pose refinement starts from the automatic pose estimate, overlays the mesh on the scene point cloud, and supports adjustable keyboard-based pose and scale edits.Operators can toggle or change point-cloud density, switch from coarse to fine translation and rotation steps, save transforms for downstream use, and retain prior versions for rollback or continued refinement. It is used chiefly to improve object alignment for Real-to-Sim experiments; for Sim-to-Real generation, broad enough spatial initialization ranges reduce its utility.Source: §K.2 · p. 45

L.1. 3D Reconstruction Evaluation Details

  • Reconstruction evaluation covers 12 tabletop YCB scenes spanning low to high object occlusion, using staged, unoccluded object poses as quasi-ground truth for methods evaluated on the final occluded image.FoundationPose and each object’s known CAD mesh are used to infer its pose while visible; the poses are recorded as the scene is incrementally staged.Source: §L.1 · p. 47, §L.1.1 · p. 47

L.1.2. SAM3D Reconstruction Pipeline

  • SAM3D outputs are aligned with SimFoundry in a shared metric world by converting coordinate conventions, applying the same world transform, and estimating a scene-level scale.The scale is estimated by minimizing Chamfer distance between SAM3D’s merged point cloud and the ground-truth point cloud; SAM3D supplies object geometry and relative poses, so remaining metric differences reflect reconstruction quality rather than frame or scale mismatch.Source: §L.1.2 · p. 47

L.1.3. Quantitative Reconstruction Results

  • SimFoundry outperforms SAM3D in zero-shot reconstruction fidelity and improves further after a few minutes of interactive human iteration.Fidelity is measured by Chamfer Distance, F1-Score at a 0.01-meter threshold, and Object Bounding Box Position Error against quasi-ground-truth; the supplied passages do not provide the table’s metric values.Source: §L.1.3 · p. 47, Table L.2 · p. 48

L.1.4. Qualitative Reconstruction Results

  • Visual reconstructions are presented for all 12 benchmark scenes across the evaluation’s range of clutter and occlusion.The scenes are categorized as easy (no occlusion), mid (slight occlusion), and hard (strong occlusion).Source: §L.1.4 · p. 47, Table L.1 · p. 48

L.2.1. Qualitative Reconstruction Results

  • In five in-the-wild scenes, the automatic background pipeline can create floating support-surface artifacts that partially occlude foreground objects.The artifacts may occur when the object-removal model hallucinates content during image inpainting; side-by-side visuals compare the manual and automatic pipelines.Source: §L.2.1 · p. 48, §L.2 · p. 48

L.2.2. Quantitative Reconstruction Results

SimFoundry’s automated background reconstruction agrees with real video frames better than manual, hand-curated reconstruction across all reported metrics. The evaluation combines pixel, structural, color, and geometric-alignment measures over five scenes.

L.2.2. Quantitative Reconstruction Results

  • Seven complementary metrics cover pixel fidelity (PSNR, MAE, RMSE), structural and perceptual similarity (SSIM, EdgeMAE), color accuracy (∆E), and geometric alignment (NCC).PSNR and SSIM are higher-is-better; MAE, RMSE, ∆E, and EdgeMAE are lower-is-better, while NCC is higher-is-better.Source: §L.2.2 · pp. 48–49
  • Each metric is evaluated on 50 uniformly sampled frames per scene across five scenes, with per-scene means and a cross-scene average reported in Table L.6.The comparison includes the real scene and manual and automated background pipelines for five dorm scenes.Source: §L.2.2 · p. 49, Table L.6 · p. 52, Table L.5 · p. 52
  • The automated pipeline outperforms manual, hand-curated background reconstruction across all reported metrics.Its background-to-world transform is composed analytically from the camera poses defining evaluation viewpoints, keeping the reconstruction registered across views; manual alignment approximates the transform by eye. The gains are clearest in NCC, which isolates spatial alignment.Source: §L.2.2 · p. 49, Table L.6 · p. 52
Loading 2606.28276v4…