PaperlayerView exact revision on arXiv

Source-linked AI summary

Automated Creation of Digital Cousins for Robust Policy Learning

Tianyuan Dai, Josiah Wong, Yunfan Jiang, Chen Wang, Cem Gokmen, Ruohan Zhang, Jiajun Wu, Li Fei-Fei

arXiv:2410.07408v3cs.RO

TL;DR

Real-world robot-policy training is costly and difficult to scale, while simulation and exact digital twins have transfer and generalization limitations. The paper proposes automated creation of digital cousins and policy training across them; cousin-trained policies show comparable in-distribution performance, stronger out-of-distribution robustness, and zero-shot sim-to-real transfer.

  • Problem

    Real-world robot training is unsafe, costly, and difficult to scale, while simulated policies face semantic and physics disparities and digital twins are labor-intensive and limited to one scene instance.

    Source: §1 · p. 2

  • Method

    The paper defines digital cousins as assets or scenes preserving geometric and semantic affordances without exact replicas, and presents ACDC to generate them from images for policy training.

    Source: §1 · p. 2

  • Results

    Digital-cousin policies show comparable in-distribution and more robust out-of-distribution performance than digital-twin policies, and can enable zero-shot sim-to-real transfer.

    Source: §3.4 · p. 7, §3.2 · p. 7, §3.3 · p. 7

  • Takeaways & Limitations

    Digital cousins provide a way to train policies across scene variations without reconstructing exact twins, with reported transfer to the original real-world scene.

    Source: §1 · p. 2, §5 · p. 8

    ACDC is bounded by its asset dataset, which the authors find insufficient to densely capture the real-world object distribution.

    Source: §5 · p. 8

Abstract

from arXiv · show

Training robot policies in the real world can be unsafe, costly, and difficult to scale. Simulation serves as an inexpensive and potentially limitless source of training data, but suffers from the semantics and physics disparity between simulated and real-world environments. These discrepancies can be minimized by training in digital twins, which serve as virtual replicas of a real scene but are expensive to generate and cannot produce cross-domain generalization. To address these limitations, we propose the concept of digital cousins, a virtual asset or scene that, unlike a digital twin, does not explicitly model a real-world counterpart but still exhibits similar geometric and semantic affordances. As a result, digital cousins simultaneously reduce the cost of generating an analogous virtual environment while also facilitating better robustness during sim-to-real domain transfer by providing a distribution of similar training scenes. Leveraging digital cousins, we introduce a novel method for their automated creation, and propose a fully automated real-to-sim-to-real pipeline for generating fully interactive scenes and training robot policies that can be deployed zero-shot in the original scene. We find that digital cousin scenes that preserve geometric and semantic affordances can be produced automatically, and can be used to train policies that outperform policies trained on digital twins, achieving 90% vs. 25% success rates under zero-shot sim-to-real transfer. Additional details are available at https://digital-cousins.github.io/.

1 Introduction

The paper introduces digital cousins: virtual assets or scenes that preserve geometric and semantic affordances without reconstructing exact real-world counterparts. It presents ACDC as an automated way to create them and train policies for transfer to real scenes.

Motivation

  • Digital twins capture scene details but are labor-intensive to create, and policies trained on them are optimized for one real-world instance.Such policies cannot generalize to variations in the original scene.Source: §1 · p. 2

Digital cousins

  • Digital cousins preserve similar geometric and semantic affordances without explicitly modeling an exact real-world counterpart.Unlike a digital twin, a cousin may differ in material, detailing, or individual object models.Source: §1 · p. 2
  • Multiple cousins can represent one real-world scene, relaxing exact-replica requirements while supporting automated creation and training across scene variations.The paper links this relaxation to less manual fidelity tuning and improved robustness to variations in the original scene.Source: §1 · p. 2

ACDC

  • ACDC matches digital cousins to objects from a real-world image using DINOv2 feature distances, then generates an interactive scene for zero-shot policy deployment.The pipeline extracts per-object information, matches assets, and post-processes the selected cousins into a simulated scene.Source: §1 · p. 2, Figure 2 · p. 3

Contributions

  • Policies trained on digital cousins match digital-twin performance and outperform twin-trained policies on unseen objects in simulation and the original real-world scene.The paper also presents an automated recipe for training policies in simulation.Source: §1 · p. 2

2 Methodology

ACDC turns a single RGB image into a fully interactive digital-cousin scene through extraction, asset matching, and scene generation. The authors then use automated demonstrations and domain randomization to train policies across those scenes.

2.1 Automated Creation of Digital Cousins (ACDC)

  • ACDC extracts object masks, matches each detected object to digital cousins, and compiles the selected assets into a physically plausible interactive scene.The pipeline consists of extraction, matching, and scene-generation steps.Source: §2.1 · p. 3
  • From a calibrated RGB image, ACDC uses generated captions and GroundedSAM-v2 masks, then estimates depth to form per-object point clouds and image representations.The depth map supports positioning and rescaling matched cousins during scene generation.Source: §2.1 · p. 3
  • Matching first filters asset categories with CLIP label similarity, then ranks models and their snapshots using DINOv2 feature distances.The asset dataset is assumed to provide semantic categories and multiple snapshots per model; this work uses BEHAVIOR-1K assets.Source: §2.1 · p. 4
  • Scene generation aligns each matched asset's bounding box to the input object's point cloud, fits floor and wall planes, and de-penetrates objects for stability.GPT-4 is queried to determine whether objects should be mounted on the floor or wall.Source: §2.1 · p. 4

2.2 Policy Learning

  • Policy learning uses scripted demonstrations and sample-based Open, Close, Pick, and Place skills, avoiding the need for human demonstrations.The authors collect demonstrations across cousin scenes with visual, physics, kinematic, and instance-level randomization, then train a behavior-cloning policy.Source: §2.2 · p. 4

3 Experiments

The experiments assess whether ACDC can generate semantically and spatially faithful digital cousins, support robust policy learning, and enable zero-shot sim-to-real transfer. Policies trained on cousins show comparable in-distribution performance to digital-twin policies, stronger out-of-distribution robustness, and successful real-world deployment in the reported settings.

3.1 Digital Cousin Scene Generation via ACDC

  • For sim-to-sim scene-generation evaluation, the authors measure category and model preservation alongside pose and scale fidelity, while real-to-sim scenes receive qualitative side-by-side assessment.The supplied Table 1 caption defines the reported metric categories but does not provide cell values.Source: §3.1 · p. 5, Table 1 · p. 5
  • ACDC-generated cousins preserve scene-level semantic and spatial details and produce physically plausible real-to-sim scenes; sim-to-sim results also show that categories, models, scales, and positions are often reproduced.The real-world comparison is qualitative because the authors lack real-object digital twins and ground-truth spatial information.Source: §3.1 · pp. 5–6

3.2 Sim-to-Sim Policy Learning with Digital Cousins

  • Policy-learning experiments compare cousin-trained policies with twin-only and all-feasible-asset policies across Door Opening, Drawer Opening, and Putting Away Bowl tasks.Training uses 10,000 programmatic demonstrations per task, except Putting Away Bowl, which uses 2,000; each policy is evaluated over 100 rollouts across six runs.Source: §3.2 · p. 6
  • Cousin-trained policies can perform similarly to twin-trained policies on the original setup, whereas policies trained on all feasible assets perform much worse than the twin policy.The authors describe cousins as a more beneficial, conditional form of randomization than training across all feasible assets.Source: §3.2 · p. 7
  • In held-out setups unseen by both policy types, cousin-trained policies are more robust while twin-trained policies degrade, and twin performance generally falls as DINOv2 distance increases.The authors suggest DINOv2 distance may serve as a proxy for out-of-distribution performance.Source: Figure 4 · p. 6, §3.2 · p. 7

3.3 Sim-to-Real Policy Learning with Digital Cousins

  • The sim-to-real evaluation trains an ACDC-matched digital-cousin policy and multiple digital-twin baselines using a real-world IKEA cabinet and its corresponding digital twin, then evaluates them zero-shot on the real cabinet.Source: §3.3 · p. 7

3.4 Real-to-Sim-to-Real Scene Generation and Policy Learning

  • The experiments also test the full real-to-sim-to-real pipeline in a kitchen, where a policy trained exclusively in simulation on cousins successfully opens a cabinet.The reported in-the-wild demonstration is referenced in Figure 1.Source: §3.4 · p. 7
  • Digital-cousin policies show comparable in-distribution and stronger out-of-distribution performance than digital-twin policies, enable zero-shot real transfer, and open an in-the-wild cabinet after simulation-only training.The real-world evaluation uses door opening on an IKEA cabinet; the in-the-wild kitchen cabinet result is also reported.Source: §3.4 · p. 7, §3.3 · p. 7

4 Related Work

The related work covers real-to-sim scene creation, synthetic-data methods for robot learning, and sim-to-real transfer. ACDC is positioned as an automated scene-generation approach that uses physically plausible cousins to broaden simulation training coverage.

Real-to-Sim Scene Creation for Robotics

  • Prior real-to-sim scene methods include manual curation, procedural generation, few-shot interaction, inverse graphics, and foundation-model-assisted generation, with trade-offs involving scene-level handling, human labor, or physical plausibility.The authors characterize ACDC as fully automated and its recovered cousins as faithful to input physical scenes.Source: Real-to-Sim Scene Creation for Robotics · p. 8
  • Robot-learning synthetic-data approaches use simulation skills, task and motion planning, reinforcement learning, or human demonstrations; ACDC also uses primitive skills, while emphasizing physically plausible reconstructed scenes.The authors state physical plausibility eases policy learning and better facilitates transfer to real hardware.Source: Real-to-Sim Scene Creation for Robotics · p. 8
  • Sim-to-real methods include domain randomization, system identification, and simulator augmentation; ACDC extends diverse-scene training by using a wider distribution of digital cousins than nearest-asset training.The passage situates prior transfer demonstrations across manipulation, locomotion, and flight.Source: Real-to-Sim Scene Creation for Robotics · p. 8

5 Conclusion

The conclusion presents ACDC as a fully automated way to generate digital cousins from a single real-world RGB image. It reports comparable in-domain performance, stronger out-of-domain generalization than digital twins, and zero-shot transfer, while noting constraints from asset diversity and pretrained models.

5 Conclusion

  • ACDC generates digital cousins from a single real-world RGB image, and cousin-trained policies have comparable in-domain performance, superior out-of-domain generalization, and zero-shot sim-to-real transfer.The conclusion characterizes the pipeline as fully automated.Source: §5 · p. 8
  • The pipeline is bounded by the diversity of its asset dataset, whose thousands of BEHAVIOR-1K assets remain insufficient to densely capture the real-world object distribution.The conclusion also notes that reliance on large pretrained models carries their limitations, including adversarial and out-of-distribution scenes, and that cousin-based policy learning could improve.Source: §5 · p. 8

A.1 Offline Dataset Generation

The appendix details how ACDC preprocesses assets, classifies object mounting, reconstructs physically plausible scenes, collects demonstrations, and matches cousins using DINOv2. It also describes implementation choices and optional heuristics that support scene and affordance fidelity.

A.1 Offline Dataset Generation

  • Offline preprocessing rotates each asset in BEHAVIOR-1K and captures fixed-view snapshots, representing assets with visual images for later matching; this stage runs once and can be cached.The dataset contains over 10,000 object assets, and each asset has a semantic category annotation.Source: §A.1 · p. 17

A.2 Mounting Type

  • GPT classifies objects as wall-mounted, floor-or-object placed, or mixed wall-contact cases, and can identify the specific wall for mounted or mixed objects.Users may disable the second prompt and distinguish only wall-mounted from non-wall-mounted objects.Source: §A.2 · p. 17

A.3 Generated Scene Post-Processing

  • Scene post-processing infers support relationships from vertical order and overlapping projected boxes, then adjusts assets by mounting type and resolves pairwise collisions.An object is considered on another when projected overlap exceeds 70% of the smaller polygon area; wall-mounted objects are aligned to the wall plane.Source: §A.3 · pp. 17–18

A.4 Skill Definition

  • Programmatic skills chain collision-free motion planning and grasp generation, while articulated-object opening trajectories are computed from handle geometry and joint properties.The appendix names CuRobo for motion planning and GPG for sampling-based grasp generation.Source: §A.4 · p. 18

A.5 Demonstration Collection

  • Demonstration collection uses task-specific skill sequences and saves only successful episodes, while randomizing robot and scene-object poses and scales between episodes.Putting Away Bowl uses Open, Pick, Place, and Close; door and drawer opening use Open.Source: §A.5 · p. 19

A.6 Using DINOv2 for Digital Cousin Matching

  • DINOv2 matching ranks candidates by accumulated per-pixel nearest-neighbor feature correspondences, and the embedding distance averages nearest-neighbor distances after excluding the largest 10%.Top-k matching repeats selection while removing each already selected candidate.Source: §A.6 · p. 19

A.7 Additional Real-to-Sim Details

  • Implementation details include synthetic depth with DBSCAN denoising, optional point-cloud orientation refinement, and restricting articulated-object matches to articulated assets.For reported real-to-sim results, cousin selection also uses a door/drawer-count threshold of 2; simulated scene generation takes less than 30 seconds per scene.Source: §A.7 · p. 20

B.1 Visual Encoder Ablation Study

The ablation compares visual encoders and GPT-assisted selection for choosing digital cousins and orientations. DINO-based cousin selection performs better than CLIP for policy success, while GPT selection and orientation offer different reconstruction and runtime trade-offs.

B.1 Visual Encoder Ablation Study

  • Pipeline (d) gives the best qualitative scene reconstruction, while pipeline (c) balances inference time and reconstruction quality.The ablation compares combinations of DINO or GPT for model selection and orientation selection.Source: §B.1 · p. 20, Figure 6 · p. 22
  • Approximately 90% success on the exact twin for DINO and DINO+GPT, versus below 80% for CLIP-based selection; DINO+GPT also generalizes strongly to the second unseen cousin.Twin policies achieve higher success on the sixth unseen cousin selected by DINO+GPT than on the sixth selected by DINO.Source: Figure 7 · p. 23, §B.1 · p. 24
  • GPT model selection outperforms pure DINOv2 embedding-distance selection on category and model matching accuracy.The authors attribute this likely advantage to GPT focusing better on geometry despite lighting, occlusion, scale, and orientation variations affecting DINO.Source: §B.1 · p. 23
  • GPT and DINO orientation selection are generally comparable on orientation difference and IoU-related metrics, but GPT takes less than 10 seconds per object versus about 60 seconds for DINO.The authors recommend GPT orientation selection because orientations are randomized during policy training.Source: §B.1 · p. 23
  • For the first seven test scenes, mean L2 placement errors are below 10 cm, and the eighth scene has a 17 cm error at a scale of 10.23 m.Orientation-difference and IoU metrics are minimal between cousin-based and twin-based reconstructions.Source: §B.1 · p. 24
  • DINO and DINO+GPT cousins show more consistent geometry and handle design than CLIP-selected cousins; DINO+GPT selections have the least geometric variance.The DINO+GPT cousins feature two or four symmetrically arranged doors with handles similar to the twin.Source: Figure 8 · p. 23, §B.1 · p. 24

B.2 Real-to-Sim Scene Generation: Additional Results

Additional results show that ACDC can generate real-to-sim digital cousin scenes from a single RGB image without ground-truth camera intrinsics. The pipeline estimates camera intrinsics from the image, a capability the authors suggest could support future demonstration collection from in-the-wild web images.

B.2 Real-to-Sim Scene Generation: Additional Results

  • Additional qualitative results of real-to-sim digital cousin creation and scene generation are presented in Fig. 9.The authors refer readers to the accompanying video and website for multi-view visualizations.Source: §B.2 · p. 24
  • ACDC can generate real-to-sim digital cousin scenes from a single RGB image without ground-truth camera intrinsics.The pipeline estimates the camera intrinsic matrix from the input image using a Paramnet-360Cities-edina-uncentered model from PerceptiveFields.Source: §B.2 · p. 24
  • The authors suggest that this capability may enable large-scale demonstration collection using in-the-wild web images that lack ground-truth camera intrinsics.Source: §B.2 · p. 24

B.3 Failure Cases

ACDC failures arise from limitations in depth and object matching, scarce candidate assets, and its restricted model of object relationships. Updated foundation models improve some reconstruction details.

B.3 Failure Cases

  • Depth inaccuracies can degrade object-model estimates, while occlusion can distort bounding boxes, especially for smooth objects such as balls or plushes.Depth discontinuities near object boundaries and fine boundaries such as plants and fences compound estimation errors.Source: §B.3 · pp. 25–26
  • Category-label mismatches can cause CLIP to select suboptimal asset candidates when dataset naming or category abstraction differs from input labels.Examples include esoteric names and mismatches such as “cup” versus “coffee cup.”Source: §B.3 · p. 26
  • Replacing foundation models improves some reconstructions: DepthAnything-v2 captures finer plant details, SAM-v2 improves masks, and GPT-4o yields smaller orientation differences and higher bounding-box IoU.Source: §B.3 · p. 27
  • Too few candidate assets in a category can yield poor cousin matches and inaccurate orientation estimates.BEHAVIOR-1K has only one pot, one toaster, and two coffee maker assets.Source: §B.3 · p. 27
  • ACDC models only “on top” object relationships, though placing an object on a bounding box and moving it downward can approximate some inside placements.The passage reports reasonable reconstruction for a cushion placed inside a sofa this way.Source: §B.3 · p. 27

B.4 Comparison with URDFormer

In a qualitative comparison across five real-world kitchen scenes, ACDC matches or can outperform URDFormer in spatial reconstruction while offering object-agnostic, fully automated scene generation. URDFormer can produce textures resembling the real scenes, whereas ACDC leaves matched asset textures unchanged.

B.4 Comparison with URDFormer

  • The qualitative comparison evaluates ACDC and URDFormer on five real-world kitchen scenes, including URDFormer’s highlighted scene and three additional scenes.Source: §B.4 · p. 27
  • URDFormer focuses on trained object categories and relies on accurate bounding boxes that often require manual annotation; ACDC is object-agnostic and fully automated.Source: §B.4 · p. 27
  • URDFormer generates synthetic textures resembling the input, while ACDC does not modify matched assets’ textures.Source: §B.4 · p. 27
  • ACDC can match or outperform URDFormer in spatially reconstructing scenes, while also handling a more diverse set of object categories without manual annotation.Source: §B.4 · p. 27

B.5 Policy Training Details

Policies are trained from demonstrations using end-effector commands and point-cloud observations. The section specifies point-cloud construction, policy architecture, and evaluation procedure.

B.5 Policy Training Details

  • The action space uses six-dimensional delta end-effector position and axis-angle commands executed through inverse kinematics, while observations combine proprioception and a unified point cloud.Source: §B.5 · p. 27
  • Point clouds combine depth images in the robot frame, mask non-task objects, and add gripper points placed using proprioception and forward kinematics.Source: §B.5 · p. 27
  • The point cloud adds a binary scene-versus-gripper label and is downsampled; Putting Away Bowl additionally uses a second over-the-shoulder camera and a wrist camera for heavier occlusion.Source: §B.5 · p. 28
  • Policies use behavioral cloning with an RNN and GMM, encoding point clouds with a two-layer, 512-dimension PointNet and augmenting observations with downsampling, translation, and noise jitter.Source: §B.5 · p. 28
  • Each selected checkpoint is evaluated 100 times, with results aggregated across multiple runs.Source: §B.5 · p. 28

B.6 Sim-to-Sim Policy Learning with Digital Cousins

The experiments compare cousin-trained policies with twin and randomization baselines, reporting improved robustness on unseen assets and greater training stability. A small candidate pool can make four-cousin selection suboptimal for drawer opening.

B.6 Sim-to-Sim Policy Learning with Digital Cousins

  • For Figures 4 and 13, the authors selected two hyperparameter combinations using success on the original twin, then evaluated six policies per training set across three seeds.Putting Away Bowl's third task was evaluated on five runs due to resource constraints.Source: §B.6 · p. 28
  • Four-cousin policies performed sub-optimally for drawer opening, while eight-cousin policies outperformed all-assets and twin policies in that limited category.The authors attribute the four-cousin result partly to a limited cabinet pool and a gap in DINO distances; for door opening, more than 40 selectable assets supported a more continuous similarity distribution.Source: Table 4 · p. 28, §B.6 · p. 29, Figure 13 · p. 32
  • Eight-cousin policies were the most stable across random seeds, compared with twin policies and all-assets policies.The comparison uses standard deviations of success rates across Door Opening, Drawer Opening, and Putting Away Bowl evaluations.Source: §B.6 · p. 29, Figure 13 · p. 32
  • Naive domain randomization was insufficient for unseen cabinets, while twin-plus-cousin or twin-plus-all-assets policies performed similarly to policies trained exclusively on cousins.Figure 14 compares door-opening success rates across six test assets; Table 5 reports success rates for policies tested on assets unseen during training, with the stated exception for Twin + All Assets (More Rand.).Source: Table 5 · p. 28, §B.6 · p. 29, Figure 14 · p. 32
Loading 2410.07408v3…