PaperlayerView exact revision on arXiv

Source-linked AI summary

URDFormer: A Pipeline for Constructing Articulated Simulation Environments from Real-World Images

Zoey Chen, Aaron Walsman, Marius Memmel, Kaichun Mo, Alex Fang, Karthikeya Vemuri, Alan Wu, Dieter Fox, Abhishek Gupta

arXiv:2405.11656v3cs.ROcs.AI

TL;DR

Simulation scene construction is labor-intensive, while scalable robotic learning calls for scenes that are realistic, diverse, and controllable. URDFormer learns to infer articulated simulation scenes from images using paired data generated with controllable image models. The paper reports better zero-shot real-world performance with predicted URDFs and targeted domain randomization, while current support is limited to simpler articulated joint types.

  • Problem

    Simulation content for downstream robotics and computer vision must combine visual realism, rich kinematic structure, and the natural statistics needed for realistic, diverse, and controllable environments.

    Source: §II · p. 2, §I · p. 2

  • Method

    URDFormer learns to map real-world images to simulation scenes by training on paired data built from procedurally generated scenes and controllably generated realistic images.

    Source: §I · p. 2

  • Results

    Predicted URDFs with targeted domain randomization enable better zero-shot performance in the real world.

    Source: §VII · p. 12

  • Takeaways & Limitations

    The pipeline provides a first step toward cheaper, scalable realistic scene generation for robot learning.

    Source: §VII · p. 12

    URDFormer supports only limited joint types such as prismatic and revolute, and cannot predict complex objects such as cars and lamps.

    Source: §VI · p. 12

Abstract

from arXiv · show

Constructing simulation scenes that are both visually and physically realistic is a problem of practical interest in domains ranging from robotics to computer vision. This problem has become even more relevant as researchers wielding large data-hungry learning methods seek new sources of training data for physical decision-making systems. However, building simulation models is often still done by hand. A graphic designer and a simulation engineer work with predefined assets to construct rich scenes with realistic dynamic and kinematic properties. While this may scale to small numbers of scenes, to achieve the generalization properties that are required for data-driven robotic control, we require a pipeline that is able to synthesize large numbers of realistic scenes, complete with 'natural' kinematic and dynamic structures. To attack this problem, we develop models for inferring structure and generating simulation scenes from natural images, allowing for scalable scene generation from web-scale datasets. To train these image-to-simulation models, we show how controllable text-to-image generative models can be used in generating paired training data that allows for modeling of the inverse problem, mapping from realistic images back to complete scene models. We show how this paradigm allows us to build large datasets of scenes in simulation with semantic and physical realism. We present an integrated end-to-end pipeline that generates simulation scenes complete with articulated kinematic and dynamic structures from real-world images and use these for training robotic control policies. We then robustly deploy in the real world for tasks like articulated object manipulation. In doing so, our work provides both a pipeline for large-scale generation of simulation environments and an integrated system for training robust robotic control policies in the resulting environments.

I. INTRODUCTION

Simulation offers scalable, inexpensive data collection, but scene construction must be realistic, diverse, and controllable for downstream learning. URDFormer addresses the lack of paired real-image and articulated-scene data by learning to infer simulation structure from images using generated training pairs.

I. INTRODUCTION

  • Simulation content must be realistic, diverse, and controllable, while existing manual, procedural, and generative approaches each fall short on scalability, natural complexity, or physical and kinematic structure.Manual construction is laborious, procedural methods can miss natural complexity and scene-specific control, and generative methods may lack articulation or suitable physical behavior.Source: §I · p. 2

II. URDFORMER - A PIPELINE FOR SCALABLE CONTENT CREATION FOR SIMULATION FROM REAL-WORLD IMAGES

  • URDFormer maps real-world images to plausible simulation scenes by training on paired scenes and realistic images generated with controllable text-to-image models.The resulting scenes can be generated from diverse uncurated images or targeted images and used in a real-to-sim-to-real robotic learning pipeline.Source: §I · p. 2

A. Problem Formulation

  • The inverse task is to recover a scene’s semantic and kinematic structure from an image, but such structure is unknown and paired scene-description data is not readily available.The paper uses an inversion-through-synthesis approach to create paired training data and splits prediction into object-level and global-scene structure.Source: A. Problem Formulation · p. 3

B. Controlled Generation of Paired Datasets with Pretrained Generative Models

The pipeline creates separate paired datasets for global scenes and articulated objects, then trains inverse models to recover their structures from images. URDFormer combines scene-level placement and parent prediction with object-part structure prediction to produce full scene descriptions.

B. Controlled Generation of Paired Datasets with Pretrained Generative Models

  • Scene-level generation uses guided diffusion to retain broad layout, but generated images can alter object categories and parts, leaving high-level labels incomplete.The resulting scene dataset has images with high-level boxes, transforms, and parent relationships rather than accurate low-level structure.Source: B. Controlled Generation of Paired… · p. 3, Figure 3 · p. 4
  • Object-level data uses part-by-part texture generation and background replacement to preserve articulated details while varying appearance, pairing single-object images with complete part labels.This contrasts with the scene dataset’s complete images and partial labels.Source: B. Controlled Generation of Paired… · pp. 3–4

C. Learning Inverse Generative Models for Scene Synthesis

  • URDFormer uses a shared architecture for scene and part prediction, combining image features and object boxes to predict object properties and parent-child relationships.Scene prediction operates on full images and object boxes; part prediction operates on object crops and extracted part boxes.Source: C. Learning Inverse Generative Models… · pp. 4–5, Figure 4 · p. 4
  • At test time, scene prediction estimates object locations and parents, then part prediction infers each object’s kinematic structure to assemble a full scene from a real image.The nested process supports generating simulation content from web-scraped RGB images.Source: C. Learning Inverse Generative Models… · p. 5

III. USING URDFormer FOR ROBOTIC CONTROL VIA A REAL-TO-SIMULATION-TO-REAL PIPELINE

URDFormer supports robotic policy learning by generating a simulation environment from an observed scene and training in a targeted distribution of related environments. Policies trained from simulation trajectories and RGB point clouds can then transfer to the real world.

III. USING URDFormer FOR ROBOTIC CONTROL VIA A REAL-TO-SIMULATION-TO-REAL PIPELINE

  • Rather than relying on an imperfect one-to-one digital twin, the pipeline trains policies in a narrow, representative distribution of simulated environments around the deployment scene.This addresses simulation inaccuracy and the lack of access to Lagrangian state in the real world.Source: §III · p. 5
  • URDFormer converts an RGB point-cloud observation into a scene URDF, which is resized to the point-cloud scale and imported into simulation for data collection and training.The resulting simulation approximates the deployment scene rather than serving as an arbitrary model.Source: §III · p. 5
  • Targeted randomization varies minor details such as textures, shapes, and part sizes around the real-world environment, unlike uninformed procedural variation.An efficient motion planner collects approximately optimal task trajectories in simulation.Source: §III · p. 5
  • Language-conditioned behavior cloning trains policies from simulated RGB point clouds, with image augmentations supporting transfer to real-world control.The pipeline can also use policy-search methods beyond motion-planning data and behavior cloning.Source: §III · p. 5

IV. EXPERIMENTS

The experiments evaluate a URDFormer-based real-to-sim-to-real pipeline for cabinet manipulation and generated simulation content from internet images. The robot policy transfers zero-shot, and targeted randomization outperforms un-targeted randomization in the reported comparison.

improve policies that can transfer zero-shot to the real world?

  • The Fig. 5 workflow turns cabinet images into depth-scaled, textured simulations for policy training, while red boxes mark some inaccurate URDFormer predictions.The caption describes bounding-box detection, URDF prediction, texture projection, and simulation data generation before zero-shot transfer.Source: Figure 5 · p. 6, improve policies that can transfer… · p. 8
  • The real-world evaluation used a UR5 robot and five cabinets varying in size, shape, texture, joint type, and handles, with two tasks per cabinet.The pipeline detects cabinet parts, predicts a URDF, scales it using depth, trains from planner-generated trajectories with targeted randomization, and transfers a language-conditioned policy.Source: improve policies that can transfer… · pp. 6–7
  • 78% success across all tasks and 85% on opening/closing tasks were reported for the real-to-sim-to-real pipeline using URDFormer and targeted randomization.Table I reports success divided by number of trials.Source: Table I · p. 7, improve policies that can transfer… · p. 7
  • 0% success on all tasks resulted when OWL-ViT failed to localize fine-grained regions such as specific drawers, doors, and handles.Without those regions of interest, motion planning could not solve the tasks.Source: improve policies that can transfer… · p. 7
  • 40% average improvement over domain randomization was reported for URDFormer-TR across the evaluated cabinet variations and tasks.Domain randomization worked on a simple single-drawer cabinet but failed on more complicated configurations.Source: improve policies that can transfer… · p. 7
  • 50% success on putting an object in the bottom drawer was achieved by URDFormer-TR, while ICP-based pose transfer failed when it matched the cabinet instead of the smaller object.Neither baseline reliably solved putting or retrieving objects in drawers.Source: improve policies that can transfer… · p. 7

IV. EXPERIMENTS

  • Reality Gym depicts internet-image-derived assets randomized with PartNet meshes across opening, closing, fetching, and collecting tasks.Fig. 7 also shows kitchen scenes predicted from internet images with labeled bounding boxes, including highlighted failure parts.Source: Figure 6 · p. 8, Figure 7 · p. 8

B. Can URDFormer generate plausible and accurate simula-

The experiments describe generation of paired image–URDF training data and URDFormer’s image-and-part-box inputs. The resulting dataset covers both articulated objects and global kitchen scenes.

1) Paired Training Dataset Generation:

  • Paired data generation combines procedural scene URDFs with realistic images, using texture templates and in-painting to address local-part inconsistencies from depth-guided Stable Diffusion.The pipeline warps selected texture templates onto part regions before smoothing boundaries and generating background content.Source: 1) Paired Training Dataset Generation · p. 8
  • Approximately 118K image–URDF pairs were generated across seven categories of articulated objects, alongside approximately 200K pairs for global kitchen scenes.These pairs were produced from procedurally generated URDF scenes and corresponding realistic RGB images.Source: 1) Paired Training Dataset Generation · p. 8

B. Can URDFormer generate plausible and accurate simula-

2) Object and Part Detection during Inverse Phase:

The inverse phase combines object and part detection with URDFormer scene prediction, evaluated on labeled object and kitchen sets. Results cover detector performance, qualitative reconstructions, texture ablations, and the RealityGym learning suite.

2) Object and Part Detection during Inverse Phase:

3) Real World Evaluation Datasets:

  • The object-level test set contains 300 labeled images across five categories, while the kitchen set contains 54 scenes with 5–15 articulated objects each.Labels include bounding boxes, URDF primitives, parent IDs, positions, and scales; positions and scales are discretized into 12 bins.Source: 3) Real World Evaluation Datasets · p. 9

4) Evaluation Metrics:

  • Scene evaluation aligns predicted and ground-truth boxes by IoU and Hungarian matching, then measures category accuracy, parent accuracy, spatial error, precision, and recall.Spatial error is averaged over bounding-box coordinates for larger objects and centers for small parts.Source: 4) Evaluation Metrics · p. 9

5) Qualitative Results:

  • URDFormer largely reproduces kitchen structure and object configurations from internet images, though its reconstructions include mistakes.The qualitative comparisons include predictions using manually labeled boxes as well as detected boxes.Source: 5) Qualitative Results · p. 9

6) Ablation Study:

  • Realistic generated textures particularly help global object-type, parent, and placement predictions, while their effect on part-structure prediction is small and sometimes 1%–2% worse than random textures.Detected boxes average worse than ground-truth boxes due to detection error, although detected boxes slightly help global object-type identification.Source: 6) Ablation Study · p. 9

7) Reality Gym:

  • RealityGym provides image-derived articulated assets and scenes for robot learning, with automatically generated tasks and language descriptions completed using a motion planner.Its initial collection includes 300 objects and 50 kitchen scenes, alongside additional cabinet-part meshes.Source: 7) Reality Gym · p. 9

C. Can URDFormer generalize to diverse objects and scenes?

The paper evaluates URDFormer on added object and scene categories, including unseen rooms, and demonstrates a Stretch-robot task. It also frames single-image URDF reconstruction as an alternative to asset pipelines requiring richer sensing or interaction.

C. Can URDFormer generalize to diverse objects and scenes?

  • URDFormer is qualitatively evaluated on five added object categories and four scene categories, including laundry rooms and study rooms not used for global-scene training.The added part model receives approximately 6k examples per object category; the global model receives approximately 10k examples for bathroom and bedroom scenes.Source: C. Can URDFormer generalize to diverse… · pp. 9–10, Figure 12 · p. 11

D. Can URDFormer support different robots and tasks?

A. Asset Creation for Robot Manipulation

  • The paper positions single-image URDF reconstruction as a faster, cheaper alternative to approaches requiring video, depth sensing, 3D scans, or object interaction.Prior learned scene reconstruction can offer visual realism but does not usually produce articulated models for dynamic interaction.Source: A. Asset Creation for Robot Manipulation · pp. 11–12

B. Synthetic Data Augmentation with Generative Models

The paper uses generative models to create realistic synthetic textures while identifying limitations in detection, mesh assumptions, supported joints, and physical-property inference. Its conclusion presents the pipeline and RealityGym as steps toward scalable image-derived simulation for robot learning.

B. Synthetic Data Augmentation with Generative Models

  • Controllable generative models are used to augment synthetic renderings with realistic textures, while the proposed method aims to preserve part-level structure details.The authors describe existing controllable image-generation and mesh-texturing methods as lacking in part-level structural consistency.Source: B. Synthetic Data Augmentation with… · p. 12

VI. LIMITATIONS AND FUTURE WORK

  • URDFormer depends on bounding-box detection, and fine-tuned Grounding DINO still has room to improve, especially for global-scene detection.The pipeline is not trained end-to-end and comprises multiple learning components.Source: §VI · p. 12
  • URDFormer predicts kinematic structures using predefined meshes that may not match real scenes, and its rectangular-part texture mapping fails on irregular meshes or tilted objects.The paper specifically notes donut-shaped doors as an irregular-mesh example.Source: §VI · p. 12
  • The system supports only limited joint types, such as prismatic and revolute, and cannot predict complex objects such as cars and lamps.Source: §VI · p. 12
  • URDFormer predicts URDF primitives for bounding boxes, which can cause collisions between links and require further post-processing.Source: §VI · p. 12

VII. CONCLUSION

  • The system does not infer physical properties such as mass, inertial moments, or friction from scene observations.The conclusion presents the pipeline as a first step toward cheaper, scalable realistic scene generation for robot learning.Source: §VI · p. 12, §VII · p. 12

APPENDIX A PAIRED DATASET GENERATION FOR TRAINING URDFormer

The appendix describes a texture-guided image-generation process intended to retain low-level part consistency while diversifying appearance. It compares generation methods and training textures, then reports the scale of the paired dataset.

A. Consistency-aware Image Generation

B. Part-Consistency

  • Compared qualitatively with depth-guided diffusion and inpainting, part-wise generation is reported to preserve semantic low-level details while producing photorealistic images paired with textureless simulation data.Source: B. Part-Consistency · p. 13, Figure 17 · p. 13

C. Visualization on Ablations

  • Generated textures appear closer to real-world pixel distributions and improve prediction of mesh classes such as ovens and cabinets.The visual ablation compares random colors, selected textures, random textures, and generated textures.Source: C. Visualization on Ablations · p. 13, Figure 18 · p. 13

A. Dataset

  • The dataset contains approximately 118K image–URDF pairs spanning five common articulated kitchen-object categories and two rigid-object types.The articulated objects use eight part-mesh types, including drawers, doors, handles, and knobs.Source: A. Dataset · p. 13

B. Training Details

URDFormer is trained on generated data and applied to internet images to produce articulated simulation assets. Mesh and texture randomization are used to broaden asset appearance.

B. Training Details

  • Global scenes are trained for 130 epochs and objects for 80 epochs, using one A40 GPU, batch size 256, and the last checkpoint for ablations.The ablation methods use equal numbers of epochs.Source: B. Training Details · pp. 13–14

A. Realistic Assets from Images

  • Internet images are processed with finetuned GroundingDINO for part boxes, then URDFormer predicts matching articulated URDF assets.The pipeline also generates demonstrations for opening or closing parts, fetching objects, and collecting objects.Source: Figure 19 · p. 14, A. Realistic Assets from Images · p. 14

B. Targeted Domain Randomization

APPENDIX D REAL-WORLD EXPERIMENTS

The appendix describes simulation data collection, tests OWL-ViT part detection, and outlines policies trained for UR5 and Stretch tasks. OWL-ViT misses some spatially specified parts.

A. Data Collection in Sim

  • Each object receives two tasks, with 500 simulation demonstrations per task collected across different object poses.Tasks include opening or closing articulated parts, fetching an object from a drawer, and collecting an object into a drawer; training also uses pose and color augmentation.Source: A. Data Collection in Sim · p. 15

B. Visualization of OWL-VIT

  • OWL-ViT fails to detect useful parts for language instructions, including spatially specified regions such as “top drawer” and “opened door.”The visualizations show that it cannot accurately detect specific articulated parts or regions.Source: B. Visualization of OWL-VIT · p. 15, Figure 24 · p. 15

C. Policy Training

  • The UR5 object policy uses RGB point clouds and CLIP embeddings to predict grasp and release end-effector poses for opening a door.Its architecture is adapted from M2T2 and predicts the next end-effector pose.Source: C. Policy Training · p. 15
  • The Stretch visual policy uses task embeddings with image features to generate affordance maps for a simulated table-cleanup task.The task sequence is opening a drawer, picking up an object, and putting it in the drawer.Source: C. Policy Training · p. 15

APPENDIX E LIMITATIONS AND FUTURE WORK

The appendix identifies limits in detection, geometry, supported structures, and scene coverage, while showing examples across trained object and scene categories.

APPENDIX E LIMITATIONS AND FUTURE WORK

  • URDFormer depends on bounding-box detection, which still has room to improve, particularly for global scenes.The authors propose removing bounding-box constraints and directly predicting URDFs in language format as future work.Source: Appendix E · p. 15
  • Predefined meshes may not match real scenes, and rectangular-part texture mapping fails for irregular shapes or tilted objects.The texture process crops each part using its bounding box and maps it into a UV template.Source: Appendix E · p. 15
  • The current system supports only limited articulated joint types, such as prismatic and revolute, and cannot predict complex objects such as cars and lamps.Source: Appendix E · p. 15
  • URDF primitives predicted for bounding boxes can cause link collisions, requiring further post-processing.Source: Appendix E · p. 15
  • High-quality part detection remains challenging for global scenes, and the authors identify other room types as a future expansion beyond kitchens.Source: Appendix E · p. 15
  • The system predicts kinematic structure for a few common articulated objects but does not predict poses for items such as counter-top water bottles.Source: Appendix E · p. 15
  • Multiple independently optimized components complicate the pipeline, though the authors say this lets it leverage pretrained models and maintain consistency in generative data construction.Source: Appendix E · pp. 15–16

APPENDIX F GENERALIZATION ON DIVERSE OBJECTS AND SCENES

  • URDFormer is trained for diverse object and scene categories, with one part model across object categories and one global model across scene categories.The appendix also visualizes examples of data generation beyond kitchen objects and scenes.Source: Figure 25 · p. 16, Appendix F · p. 16, Figure 26 · p. 17
Loading 2405.11656v3…