IGen: Scalable Data Generation for Robot Learning from Open-World Images

Motivation

• The framework addresses the high cost and environment-specific limits of real-robot data collection, which constrain visuomotor policy generalization.Open-world images are inexpensive and diverse but lack executable robot actions required for policy learning.Source: §1 · p. 2

Related Work

• Unlike prior reconstruction-based or video-generation approaches, IGen targets arbitrary open-world images while providing explicit actions for long-horizon, instruction-driven tasks.Earlier methods require scene-specific workspace acquisition or struggle to produce explicit actions and complex task sequences.Source: §1 · p. 2

Method

• Vision-language models transform task instructions into high-level plans and low-level control functions, while SE(3) end-effector trajectories drive scene interaction synthesis.Frame-wise rendering of the resulting point-cloud sequence produces action-consistent visual observations.Source: §1 · p. 2

Results

• Policies trained solely on IGen-generated data can outperform policies trained on real-world trajectories, supporting open-world images as a source for robot policy training.The evaluation covers visual fidelity, action quality, and transfer to real-world visuomotor tasks.Source: §1 · p. 2

Contribution

• IGen generates scalable visual-action datasets from open-world images, integrating cross-scene generalization, instruction diversity, and long-horizon task applicability.The framework is designed to produce robot behaviors and training data at scale without human annotation.Source: §1 · p. 2

Method

• IGen converts unstructured open-world images into actionable 3D scene representations for robotic task reasoning and motion planning.Large vision models reconstruct 3D point clouds and spatial keypoints from the input images.Source: §1 · p. 2

2.1. Visuomotor Robot Learning

• Visuomotor learning depends on large paired visual-action datasets, but real-robot collection is labor-intensive and tied to specific environments.This limits the diversity and generalization of learned policies.Source: §2.1 · p. 3

2.2. Real-Sim-Real Data Generation

• Real-to-sim methods reconstruct executable workspaces from visual observations, yet often require task-specific acquisition and struggle with fine-grained scene realism.IGen instead uses open-world images as its primary source and offers a lightweight alternative without explicit physics-based reconstruction.Source: §2.2 · p. 3

2.3. Experience Synthesis from Unstructured Data

• Prior internet-data and video-generation approaches provide scalable visual sources but often lack grounded robot actions, complex-task capability, or computational efficiency.IGen targets grounded robotic experiences directly from abundant still images.Source: §2.3 · p. 3

Methodology

• IGen addresses the shortage of scalable open-world robot demonstrations by converting images into task-specific actions and visual observations.Its three stages are scene reconstruction, action planning, and observation synthesis.Source: §3 · p. 3, §2.1 · p. 3, §2.3 · p. 3

3.1. From Pixels to Structured 3D Representations

• IGen reconstructs open-world scenes as 3D point clouds and spatial keypoints using depth estimation, object segmentation, feature extraction, and 3D object completion.It also restores object poses and samples feasible placement points for spatial randomization.Source: §3.1 · pp. 3–4

3.2. Spatial Planning for Behavior Generation

• A VLM decomposes instructions into sub-stages and translates keypoint-conditioned plans into executable SE(3) end-effector controls.Grasping priors determine pre-manipulation poses and gripper state, while manipulated keypoints follow the end-effector during motion.Source: §3.2 · pp. 4–5

3.3. Experience Synthesis for Robot Learning

• IGen synthesizes synchronized observations by rendering robot and scene point-cloud sequences from simulated end-effector trajectories.The 6-DoF pose trajectory produces RGB-depth frames; grasped objects undergo rigid transformations, while the background remains static.Source: §3.3 · pp. 5–6
• Visual similarity is compared using three LPIPS variants, while generated robotic behavior is assessed for Instruction Following and Physics Alignment on DreamGen Bench.The LPIPS variants use AlexNet, VGGNet, and SqueezeNet. Each method generates 40 videos, and the reported metric is the proportion receiving a score of 1 from GPT-4o, Qwen-3-VL-Plus, or GLM-4.5V.Source: Table 1 · p. 6, Figure 4 · p. 6

4.1. Scene Reconstruction Fidelity

• 5.13× improvement in LPIPS similarity over the Real-to-Sim baseline shows that IGen reconstructs scenes with high visual consistency.PSNR, SSIM, and LPIPS are used to compare reconstructed scenes with original real-world images.Source: §4.1 · p. 7

4.2. Evaluation of Behavior Generation

• IGen generates complete, instruction-following robotic behaviors with physically consistent motions, while Cosmos-Predict2 and TesserAct exhibit discontinuities and inaccurate execution.The evaluation uses unlabeled open-world images paired with scene-specific task instructions.Source: §4.2 · p. 7
• 30× and 200× higher generation efficiency than TesserAct and Cosmos-Predict2, respectively, is achieved with 8.3 GB of GPU memory and approximately 18.6 seconds per sample.The comparison uses per-sample generation time and GPU memory under the same GPU memory conditions.Source: §4.2 · p. 8

4.3. Robot Learning from Unstructured Images

• On Place Bottle, IGen-generated data raises success from 0.0% to 75.0%, while 100 and 1,000 synthetic samples achieve 44.5% and 66.7%, surpassing matched teleoperation results.The study compares zero-shot, teleoperation fine-tuning, and IGen fine-tuning across placement and interaction tasks.Source: §4.3 · p. 8

• IGen automatically transforms open-world images into high-quality visuomotor data at scale, without human effort, to improve robot policy performance.The conclusion frames this capability as a potential response to limited real-world robot data.Source: §5 · p. 8

• Figure 8 summarizes the single-image scene reconstruction pipeline used to facilitate robot data generation.Source: §6 · p. 13, Figure 8 · p. 13
• Metric3Dv2 converts the RGB image into a point cloud using estimated depth, with focal length fixed to 1000 for open-world images.Known camera intrinsics are used instead for specific cameras such as iPhone or Microsoft Kinect.Source: §6 · p. 13
• TRELLIS reconstructs target objects as colored point clouds after SAM segmentation masks focus reconstruction on the relevant objects.This provides object-level reconstruction within the scene representation.Source: §6 · p. 13

• Isaac Sim contains Franka Panda and Franka Research 3 arms, while Curobo computes feasible trajectories from target end-effector poses.The simulation uses default illumination settings.Source: §7 · p. 13
• The virtual depth camera is placed at [0, 0, 0], the robot at [xr, yr, zr], and synchronized RGB-depth frames are captured at 30 fps during motion.The camera focal length matches the depth-estimation module for subsequent dynamic point-cloud reconstruction.Source: §7 · p. 13, Figure 9 · p. 13

Manipulation Synthesis Details

• IGen separates the scene into static background, dynamic robot, and dynamic object point clouds, using GraspGen for grasp pose estimation.Source: §8 · p. 13
• The grasp width is inferred from inter-point distances along the reconstructed object point cloud’s principal axis.Source: §8 · p. 13
• During manipulation, the object point cloud is transformed according to the end-effector pose after grasping.Source: Figure 10 · p. 13, §8 · p. 13

Real-World Experiment Details

• The evaluation uses a Franka Research 3 arm, a front-facing Microsoft Kinect RGB camera, and a tabletop workspace.Source: §9 · p. 14, Figure 12 · p. 14

Spatial Randomization

• Real-world object positions are randomized within a 40 cm × 30 cm tabletop grid, while IGen samples a corresponding 200 × 150 pixel placement region.Source: §9 · p. 14

Task Evaluation

• Each task uses 12 independent trials with initial positions sampled on a 30 cm × 25 cm grid and 7 cm spacing, shared across models.Source: §9 · p. 14

Policy Learning

• Policies are fine-tuned for 30k steps with LoRA, batch size 8, and a 10-step relative joint-angle action chunk from 224 × 224 RGB input and joint-position state.Training takes approximately 10.8 hours per model on one NVIDIA A40 GPU.Source: §9 · p. 14

Real-World Deployment

• Real-world deployment evaluates IGen-trained policies on watering, placement, hammering, grasping, and pouring tasks.Source: Figure 15 · p. 16, Figure 16 · p. 17

Ablation Study

• The ablation study compares different components of the IGen pipeline.Source: Figure 17 · p. 18

Details for Keypoints Generation

• The VLM input includes an image annotated with keypoints, their spatial coordinates, and a task description.Source: Figure 18 · p. 18, §11 · p. 18