PaperlayerView exact revision on arXiv

Source-linked AI summary

IM-ENGINE: Image Editing for Embodied Data Generation

Yian Wang, Junyi Cao, Xiaowen Qiu, Chuang Gan

arXiv:2609.06279v1cs.ROcs.CV

TL;DR

Learning-based manipulation needs supervision that is both semantically meaningful and physically executable, while human demonstrations are costly and simulation can miss functional intent. IM-ENGINE edits simulator-rendered scenes, recovers and validates explicit 3D states, and produces robot-usable grasp and goal data. It reports 35/40 goal-generation successes against 11/40 for PhyScensis and the best overall grasp lifting and functional success among evaluated methods.

  • Problem

    Human demonstrations are costly and constrained by embodiment transfer, while simulation can scale data generation but often under-specifies functional behavior.

    Source: §1 · p. 2

  • Method

    IM-ENGINE edits simulator-rendered scenes to propose functional grasps or desired object relations, recovers explicit 3D states using simulator priors, validates them in physics, and generates robot supervision or trajectories.

    Source: §1 · p. 2, §3 · p. 3, §3.2 · p. 4

  • Results

    IM-ENGINE achieves 35/40 goal-generation successes versus PhyScensis (11/40), and DRO w/ FT (Ours) achieves the best overall lifting (99.4%) and functional success (83.2%).

    Source: §4.1 · p. 6, §4.2 · p. 7

  • Takeaways & Limitations

    The same simulator-grounded editing pipeline supports functional dexterous grasp synthesis and relation-heavy goal-state generation, including hanging and fitting tasks.

    Source: §1 · p. 2

    The pipeline assumes suitable paired 3D assets for goal generation, and grasp validation can occasionally favor stable but semantically incorrect contacts.

    Source: §5 · p. 8

Abstract

from arXiv · show

Learning-based manipulation requires supervision that is both semantically meaningful and physically executable, but current data pipelines often provide only one of these properties. Human demonstrations capture intent but are costly to collect and constrained by the human-robot embodiment gap, while simulation can scale data generation but often under-specifies functional behavior. We present IM-ENGINE, a simulator-grounded pipeline that uses image editing as an intermediate representation for embodied data generation. Given a rendered scene with known geometry, depth, segmentation, and camera parameters, IM-ENGINE edits the image to inject task-relevant semantics, recovers explicit 3D state using simulator priors and an unchanged anchor object, refines the state in physics, and converts it into robot-executable supervision. We instantiate the pipeline for dexterous grasp synthesis and goal-state generation. For grasping, IM-ENGINE generates a human grasp in image space, recovers the hand-object interaction, retargets it to a robot hand, and refines it into physically validated robot grasps. For goal generation, it edits a rendered scene into a desired outcome, recovers the target-object pose, and refines it into physically valid, semantically meaningful goals and trajectories. This combination of generative semantic priors and simulator grounding enables scalable task-relevant supervision for robot learning.

1 Introduction

Robot manipulation needs supervision that is both semantically meaningful and physically executable, but human data is costly and simulation can miss functional intent. IM-ENGINE uses image editing within a simulator-grounded pipeline to recover explicit 3D states and produce robot-usable supervision for grasping and goal generation.

1 Introduction

  • Human demonstrations capture task semantics but are burdensome to collect and require cross-embodiment alignment, while simulation scales data generation but can prioritize physical feasibility over functional intent.Existing approaches also leave a gap between semantic visual guidance and explicit 3D supervision for relation-heavy goal states.Source: §1 · p. 2
  • IM-ENGINE edits simulator-rendered scenes to express functional grasps or desired object relations, then grounds those edits in explicit 3D state and physics-validated robot supervision.The simulator provides geometry, calibration, visibility, segmentation, and depth to support metric recovery and verification.Source: §1 · p. 2
  • The framework combines simulator-grounded image editing, tailored 3D perception, physics validation, and trajectory generation for dexterous grasp synthesis and relation-heavy goal-state generation.The demonstrated goal tasks include hanging and fitting, which prior visual-goal or scene-generation pipelines do not directly address.Source: §1 · p. 2

2 Related Work

Prior work generates robot grasps at scale, gathers richer human grasp semantics, and uses generated imagery for manipulation guidance or scene construction. IM-ENGINE instead treats an edit as a semantic proposal grounded in a known simulator scene for metric pose recovery and execution.

Dexterous grasp data generation

  • Synthetic grasp pipelines produce robot-centric supervision that may miss functional intent, while human hand-object datasets offer richer semantics but are costly and do not scale directly to robot-executable data.Examples include DexGraspNet, DexGrasp Anything, DRO Grasp, and human HOI datasets.Source: §2 · pp. 2–3

Visual goal generation and spatially grounded placement

  • Visual-foresight methods use generated images or videos as policy guidance, while some image-generation pipelines recover simulation states, motions, or trajectories.Gen2Act, CoT-VLA, and ForeAct condition policies on imagined observations; LuciBot reconstructs object-pose trajectories, masks, and depth.Source: §2 · p. 3
  • IM-ENGINE grounds edited-image semantics in a simulator-rendered scene with known geometry, depth, and camera parameters to estimate metric target poses, refine them in physics, and execute them with a robot.SceneThesis recovers plausible 3D layouts but lacks a known simulator anchor for depth and scale alignment.Source: §2 · p. 3

3 Method

IM-ENGINE places image editing inside a simulator-grounded loop that turns task-specific visual proposals into explicit 3D states, validates them in physics, and converts them into robot supervision or motion. The same four stages support functional grasp synthesis and relation-heavy goal generation.

3 Method

  • The shared pipeline edits a rendered scene into a task proposal, recovers structured 3D state from simulator priors and an unchanged anchor, refines it in physics, and generates robot data.Grasp synthesis recovers a hand-object interaction; goal generation recovers a 6-DoF target pose relative to a fixed anchor.Source: §3 · p. 3, Figure 2 · p. 4

3.1 Image Generation

  • Image generation inserts a human hand to depict a proposed functional grasp or moves a target object into a requested relation while preserving the scene viewpoint and anchor pose.The initial render uses known meshes, poses, a calibrated camera, segmentation, and metric depth.Source: §3.1 · pp. 3–4

3.2 Perception

  • Perception segments edited entities, aligns predicted depth to simulator metric depth over the visible anchor, and estimates either a hand-object interaction or a target-object pose.The anchor mask comes from simulator supervision, and affine depth alignment corrects scale and offset mismatch before back-projection.Source: §3.2 · pp. 4–5

3.3 Physics Validation

  • Physics validation filters and refines grasp or goal proposals, rejecting collisions and retaining grasps or object poses that satisfy contact, lift, stability, and drift constraints.For goal generation, nearby poses are simulated with the anchor and environment fixed; unstable or penetrating edits are rejected.Source: §3.3 · p. 5

3.4 Trajectory Generation

  • Trajectory generation converts validated grasps into arm-hand rollouts and validated goal poses into insertion references using a reversed collision-free escape path to warm-start forward MPC.Grasp rollouts plan an approach, close the hand, and lift; a collision-aware cuRobo approach is used if default inverse kinematics would disturb the object before contact.Source: §3.4 · p. 5

4 Experimental Results

IM-ENGINE produces functional grasps and physically plausible goal states, then turns simulated supervision into real-world policies that preserve intended grasp modes or object-object relations. Across both tasks, its evaluations report successful transfer to physical scenes.

4.1 Grasp Synthesis

  • 99.4% lifting success and 83.2% functional success make DRO w/ FT (Ours) the best overall grasp-synthesis method.Pipeline wrist initialization raises functional success from 10.4% to 30.2%, and fine-tuning raises it to 83.2%.Source: Table 1 · p. 6, §4.1 · p. 6

4.2 Goal Generation

  • 35/40 successes for Ours versus 11/40 for PhyScensis across eight goal-generation tasks, with the largest gains on relation-heavy placements.A goal counts as successful only when physically plausible and satisfying the intended task relation.Source: §4.2 · pp. 6–7, Table 2 · p. 7

4.3.1 Grasp Synthesis

  • 80% top-down and 60% handle-grasp policy success, versus 0% for real-only training, across 10 trials with varied mug positions.Each policy used approximately 800 IM-ENGINE simulation episodes and 40 real demonstrations; both preserved their intended grasp modes across mug instances.Source: §4.3.1 · p. 7, Figure 5 · p. 8

4.3.2 Goal Generation

  • Hanger-hook and mug-mugtree policies achieved 70% and 50% success, respectively, across 10 trials with varying object instances and initial poses.Training used approximately 1,000 simulation trajectories per task, replayed with randomized backgrounds and object textures.Source: §4.3.2 · pp. 7–8, Figure 6 · p. 8

5 Conclusion and Limitations

IM-ENGINE grounds image edits in simulator state, then refines them into robot-usable grasps, goals, and trajectories. Its scope is constrained by image-editing errors, simulation challenges, asset requirements, and occasional semantically incorrect grasp validation.

5 Conclusion and Limitations

  • IM-ENGINE grounds image edits in known geometry, segmentation, depth, and physics to produce validated grasps, stable 6-DoF placements, and executable trajectories.The framework is instantiated for functional dexterous grasp synthesis and goal-state generation.Source: §5 · p. 8
  • Edited images may specify infeasible interactions, alter anchor objects, or miss functional contacts, while grasp validation can favor stable but semantically incorrect contacts.These limitations apply to the image-editing and grasp-validation stages.Source: §5 · p. 8
  • Collision handling and contact-rich simulation remain challenging for thin structures and constrained insertion or hanging tasks, and goal generation assumes suitable paired 3D assets.Source: §5 · p. 8

A.1 HOI Reconstruction Evaluation

The evaluation compares simulator-grounded hand-object reconstruction with methods that lack simulator priors, using geometric metrics and qualitative functional-contact checks. The reconstruction reduces penetration while preserving intended grasp modes, though aggregate scores alone do not capture functional fidelity.

A.1 HOI Reconstruction Evaluation

  • The evaluation compares against iHOI and adapted EasyHOI reconstruction pipelines, with OakInk-Shape serving as an approximate ground-truth scale rather than a reconstruction baseline.Source: §A.1 · p. 14
  • Our reconstruction substantially reduces penetration versus iHOI checkpoints and adapted EasyHOI, while its contact region ratio is closer to OakInk-Shape with much lower penetration.Lower is better for PDMmax, PV, and DD; higher is better for CR.Source: §A.1 · p. 14, Table 3 · p. 14
  • The adapted EasyHOI baseline can match aggregate scores yet miss edited functional intent, whereas our reconstruction more closely preserves task-relevant contacts.Examples include fingers missing the trigger sprayer trigger or failing to grip a cup handle precisely.Source: §A.1 · p. 15, Figure 7 · p. 15
  • Direct retargeting lifts objects successfully at 57%, 98%, 94%, 98%, and 81% for paper cups, power drills, spray bottles, trigger sprayers, and wine glasses, respectively.This test is performed before physics validation.Source: §A.1 · p. 15

A.2 Method Details

The method turns edited grasp images into robot-hand states through simulator-frame reconstruction, physics validation, and trajectory generation. It also supports coordinated two-hand grasps and evaluates functionality using category-specific contact criteria.

A.2 Method Details

  • The renderer fixes a 3/4 elevated front view, fits each object into the image, and records camera parameters for downstream reconstruction; grasp prompts preserve the viewpoint and scene while specifying functional hand placement.Source: §A.2 · pp. 15–16
  • HaMeR initializes a MANO hand from the edited image, which is aligned to the simulator world using visible hand points and object-surface constraints before its joints are retargeted to robot kinematics.The refinement includes visible-surface alignment, penetration discouragement, and attraction of near-contact vertices to the object surface.Source: §A.2 · p. 16
  • Physics validation searches parallel two-frame dexterous actions, rewarding collision-free pre-grasps, successful lifts, perception fidelity, and fingertip contact while enforcing collision, lift, and contact constraints.A lift must raise the object at least 0.15 m, and the contact constraint requires thumb, index, and middle fingertips to touch within the specified maximum distance.Source: §A.2 · pp. 16–17
  • Validated hand states become supervision directly or full xArm7 rollouts, with a collision-aware cuRobo approach used if inverse kinematics would disturb the object before grasping.Source: §A.2 · p. 17

A.3 Dual-Hand Grasping

  • For dual-hand tasks, the prompt can specify two hands and validation jointly checks their poses, contacts, and shared-object lift stability; the example grasps both handles of a pot.Source: §A.3 · p. 17, Figure 8 · p. 18

A.4 Functionality Criteria

  • A functional grasp must pass a simulated lift and satisfy category-specific criteria, including appropriate cup coverage, trigger or pump contact, and stable stem or lower-bowl contact for a wine glass.Source: §A.4 · p. 18

A.5 Visual Results

Qualitative results show that fine-tuned DRO grasps more consistently target functional regions than methods that prioritize stable lifting. Intermediate visualizations also show the pipeline across varied object geometries.

A.5 Visual Results

  • DexGraspNet and DexGrasp-Anything often stabilize objects without contacting task-relevant regions, while language-conditioned DexGYSGrasp can remain imprecise at the fingertips.Source: §A.5 · p. 18
  • DRO fine-tuned with IM-ENGINE data more consistently places grasps on intended functional regions across five categories than baselines that often make stable but generic contacts.Examples include drill handles, spray triggers, and glass stems.Source: §A.5 · p. 18, Figure 9 · p. 19
  • Pipeline examples span diverse object geometries, supporting generalization without category-specific shape assumptions.The visual sequence includes edited images, estimated MANO poses, retargeted robot-hand poses, and physics-refined grasps.Source: §A.5 · pp. 18–19

A.6 Failure Cases

Functional grasp failures most often involve stable grasps that miss intended contacts, while infeasible edits, anchor changes, and table penetrations occur less often. Reported functional proportions vary across objects.

A.6 Failure Cases

  • Non-functional grasps are the most common failure: loose contact constraints can yield stable grasps that miss intended functional contacts.The reported functional proportions are 95% for paper cup, 77% for power drill, 71% for spray bottle, 90% for trigger sprayer, and 98% for wine glass.Source: §A.6 · p. 19
  • Infeasible edited grasps, anchor-object changes, and table penetrations occur only rarely but can disrupt reconstruction or validation.A pure black table can make its boundary ambiguous during editing; a textured table could reduce table-penetration cases.Source: §A.6 · p. 19, Figure 11 · p. 20

B.1 Method Details

The goal-generation pipeline turns an edited image into a world-frame target pose, checks that pose in physics, and plans trajectories into constrained goal relations. Prompts specify the desired task while preserving the scene viewpoint and existing objects.

B.1 Method Details

  • Prompts preserve viewpoint and lighting, reposition existing target objects rather than creating new ones, and specify case-specific goal relations.Examples include hanging a mug, wedging a plate rim between rack rails, and inserting a tube into a rack.Source: §B.1 · p. 20
  • Goal-pose estimation uses rendered templates, DINOv3 correspondences, depth-consistent 3D registration, and ICP to recover a world-frame target-object pose.The best Procrustes pose initializes ICP, which refines alignment using nearest neighbors and depth-derived goal points.Source: §B.1 · pp. 20–21
  • Physics validation rejects initial collisions and accepts only poses whose translation and rotation remain below drift thresholds after simulation.Local pose perturbations are simulated with the anchor, table, and optional wall to filter visually plausible but physically invalid goals.Source: §B.1 · p. 22
  • For constrained goals, an MPC escape path is reversed into an insertion-path warm start for forward simulation search.The escape search rejects collisions with the anchor, table, or wall; without this initialization, forward search rarely finds narrow insertion trajectories.Source: §B.1 · p. 22

B.2 Additional Baseline Analysis

Qualitative baseline comparisons show that physically plausible placement alone can miss constrained semantic relations. Image editing proposes a meaningful configuration before metric pose recovery and physics validation refine it.

B.2 Additional Baseline Analysis

  • Molmo-2D often finds a plausible image location, but lifting one point gives coarse translation without the rotation needed for hanging, insertion, or fitting.The baseline lifts the selected pixel using depth and camera calibration while fixing object rotation.Source: §B.2 · pp. 22, 24, Figure 13 · p. 22
  • More VLM-6DoF iterations correct some gross placement errors, yet noisy metric poses remain; the 10-iteration variant improves over 5 iterations but remains far below the method.Small translation or rotation errors can break contact-rich tasks with narrow clearances.Source: Figure 14 · p. 23, Figure 15 · p. 23, §B.2 · p. 24
  • PhyScensis can produce plausible support placements but may miss constrained relations because geometric search and post-hoc selection do not directly propose semantic configurations.The image-editing approach instead proposes a relation for pose estimation and physics validation to refine.Source: Figure 16 · p. 23, §B.2 · p. 24

B.3 Failure Cases

Goal-generation failures mainly stem from edits that specify infeasible goals or change the anchor, with approximate symmetries also making exact rotation ambiguous. Additional examples show how depth alignment and physics validation refine imperfect visual proposals.

B.3 Failure Cases

  • Most failures arise when image editing specifies an infeasible goal or changes the anchor; approximate target symmetries can also make exact rotation ambiguous.The failure examples include infeasible goals and altered anchors.Source: §B.3 · p. 24, Figure 17 · p. 24
  • An upside-down tube is counted as a success when its goal remains largely correct, physically feasible, and semantically reasonable.This illustrates the stated success criterion for an approximately symmetric object.Source: §B.3 · p. 25

B.4 Additional Goal-State Generation Visualizations

  • Edited images can provide informative visual goals despite imperfect edits, and pose jitter followed by physics validation can recover precise, plausible poses.Figure 19 columns show the original image, edited image, estimated pose, and physics-validated goal pose.Source: §B.4 · p. 25, Figure 19 · p. 26
  • Depth alignment corrects displaced raw point clouds in the mug and hanger cases, making them accurate enough for downstream pose estimation and validation.Figure 18 presents additional anchor-based point-cloud alignment results.Source: §B.4 · p. 25, Figure 18 · p. 25
Loading 2609.06279v1…