GIF: Agentic Generation of Interactive and Functional Object Compositions for Robot Learning

1 Introduction

• Robot manipulation foundation models motivate scalable simulation pipelines for evaluation and data generation, while manual scene design has required substantial expert effort.Automated scene-generation work has made progress on diverse simulated environments.Source: §1 · pp. 1–2
• Manipulation needs precise relative 6-DoF poses that realize functional contact, avoid interpenetration, and remain stable in simulation.Existing generators mainly specify approximate scene-scale placements rather than local object-to-object relations.Source: §1 · p. 2
• GIF recasts scene generation as reconstruction followed by pose recovery, pairing CoGen’s instance-disentangled assets with GPRM’s geometry- and physics-guided refinement.A VLM verifier selects candidates against the structured specification; the stated focus is two-object interactions, with a formulation that extends naturally to multi-object scenes.Source: §1 · p. 2
• Across 120 generated scenes, GIF outperforms strong baselines on relation matching, object quality, collision rate, and human preference across eight contact-geometry classes.The benchmark is framed as a contact-precise, simulation-ready scene-generation problem.Source: §1 · p. 2

Simulation-Ready Scene Generation for Robot Learning

• Robotics-oriented scene generators mainly handle rooms, tables, support surfaces, or coarse spatial relations, whereas GIF targets contact-precise two-object relations.Examples include capping, hooking, slotting, spanning, and pegging.Source: §2 · p. 2

Image-Conditioned Asset Construction and Interactive Object Compositions

• CoGen addresses the asset bottleneck in functional pairs, where mutual occlusion can fuse instances, leak partner geometry, or hallucinate hidden surfaces.Prior work includes image-conditioned reconstruction pipelines [11] [69] [70] [71] [72] [73] [9] [74] [10] [75] and multi-instance composition methods such as REPARO [76].Source: §2 · pp. 2–3

Object-to-Object Affordance and Composition Pose Recovery

• Existing pose methods rely on coarse language specifications, potentially out-of-distribution pose estimators, or fragile registration, while GPRM refines initial poses through geometry, physics, and VLM selection.Many existing methods assume assets, fixed placement families, support-surface plausibility, or collision avoidance rather than tight semantic contact.Source: §2 · p. 3

3 Method

• GIF represents a functional object pair as rigid meshes with camera-frame poses, then refines CoGen’s initial poses and verifies candidate scenes before downstream learning.The verified scene becomes the terminal configuration for trajectory synthesis and VLA policy learning.Source: §3 · p. 3

3.1 Multimodal Input Canonicalization

• The multimodal agent converts language or image input into a foreground-only reference image and structured JSON record of object identities, attributes, and target relation.It uses an analyze–act–verify loop to create or edit the reference and accepts it, requests further editing, or redraws within an iteration budget.Source: §3.1 · p. 3

3.2 CoGen: Complementary 2D-3D Generative Asset Construction

• CoGen retains direct image-to-3D and erase-and-regenerate mesh-pose candidates because occlusion can leave visible-only geometry or leak one object into another.The erase-and-regenerate branch completes an object after erasing its partner from the reference; both candidate types proceed to refinement.Source: §3.2 · pp. 3–4, Figure 2 · p. 4

3.3 GPRM: Geometry- and Physics-Guided Pose Refinement Modules

• GPRM aligns geometry with ICP and rendered image evidence, constrains collisions, then uses simulation-aware search when collision-free poses fail to preserve the relation under gravity.EvoOpt searches pose neighborhoods using simulation feedback when the VLM selector rejects earlier refinements.Source: §3.3 · pp. 4–5

3.4 VLM-based Scene Selection

• The VLM verifier checks whether the relation holds initially and after simulated settling, then ranks surviving candidates against the structured specification.It inspects tiled multi-view, multi-timestep renders and can trigger further refinement when earlier candidates fail.Source: §3.4 · p. 5

3.5 Policy Data Synthesis and VLA Training

• GIF uses verified scenes as terminal configurations for randomized-start demonstrations, then trains π0.5 on synthesized trajectories for direct real-robot deployment.Trajectory targets account for object symmetries, and rendering and domain randomization vary lighting, textures, camera parameters, and object poses.Source: §3.5 · p. 5

4.1 Experimental Setup

• The benchmark covers 24 tasks across eight contact-geometry classes, with five semantic variants per task for 120 scenes; evaluation uses object, relation, and collision metrics.Baselines are evaluated under adapted protocols, with zero human-feedback runtime budget and image-to-3D modules uniformly replaced by SAM3D.Source: §4.1 · p. 6

4.2 Main Results

• Relationship Matching averages 7.82 for GIF, 6.33 for SceneSmith, and 1.48 for GenieSim; GIF leads every class in both prompt settings.Hooking is the hardest class, yet GIF remains at least 1.6 points ahead of every baseline.Source: §4.2 · p. 6, Figure 4 · p. 7
• Object Matching exceeds SceneSmith by 0.50 on average and never trails by more than 0.13 in any class, while collisions remain near zero.The only residual collisions are in spoon-hook, where a small ring radius trades off against collision reduction to preserve hooking.Source: §4.2 · pp. 6–7
• Human raters prefer GIF against all four baselines, with Relationship Matching wins of 95.9% against SceneSmith and 100% against GenieSim.The study reports large Cohen’s h effect sizes and tight Wilson confidence intervals.Source: Table 2 · p. 6, §4.2 · p. 7

4.3 Ablation Studies

• Disabling complementary reconstruction consistently lowers Object Matching and Relationship Matching across all eight classes, as occlusion and partner-object interference produce unreasonable assets.Figure 5 illustrates texture leakage, a missing flower stem recovered by erase-and-regenerate, and pen truncation under occlusion.Source: §4.3 · p. 7, Figure 5 · p. 8
• Each cumulative GPRM module improves Relationship Matching over the SAM3D initial pose, with EvoOpt adding the largest gain on Hooking (+0.33).The paper reports that small errors particularly affect simulation stability for Hooking.Source: §4.3 · p. 7, Table 4 · p. 7

4.4 GIF-Generated Tasks for Policy Learning

• Real-robot success rises with asset budget: serving increases from 0.20 with two assets to 0.90 with forty, while covering rises from 0.00 to 0.75.Simulation success is higher in absolute value but follows the same monotonic trend; the study trains on simulated trajectories and deploys on two real-robot tasks.Source: §4.4 · p. 8, Figure 6 · p. 8

4.5 Preliminary Extension to Multi-Object Scenes

• For multi-object scenes, an iterative loop proposes and inserts one object at a time, then reruns CoGen and GPRM while holding previously placed geometry fixed.The insertion preserves the current scene’s viewpoint, lighting, and layout.Source: §4.5 · p. 8

5 Limitations and Conclusion

• The multi-object scene extension is preliminary; further results spanning all eight relation classes are in the supplementary material.Source: Figure 7 · p. 8
• GIF targets rigid object pairs; articulated and deformable assets are out of scope, and room-scale layouts remain future work.The stated reason is that convex decomposition and MuJoCo settling assume rigid geometry.Source: §5 · p. 8
• Verified functional compositions support downstream robot learning, with policy learning and real-world deployment included in the reported demonstrations.The conclusion also reports that GIF outperforms strong baselines across eight contact-geometry classes on relationship matching, object fidelity, collision rate, and human preference.Source: §5 · pp. 8–9

A Methodology Details

• Implementation details follow the main pipeline, from multimodal input canonicalization through CoGen, GPRM, VLM selection, and policy data synthesis.Source: A Methodology Details · p. 17

A.1 Implementation Backends

• The implementation-backend subsection points to Tab. S1 for the model and software modules underlying the agentic pipeline.Prompt templates define task-level LLM/VLM behavior; Tab. S1 lists the underlying modules.Source: §A.1 · p. 17, Table S1 · p. 17

A.2 Multimodal Input Canonicalization

• The canonicalization agent accepts instructions and optional scene or asset images, then returns a foreground reference image and structured object-and-relation specification.It analyzes, generates or edits, and verifies the candidate; a 3D asset is first rendered as an image so it shares the same visual interface.Source: §A.2 · pp. 17–18

A.3 CoGen: Complementary 2D-3D Generative Asset Construction

• CoGen creates direct and partner-removed reconstructions, then supplies cross-branch mesh–pose pairings for downstream geometric and physical selection.A synonym-driven retry broadens mask preparation for long-tail object names; the erase branch uses a closed-loop verifier that prioritizes preserving the remaining object's pose.Source: §A.3 · p. 18

A.4 GPRM Optimization Details

• GPRM refines poses through point-cloud ICP and render-based mask/depth losses, accepting render optimization only when all four per-object loss components improve.The target cloud is sampled inside an eroded instance mask, while render losses cover mask recall, precision, self-depth, and occlusion.Source: §A.4 · p. 19
• ColliOpt seeks a nearby collision-free pose using separation between convex-decomposition pieces, while EvoOpt searches perturbed poses scored through simulated physical behavior.EvoOpt is used only if the VLM rejects candidates from the earlier three refinements.Source: §A.4 · pp. 19–20

A.5 Cross-Candidate Voting Example

• Cross-candidate voting presents tiled candidate timelines with labels and separators; the VLM selects the candidate it judges best against Jscene.Panels show initial and gravity-settled states in perspective and top-down views; the selector first verifies scenes individually, then ranks surviving candidates jointly.Source: Figure S1 · p. 20, §A.5 · p. 20

A.6 Policy Data Synthesis and VLA Training

• Trajectory synthesis transports the verified relative transform into a randomized reference-object frame, plans collision-free pick-and-place motions, and discards failed or unstable executions.Target poses are sampled over task-conditioned symmetry equivalence classes, including admissible cyclic rotations or uniformly sampled rotations around a continuous symmetry axis.Source: §A.6 · p. 20
• The policy uses the π0.5 architecture template with a Qwen3-VL-2B-Instruct backbone, wrist and third-person views, proprioception, and chunked action outputs.The vision encoder and visual merger are frozen; the language backbone and action expert are trainable.Source: §A.6 · p. 21
• Asset-scaling experiments hold training trajectories at 50,000 per task while varying GIF-generated task-asset variants, and evaluate on held-out assets with 80 rollouts per budget.Simulation success uses task-specific predicates for placing a sausage inside a box or stably covering a pot opening with a lid.Source: §A.6 · p. 21
• Data generation randomizes poses, distractors, table geometry, lighting, camera poses, and materials around each verified terminal relation.Rendered observations use the same third-person and wrist-camera convention as real-robot evaluation.Source: §A.6 · p. 21
• The sim-trained policy is deployed directly on the robot without real-world fine-tuning, using third-person and wrist cameras plus proprioception.Real-robot success requires the sausage to remain in the box or the lid to stably cover the pot opening, with at most two grasp attempts.Source: §A.6 · pp. 22–23

• The section explains how the benchmark is derived from egocentric manipulation data.Source: B Experimental Design Details · p. 23
• It describes how the automatic and human evaluation metrics are computed.Source: B Experimental Design Details · p. 23
• It details how each baseline is adapted to the benchmark.Source: B Experimental Design Details · p. 23
• It evaluates a preliminary extension to multi-object scenes.Source: B Experimental Design Details · p. 23

B.1 Benchmark Scene Setup

• Millions of narration descriptions and roughly 0.6M FHO actions are filtered to two-object contact events, with 50,000 examples sampled for taxonomy construction.Source: §B.1 · p. 23
• LLM-based clustering groups contact patterns into eight classes, from containment and capping to spanning and pegging.Source: §B.1 · p. 23
• Three tasks per class and five semantic variants per task produce 24 tasks and 120 benchmark scenes.Source: §B.1 · p. 23

B.2 Metrics

• Language scorers see the task description and settled scene, while image-conditioned scorers also receive the reference image.Source: §B.2 · p. 23

B.2.1 VLM-based scoring

• VLM scoring assigns Object Matching and Relationship Matching scores from 0 to 10, averaged across five variants and aggregated by relation class.Source: §B.2.1 · p. 24

• The main-results table reports 95% confidence intervals for Object Matching and Relationship Matching under both prompt modalities.Source: §B.3 · p. 24, Table S6 · p. 24
• The cumulative GPRM ablation table reports 95% confidence intervals for Relationship Matching.Source: Table S7 · p. 24, §B.3 · p. 24
• The CoGen complementary-reconstruction ablation table reports confidence intervals for each entry.Source: Table S8 · p. 24, §B.3 · p. 24

B.3.1 User study

• The user study recruited 107 raters and collected more than 1,000 pairwise judgments across four baseline pairings and both prompt settings.Source: §B.3.1 · pp. 24–25
• Trials randomized output order and concealed method names; O.M. ties count as half-wins, while R.M. ties are excluded from decisive win rates.Calibration failures are excluded, and reported comparisons include Wilson intervals and Cohen’s h effect sizes.Source: §B.3.1 · pp. 25–26

B.4 Baselines and Comparison Protocol

• All systems receive zero human-feedback budget, and image-to-3D baselines use SAM3D to focus comparisons on composition, relation reasoning, and placement.RoLA and TabletopGen receive synthesized table-conditioned prompts; SceneSmith is evaluated only at its manipuland-placement stage.Source: §B.4 · pp. 26–28, Table S9 · p. 27

B.5 Efficiency, API Usage, and Statistical Uncertainty

• 425 s per scene (243–935 s), 19.2K API tokens (8.4K–31.5K), and about $0.25 per scene; the VLM verifier accepts 92% of attempts.GPRM takes 5.3 s and no tokens; its reported wall-clock time reflects multiprocessing, and image-generation token counts were not logged. MIC averages 1.1 calls per scene and Erase 2.3.Source: Table S10 · p. 28, Figure S5 · p. 28, §B.5 · p. 28

B.6 Preliminary Extension to Multi-Object Scenes

• Per-attempt wall-clock and token costs grow close to linearly with object count, but retries may still make system-level cost rise sharply.Previously placed objects are frozen as merged static geometry while only the newcomer passes through GIF; the curves are shown in Fig. S6.Source: §B.6 · p. 29, Figure S6 · p. 29
• The nine-step expansion spans all eight taxonomy classes, but remains preliminary and its reported example used a human in the loop.A human adjusted parameters, switched submodules or retried tools, and acted as verifier.Source: §B.6 · p. 28, Figure S7 · p. 29
• Placing a book into the helmet-shaped shelf at the sixth extension required several attempts and manual intervention.The example adds objects one at a time while holding previously placed geometry fixed.Source: Figure S7 · p. 29

C Task Suite Descriptions and Scene Galleries

• Each contact-geometry class is shown as a 3 × 5 gallery, with one task per row and five semantic variants per task.Across the benchmark, the appendix lists 8 × 3 = 24 tasks and prompts for 120 reference scenes.Source: C Task Suite Descriptions and Scene… · p. 29

C.1 C.1 Containment

• Containment tasks place a flower into a vase, a pen into a multi-compartment organizer, or a flowering plant pot into a hanging pot stand.The prompts specify insertion or seating within each receptacle or support, with five semantic variants per task.Source: §C.1 · pp. 30–31

C.2 C.2 Capping

• Capping tasks fit a lid onto a pot or teacup, or insert a wine cork into a bottle mouth.The prompts describe lids seated on the vessel rim or corks inserted into the bottle opening.Source: §C.2 · p. 32

C.3 C.3 Resting

• Resting tasks place a hat on a mannequin head, a coffee cup on a machine’s drip tray, or an egg in a carton cup.Each task has five semantic variants.Source: §C.3 · pp. 33–34

C.4 C.4 Leaning

• Leaning tasks support a phone on a phone stand, a shoe on a sloped display stand, or a steamer lid against a basket rim.The prompts specify supporting contacts, including phone support lips, shoe-stop walls, and lid-to-rim contact.Source: §C.4 · pp. 35–36

C.5 C.5 Hooking

• The hooking task begins with hooking a cup on a cup-rack tree.Source: §C.5 · p. 37
• Hooking scenes show cups suspended by handles, hats hanging from a single coat-rack hook, and soup spoons supported vertically by their handle ends.Some hat variants require other hooks to remain empty and visible; the spoon variants use a wall hook or hook rail.Source: §C.5 · pp. 37–38, Figure S12 · p. 39

C.6 C.6 Slotting

• Slotting scenes place plates upright between dish-rack pegs, bottles in wine-rack supports, and books upright within individual book-stand compartments.The bottle variants include diamond cells, grooves, crossed-board cells, wire holders, and grid cells; the book descriptions emphasize staying within one compartment.Source: §C.6 · pp. 39–40, Figure S13 · p. 41

C.7 C.7 Spanning

• Spanning scenes show forks bridging plate rims, calligraphy brushes resting on cradle notches, and ladles upright with bowls cradled and handles guided by stands.For fork variants, the handle may slope beyond the plate rim toward the tabletop.Source: §C.7 · pp. 41–42, Figure S14 · p. 43

C.8 C.8 Pegging

• Pegging scenes loop toss or jewelry rings around stand supports and hang mugs upside down with a rack peg entering each mug’s open mouth.Toss rings use upright pegs or tapered posts, while jewelry rings fit around tapered holders.Source: §C.8 · pp. 43–44, Figure S15 · p. 45

D.1 Input scene agent; D.2 Synonym retry for mask preparation

• The input-scene agent produces a clean two-object reference scene, while mask-retry prompts generalize or visually describe object names when masks are empty or low-confidence.The scene agent uses available language and images to choose whether to draw or edit; the synonym module returns five distinct alternatives ranked best-first.Source: §D.1 · p. 45, §D.2 · p. 46

D.3 Asset erase agent

• The erase agent iteratively edits and compares images, prioritizing preservation of the partner object's pose and stopping when its score reaches 90.Its rubric penalizes pose change most heavily, followed by residual target visibility, preserved-object appearance changes, and background artefacts.Source: §D.3 · p. 46

D.4 Asset symmetry analyzer

• The symmetry analyzer admits rotations only when the object still fulfills its contact relation after a lift-rotate-place test.It idealizes away minor bends and textures, and ignores collisions along the rotation path.Source: §D.4 · pp. 46–47

D.5 Multi-view scene verifier

• The multi-view verifier checks whether the contact relation holds in the gravity-settled final frame and whether the scene is feasible for grasping.Time-series images depict passive gravity-only settling, not robot execution; the final frame is the settled state.Source: §D.5 · p. 47

D.6 Candidate voter

• The candidate voter prioritizes relation preservation after settling, then clean separate meshes, physical plausibility, and manipulation reasonableness.It rejects candidates whose target is removed, separated, or fallen away, and favors clean geometry over perfect alignment with corrupted meshes.Source: §D.6 · pp. 47–48

D.7 Multi-object scene expander

• The expander plans up to a set insertion budget, including requested objects when specified or semantically related additions otherwise.It assigns spatial relations and reconstruction-oriented descriptions while directing the editor to preserve the existing scene, viewpoint, lighting, and style.Source: §D.7 · p. 48

D.8 VLM scene scorer

• The VLM scorer separately rates object matching and relationship matching using task text, optionally a reference image, and gravity-settled renders.Relationship scores assess named geometry, common-sense placement, interpenetration, and physical plausibility under gravity.Source: §D.8 · pp. 48–49

D.9 Ego4D-driven taxonomy clustering

• The Ego4D-driven taxonomy pipeline summarizes narration relations, merges phrases with interchangeable contact geometry, clusters them into six to ten geometric classes, and selects three tasks per class.Classes must be mutually exclusive and exhaustive, correspond to distinct contact patterns with geometric templates, and span household and workplace scenarios.Source: §D.9 · pp. 49–50