Source-linked AI summary
Representation-Guided Generation and Integration of Executable Programs for Robot Manipulation
Ruixiao Yang, Mingxin Yu, Chuchu Fan
TL;DR
VLM-generated manipulation components may not share compatible geometric and task-level information, while connecting traditional system components requires substantial expertise. RIVET generates programs around a shared object-centric representation and reuses them across configurations; its systems complete tasks more reliably than simulation baselines and work across configurations on a physical robot.
Problem
Generated components can disagree about object identities, coordinate conventions, relation semantics, and action effects, while connecting traditional manipulation tools requires substantial human effort and domain expertise.
Source: §I · p. 1Method
RIVET guides VLM generation of manipulation programs with a shared representation combining per-object 6D poses and a task-relevant relation graph.
Source: §I · p. 1, §III · p. 3Results
RIVET systems complete tasks more reliably than simulation baselines, and the same physical-robot programs work across start-goal configurations and a camera-view change.
Source: §I · p. 2Takeaways & Limitations
Abstract
from arXiv · showhide
Building a robotic manipulation system requires connecting perception, planning, and control through carefully designed representations and interfaces. VLM code generation offers a way to automate this construction, but independently generated components may operate on incompatible geometric and task-level information. We present Representation-guided Integration of VLM-generated Executable Task programs (RIVET), a framework for generating complete manipulation systems around a shared object-centric representation. The representation combines per-object 6D poses, which preserve the metric information required for action grounding, with a relation graph that exposes the task-level structure required for planning. Guided by this representation, a VLM generates cooperating perception, rendering, relation-inference, and planning programs, each combining task-specific computation with available packages where useful. The resulting programs are authored once for a manipulation domain and reused on unseen start and goal configurations without code regeneration. We evaluate RIVET on cube stacking, tangram rearrangement, and three-dimensional assembly in simulation and on a physical robot, where we achieve 83% overall success rate in the real world by reusing offline-generated systems. Our results demonstrate that representation-guided program generation can adapt a common manipulation framework to tasks with different geometric, relational, and sequential requirements.
I. INTRODUCTION
VLM-based components can disagree about geometric and task-level information, while building traditional manipulation systems requires substantial expertise. RIVET uses a shared object-centric representation to generate reusable programs, evaluated across three tasks in simulation and on a physical robot.
I. INTRODUCTION
- Independent perception, planning, and execution components may disagree on object identities, coordinate conventions, relation semantics, or action effects without a shared representation.Such mismatches make it difficult to connect visual observations, task reasoning, and execution.Source: §I · p. 1
- RIVET combines per-object 6D poses with a relation graph so generated modules share metric information for action grounding and structural information for task constraints and action order.Both components refer to the same object instances, connecting perception, relational reasoning, and planning.Source: §I · p. 1
- A VLM generates cooperating perception, rendering, relation-inference, and planning programs around the representation, using task-specific computation and available packages where useful.The programs are authored sequentially, with later functions receiving earlier generated code.Source: §I · p. 1, Figure 1 · p. 2
- Generated programs are reused across new configurations within a task domain without another authoring round; when scene interpretation and planning are local programs, repeated solving needs no runtime VLM queries.Traditional system construction requires substantial human effort and domain expertise for each new task.Source: §I · pp. 1–2
- RIVET systems complete tasks more reliably than simulation baselines, and the same physical-robot programs work across start-goal configurations and a camera-view change.The evaluation covers cube stacking, tangram rearrangement, and three-dimensional assembly; RIVET's representation supports concrete placement targets where all baselines fail on two task sets.Source: §I · p. 2
II. RELATED WORK
Prior work uses VLMs for task planning and program synthesis, and some systems author reusable planning or perception programs offline. RIVET focuses on coordinating visual processing and goal-directed planning for physical manipulation.
II. RELATED WORK
- VLM-based task-planning approaches such as ViLa, VeriGraph, and PLanAR retain model inference during task interpretation or solving at deployment.These approaches use visual feedback, scene-graph transformations, or symbolic planning and execution verification.Source: §II · pp. 2–3
- Program-synthesis systems compose perception and action tools or encode geometric objectives, with generated task logic available for inspection and conventional algorithms handling geometric computation.VoxPoser also caches generated code for reevaluation under visual feedback.Source: §II · p. 3
- Offline-authoring systems reuse planning programs, but visual grounding may still invoke a VLM; RIVET authors visual processing and planning programs together for reuse across configurations.RIVET freezes the renderer, perceiver, and solver within a domain, making their task logic available for inspection and programmatic checking.Source: §II · p. 3
III. METHODOLOGY
RIVET uses a shared object-centric scene representation to coordinate generated perception and planning modules. The generated programs process new start-goal observations through a fixed pipeline and produce grounded object-level actions.
III. METHODOLOGY
- Within a task domain, the task specification, object meshes, and camera calibration remain fixed while start and goal observations vary across instances.The objective is an executable plan that transforms the observed start configuration into the goal while satisfying the task specification.Source: §III · p. 3
- The shared representation combines per-object 6D poses with a task-relevant relation graph, linking metric action grounding to reasoning about constraints and action order.Both components refer to the same object instances.Source: §III · p. 3
- RIVET fixes module responsibilities and interfaces, then generates task-specific algorithms for recovering object states, reconstructing them visually, inferring relations, and planning actions.Generated code can include calls to available packages where useful.Source: §III · p. 3
A. Method Overview
- A manually specified scene representation serves as the interface between separately generated perception and planning modules.Object records preserve metric placement targets, while relations expose discrete conditions for valid action orderings; fixed conventions keep identities, frames, and semantics consistent.Source: A. Method Overview · pp. 3–4
- Offline-generated programs map observations into the shared representation and define planning actions, preconditions, and transitions; online, they process new start-goal pairs and produce object-level plans grounded in target poses.A fixed control interface grounds those actions into robot motions, and new instances do not trigger another authoring round.Source: A. Method Overview · p. 4
B. Perception: Constructing Planner-Ready Scene States
RIVET converts image observations into planner-ready states through fixed-interface perception, rendering, and relation functions. These functions connect metric object records to task-relevant relational facts while preserving shared object identities and coordinate conventions.
B. Perception: Constructing Planner-Ready Scene States
- Perceive recovers metric object records from RGB-D observations by selecting and executing suitable perception tools under the shared identity and coordinate conventions.Its prompt supplies reference observations, object meshes, camera calibration, the required record format, and available tool interfaces.Source: B. Perception: Constructing… · p. 4
- Render places canonical meshes according to the perceived poses and reconstructs an image that connects metric records with visual scene geometry.The reconstructed image supports comparison with the reference observation and informs relate's generation.Source: B. Perception: Constructing… · p. 4
- Relate infers task-relevant predicates from perceived object states and geometry, returning a graph whose object nodes and typed edges retain links to the metric records.The required relations vary by task, such as support and accessibility for stacking or relative position and adjacency for arrangement.Source: B. Perception: Constructing… · pp. 4–5
C. Planning with executable transitions
RIVET's planner uses relational structure and metric object records to construct grounded object actions for start and goal states. It searches executable state transitions subject to action preconditions and stops when the generated goal condition is satisfied.
C. Planning with executable transitions
- The generated planner uses both scene-representation components to plan for start and goal states rather than produce a plan only for the reference instance.Its prompt includes the task specification, reference states, and contracts and code for previously generated perception functions.Source: C. Planning with executable transitions · p. 5
- The planner searches for an action sequence whose transitions satisfy each action's precondition and terminate when the generated goal condition holds.The plan function defines task-level actions, their preconditions, their effects on the scene state, and the goal condition.Source: C. Planning with executable transitions · p. 5
- The relation graph supplies structural constraints for selecting movable objects and action order, while metric records provide object identities and target poses.A support relation can require an obstructing object to move before its support.Source: C. Planning with executable transitions · p. 5
- If no action sequence satisfies the generated constraints, the planning function returns an explicit planning failure.Task action rules and goal conditions are incorporated when plan is authored because the task specification is fixed within a domain.Source: C. Planning with executable transitions · p. 5
- The planner outputs object-level transfers rather than robot trajectories, leaving grasp selection and motion generation to the downstream control interface.This separates task-level ordering and placement targets from robot-specific feasibility and execution.Source: C. Planning with executable transitions · p. 5
D. Plan Grounding and Robot Execution
RIVET's planner specifies object-level actions and target poses, while a fixed execution interface converts those transfers into robot motions. Actions run sequentially without intermediate visual replanning.
D. Plan Grounding and Robot Execution
- The plan specifies the moved object and its target pose, which the fixed execution interface expresses under the robot base frame using camera-to-robot calibration.The execution interface is not VLM-generated and remains unchanged across task domains.Source: D. Plan Grounding and Robot Execution · p. 5
- For each planned action, the executor selects a compatible grasp, checks kinematic and geometric feasibility, and generates approach, grasp, transfer, release, and retreat motions.These procedures realize the planned transfer without changing its object identity, target pose, or action order.Source: D. Plan Grounding and Robot Execution · p. 5
- The robot executes the actions sequentially without intermediate visual replanning.Source: D. Plan Grounding and Robot Execution · p. 5
IV. EXPERIMENTS
The experiments assess whether RIVET constructs effective manipulation systems across tasks with different perception and planning requirements, and whether authored systems transfer to new configurations without code regeneration. The evaluation includes simulation comparisons, physical deployment, camera-view changes, failure analysis, and program-generation cost.
IV. EXPERIMENTS
- The experiments compare RIVET systems across tasks with different perception and planning requirements against online VLM planners and alternative program-generation methods.Source: §IV · p. 5
- A central evaluation question is whether a fixed system authored for a task domain can be reused on new initial-goal configurations without code regeneration.Source: §IV · p. 5
- The evaluation spans simulation comparisons, real-world deployment, camera-view changes, failure sources across stages, and program-generation computation cost.Source: §IV · p. 5
A. Experimental Setup
The experiments test RIVET across three manipulation tasks with distinct geometric and relational demands, using both simulation and real-world evaluations. The protocols assess independently generated systems in simulation and fixed, reused systems in the real world.
A. Experimental Setup
- The real-world platform uses a Franka Research 3 arm and Robotiq 2F-85 gripper, with a wrist-mounted ZED M camera and a front-positioned ZED 2i RGB-D camera observing the workspace.Source: A. Experimental Setup · p. 5
- The three tasks test complementary demands: cube stacking emphasizes 3D placement and support order, tangram arrangement planar position and orientation, and 3D assembly occlusion, support, accessibility, and assembly order.Together they test metric perception, relational reasoning, and physically grounded planning.Source: A. Experimental Setup · p. 6
- In simulation, 10 independently generated systems are evaluated on six start-goal pairs each, with success measured at state estimation, symbolic planning, and final execution under a 10-minute instance limit.Each task has 20 goal configurations paired with three initial configurations, yielding 60 trials per task.Source: A. Experimental Setup · p. 6
- In the real world, one system per task is generated using Goal 1, frozen, and reused across four goals and 20 trials without modification.Goals 2–4 test generalization beyond the authoring goal.Source: A. Experimental Setup · p. 6
- The comparison includes online VLM planners ViLa [8] and VeriGraph [4], plus reusable-program methods CaP [15] and Instruct2Act [10].Instruct2Act uses SAM for segmentation and CLIP [17] for identification while retaining image-level reasoning.Source: A. Experimental Setup · pp. 6–7
B. Main comparison in simulation
The simulation results assess both final task completion and intermediate pipeline stages. RIVET reaches high final success on cube stacking and tangram arrangement, while the intermediate metrics distinguish task reasoning from physical completion.
B. Main comparison in simulation
- 100% on cube stacking and 90% on tangram arrangement are RIVET’s final success rates in simulation.The tasks respectively test accurate placement with support-aware action ordering and precise planar positioning and orientation.Source: B. Main comparison in simulation · p. 7
- Intermediate metrics separate successful task reasoning from successful physical completion, since valid symbolic plans can still fail without appropriate placement poses.The shared representation provides both geometric targets and task-relevant relationships to planning code.Source: B. Main comparison in simulation · p. 7
C. Comparative Analysis and Failure Modes
The comparison identifies gaps in how baselines represent task structure and metric geometry, while RIVET uses both in planning. RIVET nevertheless remains subject to errors in perception, search, and physical execution.
a) Baseline failures
- ViLa can miss action dependencies, while VeriGraph makes them explicit but can infer incorrect relations; neither alone supplies RIVET’s combination of dependencies and metric poses.A global translation preserves pairwise relations but still requires moving every object.Source: C. Comparative Analysis and Failure… · p. 7
b) What RIVET addresses
- RIVET’s pose-plus-relation representation lets the planner use relations for action validity and poses for concrete placement targets.Representation-guided interfaces also keep object identities, coordinate conventions, and relation semantics consistent across perception and planning.Source: C. Comparative Analysis and Failure… · p. 7
c) Remaining failures of RIVET
- RIVET’s remaining failures include incorrect pose hypotheses under severe occlusion, inefficient planners exceeding their search budget, and grasp or motion infeasibility during execution.Performance remains bounded by the selected perception methods, generated search algorithms, and robot controller.Source: C. Comparative Analysis and Failure… · p. 7
D. Applicability in Real World
RIVET’s task-domain programs are evaluated on real-robot trials and reused across goals after authoring. The paper also tests camera-view changes and concludes that future work should add execution feedback and failure-recovery replanning.
D. Applicability in Real World
- Across 60 real-world trials, RIVET succeeded in 50 (83.3%): 18/20 each for cube stacking and tangram arrangement, and 14/20 for 3D assembly.Source: D. Applicability in Real World · p. 7
- Programs generated using Goal 1 are frozen and reused across all four goals without code regeneration; performance on Goals 2–4 demonstrates generalization beyond the authoring configuration.The real-world table reports successes out of five attempts for each entry.Source: D. Applicability in Real World · p. 8, Table II · p. 8
E. Robustness to Camera Viewpoint
- The tested viewpoint changes are accommodated when task-relevant objects and relationships remain visible, while substantial occlusion can hinder pose recovery and affect relation inference and planning.The generated system takes the camera as a parameter, allowing a new view without code modification.Source: E. Robustness to Camera Viewpoint · p. 8
V. CONCLUSION AND DISCUSSION
- Future work will incorporate execution feedback and failure-recovery replanning while extending RIVET to more complex, contact-rich manipulation tasks.The framework generates task-specific perception, rendering, and planning programs around shared geometric and relational representations.Source: §V · p. 8