PaperlayerView exact revision on arXiv

Source-linked AI summary

RAPID: Robot Agentic Programming from Demonstrations

Yuyao Liu, Jiayuan Mao, David Hsu, Leslie Pack Kaelbling, Tomás Lozano-Pérez

arXiv:2609.30249v1cs.ROcs.AIcs.CV

TL;DR

Coding agents for robot manipulation typically rely on task specifications, manipulation primitives, and interactive verification environments being supplied in advance. RAPID derives these ingredients and reusable relational programs from a single visual demonstration, outperforming baselines on simulated nonprehensile tasks and also being evaluated on LIBERO-Pro and a real robot.

  • Problem

    Robotic coding-agent systems typically assume task specifications, manipulation primitives, and interactive verification environments are provided a priori, motivating how to close the agentic loop without those ingredients.

    Source: §I · p. 1

  • Method

    RAPID uses a single visual demonstration to infer task specifications, construct and compose manipulation primitives, reconstruct a verification environment, and refine reusable object-centric relational programs.

    Source: §I · p. 1

  • Results

    RAPID outperforms baselines on eight simulated contact-rich nonprehensile tasks, is evaluated on LIBERO-Pro, and executes optimized behaviors effectively on a real robot.

    Source: §I · p. 2, D. Real-World Experiments for… · p. 7

  • Takeaways & Limitations

    Object-centric relational programs express object-level effects and relational constraints to adapt demonstrated strategy structure to scene changes.

    Source: §I · p. 1

Abstract

from arXiv · show

Coding agents have demonstrated enormous success in solving complex programming problems. To leverage their potential for robot systems, this work introduces Robot Agentic Programming from Demonstrations (RAPID), which automatically generates, verifies, and refines robot programs, given a single visual human demonstration. The iterative agentic loop of code refinement requires several key ingredients: (i) a testable task specification, (ii) action primitives for robot execution, and (iii) an interactive environment for program execution and verification. RAPID infers all three from the demonstration automatically. To make the resulting program reusable beyond the demonstration setting, RAPID uses an object-centric relational program representation that focuses on the underlying structure of the demonstrated strategy rather than the specific motion per se: it expresses the action primitives as trajectory-optimization programs that realize object-level motion effects, while composing them through relational constraints that capture scene-specific geometry at run time. We evaluated RAPID in simulation on eight challenging contact-rich nonprehensile manipulation tasks as well as general prehensile manipulation tasks in the LIBERO-Pro benchmark. We also successfully deployed it on a real Franka arm and evaluated on all eight nonprehensile tasks. In all experiments, RAPID demonstrated strong performance, with generalization over object pose, shape, material, and environment. Website: https://yuyaoliu.me/projects/rapid.

I. INTRODUCTION

RAPID uses a single visual demonstration to bootstrap the specifications, primitives, and verification environment needed for iterative robot-program refinement. Its object-centric relational representation targets reusable strategies that adapt to scene changes rather than replaying fixed motions.

I. INTRODUCTION

  • A single visual demonstration lets RAPID infer the task specification, construct manipulation primitives, compose a strategy, and reconstruct a simulation for verification and refinement.It also generates scene variants to improve generalization.Source: §I · p. 1
  • RAPID represents primitives as local trajectory-optimization programs for object-level effects and composes them through relational constraints that resolve scene-dependent geometry.Examples include pushing by a displacement and flipping by an angle.Source: §I · p. 1, Figure 2 · p. 2
  • The coding agent tests candidate programs in a reconstructed environment with filtered scene variants, then revises them based on observed failures.Variants that are physically infeasible or inconsistent with the task are discarded.Source: §I · p. 2
  • Nonprehensile manipulation is a testbed for learning reusable primitives because its diverse geometry- and contact-dependent behaviors resist a fixed primitive library.The demonstrated strategy is intended to generalize across object appearance, pose, geometry, physical properties, and scene configuration.Source: Figure 1 · p. 1, §I · p. 2
  • RAPID outperforms baselines on eight simulated contact-rich nonprehensile tasks and is also evaluated on LIBERO-Pro and a real robot.The evaluation asks whether programs from one demonstrated instance generalize across object and scene variations.Source: §I · p. 2

II. RELATED WORK

RAPID differs from prior robotic coding and manipulation approaches by deriving the ingredients for iterative programming from a single visual demonstration. It represents reusable manipulation strategies through object-level effects, scene relations, and locally optimized primitives.

II. RELATED WORK

  • RAPID contrasts with prior coding-agent, demonstration-learning, and nonprehensile-manipulation approaches by deriving the task ingredients and transferable strategies from one visual demonstration.Prior systems commonly assume supplied task signals, primitives, or verification environments; related representations are often embedded in fixed pipelines.Source: §II · p. 2

III. OVERVIEW OF RAPID

  • Given a demonstration and language instruction, RAPID constructs executable primitives and a reusable strategy for task instances varying in appearance, geometry, pose, physical properties, and scene configuration.At deployment, it reconstructs a novel scene, binds objects to strategy roles, and executes the resulting trajectory while leaving the strategy and primitives unchanged.Source: §III · p. 3

A. Primitives as Local Trajectory-Optimization Programs

  • Primitives optimize robot trajectories to realize requested object effects, adapting motion to scene geometry and physical properties.RAPID uses the cross-entropy method, a gradient-free optimizer, and demonstration-informed progress heuristics for contact-rich interactions.Source: A. Primitives as Local… · pp. 3–4

IV. OBJECT-CENTRIC RELATIONAL PROGRAMS

  • Each primitive specifies an object-level effect, relational interface, verifiable success condition, geometry resolver, and optimization cost.The interface declares semantic roles, relations among them, and the requested effect parameter; runtime scene geometry is resolved within the primitive.Source: A. Primitives as Local… · p. 3

B. Strategies as Primitive Composition by Constraints

A strategy composes primitives into a reusable task-level program by specifying their sequence, role bindings, and effects. Constraints connect phases by deriving each invocation’s role binding and effect argument from the preceding phase’s outcome.

B. Strategies as Primitive Composition by Constraints

  • A strategy composes a sequence of primitives into a task-level program, specifying role mappings and the requested effect for each invocation.It also describes the overall task goal and evaluates the final outcome.Source: B. Strategies as Primitive Composition… · p. 4
  • At invocation, a semantic role binding assigns entities to strategy roles, while primitive effect parameters are derived inside the strategy rather than exposed through its interface.The role binding must satisfy the strategy’s required relations.Source: B. Strategies as Primitive Composition… · p. 4
  • Each phase combines a primitive, a role map, and a connecting constraint that derives its effect argument and depends on the previous phase’s outcome.The constraint is evaluated just before the phase begins.Source: B. Strategies as Primitive Composition… · p. 4
  • The invoked primitive runs by solving its local trajectory-optimization problem and executing the optimized trajectory.The phase’s state is the environment state after the preceding phase executes.Source: B. Strategies as Primitive Composition… · p. 4
  • In the flip-then-press example, the constraint uses the post-flip target position and its relation to the fixture axis to derive the press displacement toward a goal set by the obstacle and fixture.The resulting effect argument is passed to the press-move primitive.Source: B. Strategies as Primitive Composition… · p. 4

C. Strategy Instantiation in Novel Scenes

To deploy a strategy in a novel scene, RAPID generates a semantic binding between scene entities and the strategy’s roles. The strategy and its primitives remain frozen while trajectory optimization adapts to the new scene’s geometry and physical properties.

C. Strategy Instantiation in Novel Scenes

  • Deployment requires a semantic binding that assigns entities in the novel scene to the strategy’s exposed roles, a step taking a few seconds.Primitive bindings and effect arguments are then derived automatically during execution.Source: C. Strategy Instantiation in Novel… · p. 4
  • The strategy interface exposes semantic roles for binding to entities in a novel scene.The coding agent generates that binding from the novel scene and instruction.Source: C. Strategy Instantiation in Novel… · p. 4
  • The strategy and referenced primitives remain frozen across deployment; generalization arises from the relational representation and trajectory optimization adapted to the new scene’s geometry and physical properties.Source: C. Strategy Instantiation in Novel… · p. 4

V. AGENTIC PROGRAMMING FROM A DEMONSTRATION

RAPID constructs and verifies programs through environment reconstruction, primitive construction, and strategy composition.

V. AGENTIC PROGRAMMING FROM A DEMONSTRATION

  • The described agentic programming process includes environment reconstruction, primitive construction, and strategy composition.Source: §V · p. 4
  • Primitive construction is one of the named stages in constructing and verifying the programs.Source: §V · p. 4
  • Strategy composition is the remaining named stage in the program-construction and verification process.Source: §V · p. 4

A. Reconstructing Interactive Environments

RAPID reconstructs an interactive simulation from the demonstration’s initial RGB-D frame and creates feasible scene variants for program verification. Variants randomize scene properties, then filters retain those consistent with task meaning, declared relations, and physical feasibility.

A. Reconstructing Interactive Environments

  • RAPID reconstructs an interactive simulation from the demonstration’s initial RGB-D frame, using vision-language models to identify objects, label them, and estimate their physical properties.The reconstruction process uses object masks, 3D mesh reconstruction, and pose registration before assembling the scene and robot in MuJoCo.Source: A. Reconstructing Interactive… · pp. 4–5
  • The shared verification-and-revision procedure repeats until all candidate executions succeed.The algorithm excerpt marks shared verification and revision and loops until AllSuccessful.Source: A. Reconstructing Interactive… · p. 5
  • The reconstructed environment preserves scene entities, geometry, and spatial relations needed to execute candidate programs and evaluate their success predicates.Source: A. Reconstructing Interactive… · p. 5
  • A task-agnostic sampler varies object appearance, size, pose, mass, friction, and elasticity to generate scene variants.Source: A. Reconstructing Interactive… · p. 5
  • Coding-agent filters retain variants that preserve declared roles and relations, task meaning, and physical feasibility.Source: A. Reconstructing Interactive… · p. 5

B. Primitive Construction

RAPID derives task-specific primitives from demonstration segments, either retrieving an existing primitive or inferring a specification and verifying a generated one. This process captures interaction structure and revises each primitive until its success predicate holds across demonstrated and feasible variants.

B. Primitive Construction

  • For each ordered demonstration segment and local description, RAPID retrieves a matching library primitive or infers a semantic goal and success predicate.When inferring a primitive, the agent also reasons about contact interactions, invariant relations, and quantities that may vary between invocations.Source: B. Primitive Construction · p. 5
  • The agent generates a relational interface, geometry resolver, and optimization cost for the primitive from its specification and interaction analysis.The demonstrated rotation, for example, supplies the flipping-angle argument, while object, support-plane, fixture, and manipulator relations define reusable interaction structure.Source: B. Primitive Construction · p. 5
  • RAPID verifies and revises each primitive across the demonstrated effect–environment pair and feasible variants until its success predicate holds for every pair.The verified primitive is then added to the library.Source: B. Primitive Construction · p. 5

C. Strategy Composition

RAPID composes verified primitives into a task-level strategy using the ordered demonstration segments. It then revises the composition against the reconstructed environment and feasible scene variants while keeping the verified primitives fixed.

C. Strategy Composition

  • The full demonstration and instruction specify the overall task goal and final success predicate for the strategy.The order of interaction segments determines the order of primitive composition.Source: C. Strategy Composition · p. 5
  • For each phase, the agent selects a verified primitive, assigns its role map, and specifies a connecting constraint to form the initial strategy.These components, together with the strategy interface, make up the initial strategy.Source: C. Strategy Composition · p. 5
  • RAPID revises the strategy composition after verification failures until its success predicate holds across the reconstructed environment and feasible scene variants.The verified primitives remain fixed during this strategy-level revision.Source: C. Strategy Composition · p. 5

VI. EXPERIMENTS

The experiments evaluate RAPID's demonstration-to-program loop, the contribution of its object-centric relational representation, and its real-world deployment. They cover nonprehensile tasks, prehensile tasks, and a real robotic system.

VI. EXPERIMENTS

  • The experiments assess whether RAPID can construct reusable nonprehensile programs by closing the agentic coding loop from a single visual demonstration.The section identifies Table II as the simulation results for this evaluation.Source: §VI · p. 5
  • The experiments test whether the object-centric relational program representation improves generalization of the constructed manipulation programs.The section points to Tables II and III for this evaluation.Source: §VI · p. 5
  • The experiments evaluate RAPID's deployment on a real-world robotic system.The section points to Fig. 5 for the real-world evaluation.Source: §VI · p. 5

A. Experimental Setup for Nonprehensile Manipulation

The nonprehensile evaluation tests generalization across varied scenes and compares RAPID with alternative coding-agent approaches. The results report that RAPID outperforms baselines, while removing scene-variant generation substantially reduces performance on most tasks.

A. Experimental Setup for Nonprehensile Manipulation

  • The simulation benchmark covers eight tasks in which initially ungraspable objects require nonprehensile actions such as pushing, flipping, pivoting, or toppling.Each task has 50 novel test scenes varying pose, appearance, size, shape, and material, with irrelevant objects added; success is measured using a private criterion hidden from RAPID.Source: A. Experimental Setup for… · p. 5, Figure 4 · p. 6

B. Simulation Experiments for Nonprehensile Manipulation

  • RAPID substantially outperforms the CaP-Agent0 baselines, including when those baselines receive stronger verification access.Table II reports simulation success rates as mean±std across three trials; the construction cost is incurred once per task, while role binding takes seconds per scene.Source: Table II · p. 6, B. Simulation Experiments for… · p. 6

A. Experimental Setup for Nonprehensile Manipulation

B. Simulation Experiments for Nonprehensile Manipulation

  • Removing scene-variant generation substantially reduces RAPID's performance on all but Tasks 1 and 2 and increases success-rate variability.Adding scene variants to CaP-Agent0 does not improve its performance, and the passage suggests that it struggles to construct generalizable programs without the relational representation.Source: B. Simulation Experiments for… · p. 7

C. Simulation Experiments on LIBERO-Pro

RAPID performs strongly across LIBERO-Pro and real-world nonprehensile tasks, and the conclusion reports generalization to novel scenes through reusable programs.

C. Simulation Experiments on LIBERO-Pro

  • RAPID achieved the highest success rate in five of six LIBERO-Pro task settings, ranking second to ASPIRE on LIBERO-Goal (Pos.).RAPID received no demonstrations and one debug scene, versus ASPIRE's 15; primitives such as drawer opening discovered from one scene were less robust to spatial variation.Source: C. Simulation Experiments on LIBERO-Pro · p. 7

D. Real-World Experiments for Nonprehensile Manipulation

  • On eight real-world nonprehensile tasks, simulation-optimized contact-rich behaviors transferred to a Franka arm and executed effectively, unlike CaP-Agent0's unreliable procedural trajectories.Each task was evaluated in 10 test scenes; the setup used a Franka Research 3 with a parallel gripper and an Intel RealSense L515 camera.Source: Figure 5 · p. 7, D. Real-World Experiments for… · p. 7

VII. CONCLUSION

  • The conclusion reports that object-centric relational programs generalize across object and scene variations, with LIBERO-Pro and real-robot evaluations supporting broader applicability.The programs express primitives as trajectory-optimization programs and compose them with relational constraints.Source: §VII · p. 7
  • Future work will add primitive-learning tools such as reinforcement learning for dexterous hands and scale with advances in coding agents, robot foundation models, and simulation engines.The passage gives deformable objects as an example of a simulation-engine direction.Source: §VII · p. 7
Loading 2609.30249v1…