PaperlayerView exact revision on arXiv

Source-linked AI summary

GaP: A Graph-as-Policy Multi-Agent Self-Learning Harness For Variational Automation Tasks

Kaiyuan Chen, Shuangyu Xie, Letian Fu, Justin Yu, William Pacini, Sandeep Bajamahal, Hudson Kim, Jaimyn Drake, Daehwa Kim, Haoru Xue, Jonathan Francis, Christian Juette, Peter Schaldenbrand, Muhammet Yunus Seker, Ruwan Wickramarachchi, Uksang Yoo, Guanzhi Wang, Adithyavairavan Murali, Balakumar Sundaralingam, S. Shankar Sastry, Spencer Huang, Yuke Zhu, Linxi "Jim" Fan, Ken Goldberg

arXiv:2607.05369v1cs.ROcs.AIcs.CLcs.LG

TL;DR

Variational Automation tasks require reliable robot behavior under greater object and pose variation, while existing coding-agent approaches can be unstructured and difficult to make reliable. GaP generates modular directed robot computation graphs and rehearses them in simulation, achieving significantly higher success rates than baselines across eight VA benchmarks.

  • Problem

    Variational Automation requires reliable robot behavior under object and pose variation, while generalist robots and unstructured coding-agent approaches remain limited for such tasks.

    Source: §1 · p. 2

  • Method

    GaP uses a hierarchical multi-agent harness to generate modular directed computation graphs from atomic robot skills and iteratively rehearse and optimize them in simulation.

    Source: §1 · pp. 2–3, §3 · p. 5

  • Results

    Across 8 open VA benchmarks, GaP achieves high success rates that significantly outperform baselines; in real-world grocery fulfillment, it succeeds in 25/25 trials (100%) versus TipTop’s 8/25.

    Source: §1 · p. 3, §5.3.2 · p. 8

  • Takeaways & Limitations

    GaP supports modular, multi-agent, self-learning robot programming for VA tasks spanning simulation and real-world evaluation.

    Source: §1 · p. 3, §6 · p. 11

    GaP is not yet at industry-level reliability or execution speed, and its benchmarks mainly cover quasi-static pick-and-place tasks rather than deformable objects, dynamic forces, or moving targets.

    Source: §6 · p. 11

Abstract

from arXiv · show

For robots to work reliably in commercial and industrial applications, can recent advances in agentic coding systems combine interpretable robot programming with the open-world adaptability of model-free policies? We focus on "Variational Automation" (VA), a class of tasks that have larger variations in object geometry and pose than fixed automation. Model-free policies often struggle to close the reliability gap for VA tasks, which must be executed persistently and reliably in commercial and industrial applications. Motivated by prior work on Task and Motion Planning (TAMP) and the Robot Operating System (ROS), we introduce Graph-as-Policy (GaP), a multi-agent coding harness that generates directed computation graphs with perception, planning, and control nodes from a Modular Open Robot Skill Library (MORSL). GaP then generates an internal simulation environment to rehearse task instances with different graphs in parallel to iteratively refine the graph structure and parameters to improve success rates and throughput. Evaluation with 8 new open VA task benchmarks, 4 in-simulation and 4 in real-world, suggests that GaP can achieve success rates that significantly outperform baselines. Details, code, and data can be found online: https://graph-robots.github.io/gap

1 Introduction

The paper defines Variational Automation (VA) as persistent robotic work under substantial object variation and introduces Graph-as-Policy (GaP), a structured multi-agent harness for generating and optimizing executable robot computation graphs. GaP is evaluated on eight open VA benchmarks, with results suggesting significantly higher success rates than baselines including π0.5 and MolmoAct2 [7].

Problem

  • Variational Automation tasks require persistent execution across instances with non-trivial variation in object geometry and pose, unlike repetitive fixed automation.Examples include sorting packages, making coffee, and building sandwiches.Source: §1 · p. 2

Motivation

  • Fixed automation offers high reliability and throughput through rigorous human engineering, whereas VA requires substantially more human effort to set up and tune reliably.This motivates methods that reduce the burden of configuring specialized robotic systems.Source: §1 · p. 2

Method

  • GaP represents robot policies as directed computation graphs of modular skill nodes, separating graph generation from testing to constrain agents and reduce hallucination and cheating.The design draws on Task and Motion Planning [32] [33] and the Robot Operating System [1], supporting modularity, reuse, and composability.Source: §1 · pp. 2–3
  • A hierarchical orchestration agent decomposes VA descriptions, skill agents synthesize localized subgraphs, and a self-learning loop simulates and evaluates graphs against success rate and throughput.The resulting graph is executed repeatedly by an edge-based graph interpreter after optimization plateaus.Source: §1 · p. 3

Evaluation

  • GaP achieves high success rates that significantly outperform baselines including π0.5 and MolmoAct2 [7] across eight new open VA benchmarks.The benchmarks cover grocery fulfillment, grocery packing, popcorn making, USB-C insertion, and crate washing, with the first three spanning simulation and real-world settings.Source: §1 · p. 3

Contributions

  • The paper contributes the VA task class, eight benchmarks, GaP, a computation-graph harness, MORSL with 51 initial skills, Isaac-Lab graph rehearsal, and comparative experimental data.The comparisons include TAMP, model-free, and other baselines across the first six benchmarks, plus GaP results on the remaining two.Source: §1 · p. 3

2 Related Work

This section positions Variational Automation between fixed and generalist robotics, then situates GaP among modular robot control, code-generating agents, and self-improving workflows. GaP combines LLM-driven adaptability with an interpretable graph structure and simulation-based iterative refinement.

Variational Automation

  • Variational Automation (VA) persistently handles varying task instances, unlike Fixed Automation’s identical objects and Generalist Robotics’ diverse tasks and environments.VA includes variable object geometry, initial poses, and task arrangements while retaining a persistent task objective.Source: §2 · pp. 3–4

Modular Robot Control Methods

  • Modular robot control builds on sensing, planning, and execution components, with ROS graphs and TAMP coordinating dependencies, task sequences, and geometric feasibility.These approaches provide explicit structure and constraint handling for reliable robot execution.Source: §2 · p. 4
  • Code-as-Policy uses LLM agents to generate robot control code as an alternative to manually coded model-based methods and purely end-to-end model-free policies [28].Extensions include CaP-X [30], GRAPPA, Maestro [40], and integrations with TAMP such as TiPToP [33].Source: §2 · p. 4
  • GaP tempers Code-as-Policy flexibility with a graph structure that preserves interpretable organization while harnessing pretrained LLM coding agents for open-world adaptability in persistent VA.This graph-based design addresses the need to reduce human effort in setting up VA systems.Source: §2 · p. 4

Self-Improving Agentic Workflows For Robotic Control

  • Self-improving agentic workflows use LLMs to synthesize and refine solutions through reusable skills, failure reasoning, and multi-agent prompt optimization [53] [54] [55] [56].For robotics, CaP-X [30] uses visual feedback but may hallucinate and cannot handle geometric or numerical motion feasibility; GaP instead iteratively improves multi-agent-generated computation graphs through simulation experiments.Source: §2 · p. 4

3 Problem Formulation

The paper defines Variational Automation tasks over a known workcell and bounded object-pose envelope, formalized by task, state, belief, and reward spaces. GaP represents the policy as a directed execution graph and seeks the graph maximizing expected performance across task instances.

Assumptions

  • VA assumes a fixed, known workcell, robot, and sensor configuration, with known ranges of possible objects and initial poses.These assumptions define a bounded operating envelope rather than oracle information, enabling object models, calibrated sensors, and reusable skills when available.Source: §3 · p. 4

Variational Automation Task Class

  • A VA task is formalized as T = ⟨L, E, R, O, X, B, J⟩, including language, environment, robot configuration, objects, state, belief, and reward.Task instances sample objects from O and states from B; J combines success and throughput using weighting factors.Source: §3 · pp. 4–5

Policy Representation via Directed Execution Graphs

  • GaP represents π(a | x, T) as a directed computation graph whose typed nodes perform modular manipulation, perception, or code operations, while edges encode data flow, control, and execution order.Independent branches may execute concurrently, and natural-language skills specify how LLM agents configure and compose atomic nodes for subtasks.Source: §3 · p. 5

Problem Definition

  • GaP must synthesize G* that generalizes across B and, from real-time multi-view observations, induces a closed-loop policy π_G(a | I) achieving the language-specified goal.The optimized graph maximizes E_xi∼B[J(π(a | I, G))] and is sent to an edge-based interpreter for repeated robot execution.Source: §3 · p. 5

4 Graph-as-Policy

GaP converts natural-language automation objectives into computation graphs of robotic skills, then rehearses and refines those graphs in internal simulation before physical deployment. Its MORSL library specifies skill interfaces and prerequisites, while parallel variational rollouts guide graph updates.

4 Graph-as-Policy

  • GaP partitions a natural-language objective into semantic segments and maps them through MORSL into an initial computation graph for execution.The optimized graph is sent to an external interpreter for repeated execution on the physical robot.Source: §4 · pp. 5–6

4.1 Modular Open Robot Skill Library (MORSL)

  • MORSL provides 51 agentic skills whose declared inputs, outputs, parameters, and preconditions determine when skills are invoked and how they connect.The catalog spans perception, grasp planning, motion planning, vision utilities, verification, and control primitives, including SAM2 [59], Grounding DINO [61], and cuRobo.Source: §4.1 · p. 6

4.2 Self-Learning through Internal Simulation Rehearsal

  • GaP samples task instances from a belief space, rehearses graphs in parallel Isaac simulations, analyzes failures, and uses agentic graph updates to modify topology and parameters.Updates can swap equivalent nodes, change edges, or revise code parameters until performance plateaus or failure reasons terminate optimization.Source: §4.2 · p. 6

5 Experiments

Across eight variational-automation benchmarks spanning simulation and real-world settings, GaP improves robustness through graph-based perception, planning, and control, while self-learning adapts policies to task failures. GaP also transfers to ROS-integrated and coordinated bimanual tasks, approaching expert-engineered performance.

5.1 Simulation and Real-World Variational Automation Benchmarks

  • GaP evaluates eight VA benchmarks—four simulated and four real—including grocery fulfillment, grocery packing, popcorn preparation, cable insertion, and cooperative crate washing.The benchmarks vary object positions, swaps, permutations, task goals, orientations, and multi-object or bimanual interaction conditions.Source: §5.1 · pp. 6–7

5.2 Baselines

  • The baseline study compares GaP with CaP-X, π0.5, MolmoAct2, and TipTop, using 100-instance trials for simulated grocery fulfillment and packing.GaP is combined with VLA policies by first navigating the wrist camera above the target object; self-learning is not used for these benchmarks because the initial graph performs highly.Source: §5.2 · p. 7, Table 1 · p. 8

5.3.1 Simulation Results

  • Baseline VLA success drops to 0.20 on LIBERO-PRO, whereas GaP maintains robust success across positional variations and TipTop often cannot generate feasible plans.GaP also improves π0.5 and MolmoAct2 performance by more than twofold by centering the wrist camera over the target before VLA execution.Source: §5.3.1 · p. 8, Table 1 · p. 8

5.3.2 Real Physical Experiment Results

  • GaP achieves 25/25 (100%) real-world grocery-order success versus TipTop’s 8/25, while generating oriented bounding boxes in 14.1 seconds and transporting items in 36.4 seconds.TipTop fails on cubic objects, tall baskets, and varied orientations because its combined planners cannot find feasible motion plans.Source: §5.3.2 · p. 8, Table 2 · p. 8

5.4 Ablation Studies

  • Ablating graph structure or specialized agents collapses success to zero, while full GaP achieves 0.93–0.99 success rates on grocery fulfillment and packing.Graph validation prevents malformed edges and improves connectivity before runtime execution.Source: §5.4 · pp. 8–9

5.5 Benchmark III a, b: Make Popcorn in sim and real

  • Self-learning raises Make Popcorn performance from an initial 33% to 94% in simulation and 90% (18/20) in real trials.It replaces failed grasping, prompts localization of the pan handle, and tunes placement offsets; physical failures were one IK error and one accumulated-kinematic-error misgrasp.Source: §5.5 · p. 9, Figure 2 · p. 9

5.6 Benchmark IV: Insert Cables (Real)

  • GaP achieves a 0.93 success rate (121/130) across cable-insertion trials and generates correct workflows for all five text prompts.Its ROS graph uses modular align, touch, insert, and extract nodes to handle positional, orientation, and long-horizon goal variations.Source: §5.6 · p. 9, Table 3 · p. 10, Figure 4 · p. 10

5.7 Benchmark V: Wash Crates (Simulation)

  • GaP achieves 0.953 (143/150) success versus 0.987 (148/150) for an expert graph, with throughput of 18.33 versus 19.33 successes per hour.Average cycle times are 179.13 s for GaP and 176.47 s for the hand-engineered graph over the 3-hour run.Source: §5.7 · p. 10, Table 4 · p. 10

6 Conclusion · Appendix · A More Related Work

GaP bridges GOFE and model-free VLA policies through graph-structured, self-improving agentic robotics, while remaining below industry reliability and throughput standards. The related work situates GaP among generalist robotics and self-improving LLM workflows that generate and refine reusable skills and task plans.

6 Conclusion

  • GaP’s graph-structured agentic robotics enables modularity, multi-agent integration, and self-learning as robot applications and skill libraries become more complex.Source: §6 · p. 11
  • GaP improves success rates over baselines, but its execution reliability has not yet reached industry levels and requires additional self-learning and parameter tuning.Source: §6 · p. 11
  • GaP remains below the industry standard of 500 units per hour, or 7 seconds per instance, partly because of VLM inference requests and IK motion-planning time.Source: §6 · p. 11
  • The eight VA benchmarks primarily cover quasi-static pick-and-place, with only Cable Insertion (IV) requiring force sensing, leaving dynamic and deformable-object tasks for future work.Source: §6 · p. 11

A More Related Work

  • Generalist Robotics varies tasks, environments, object geometries, and initial poses substantially more than fixed automation, motivating distinct approaches beyond single generalist model-free VLA policies [35] [4] [5].Source: A More Related Work · p. 17
  • LLM-based agentic workflows support dynamic coding, software integration, API orchestration, and autonomous synthesis of solutions for open-ended tasks [16] [48] [49] [52].Source: A More Related Work · p. 17
  • Voyager [53] iteratively writes, refines, and executes Minecraft skills, while later work optimizes agentic systems through language-based failure reasoning and multi-agent prompt optimization [54] [55] [56].Source: A More Related Work · p. 17
  • Unlike CaP-X [30], whose VLM feedback can hallucinate and cannot assess geometric or numerical motion feasibility, GaP uses multiple LLM agents and simulation experiments to improve robot computation graphs.Source: A More Related Work · p. 17

B Integrating ROS with GaP in Cable Insertion Benchmark

GaP integrates ROS nodes through an intermediate skill interface that executes cable-insertion workflows sequentially and returns status to the Behavior Graph. Across 130 trials spanning five prompt types, it achieved a 0.93 success rate while supporting atomic-skill composition and long-horizon insertion graphs.

ROS integration

  • GaP exposes align-to-port, touch-port, insert, and extract through an intermediate skill interface that sequentially triggers ROS nodes and returns execution status to the Behavior Graph.A temporary orchestrator node communicates with each target ROS node until receiving a success, failure, or data signal.Source: B Integrating ROS with GaP in Cable… · p. 17

Graph generation

  • GaP composes atomic ROS skills into compatible computation graphs and subgraphs for extended workflows, including long-horizon insertion into even-numbered ports.The generated graph is shown alongside a hand-engineered ROS graph using traditional nodes and topics.Source: Figure B.1 · p. 17, B Integrating ROS with GaP in Cable… · p. 18

Benchmark results

  • 0.93 success rate (121/130) across five prompt types shows GaP successfully completed cable insertion while generating graphs comparable to an expert-engineered ROS baseline.The prompts covered individual ports, ascending and descending sequences, and odd- or even-indexed ports.Source: B Integrating ROS with GaP in Cable… · pp. 17–18

Failure modes

  • Failures arise when vision-based candidates fall outside the 1 × 1 cm2 contact-search area, depth errors cause partial insertion, or lighting and shadows disrupt port detection.The fallback probes within the search zone in 2 mm steps.Source: B Integrating ROS with GaP in Cable… · p. 18

C Sample Generated Graphs … C.4 Wash Crates

The generated graphs use service-call scripts with explicit success and failure routes across four automation tasks. Their task-specific nodes cover perception, manipulation, policy execution, navigation, alignment, and crate respawning.

C.1 Fulfill Grocery Orders

  • C.1’s grocery-order graph routes service-call success and failure outcomes while combining perception, object filtering, approach, waypoint motion, and release nodes.The graph includes DINO-VLM perception, FilterAndComputeOB, approach_above, waypoint_move, and descend_release operations.Source: §C.1 · p. 18

C Sample Generated Graphs

  • Across the grocery-order examples, generated graphs expose task execution through modular node sequences rather than a single undifferentiated operation.The examples show distinct perception, filtering, motion, release, and policy nodes across C.1–C.3.Source: §C.1 · p. 18, §C.3 · p. 19

C.2 Pack Grocery Items

  • The cited service-call script subgraph includes explicit success and failure routes and a release operation identified as descend_release, described as placed.Source: §C.2 · p. 18

C.3 Fulfill Grocery Orders with VLA Policy

  • C.3 integrates a VLA policy into the grocery-order graph, using approach_above_tar before the run_policy node succeeds.The service-call script retains explicit success and failure routing around the policy-based execution.Source: §C.3 · p. 19

C.4 Wash Crates

  • C.4’s crate-washing graph uses explicit service-call success and failure routes for its task execution.The graph structure is presented as a generated service-call script subgraph.Source: §C.4 · p. 19
  • In C.4, go_mid reaches a target pose and go_aligned achieves alignment through GoToPoseBoth operations.These are reported as successful result nodes in the crate-washing graph.Source: §C.4 · p. 19
  • C.4 also includes a respawn_top_crate operation, indicating explicit handling of the top crate within the generated graph.The node is identified as RespawnTopCrate.Source: §C.4 · p. 19

C.5 Make Popcorn … bimanual crate lift

The merged section illustrates a popcorn workflow assembled from routed perception and motion skills, then defines MORSL’s typed skill composition and reusable perception, grasping, transport, and bimanual primitives.

C.5 Make Popcorn

  • The popcorn workflow uses success/failure service-call routes, Gemini perception, cart alignment, in-place rotation, waypoint motion, and release placement.The listed graph nodes include cart_align, rotate_90, move_above, and release; an additional lift_and_align skill is also present.Source: §C.5 · p. 20

D MORSL Library

  • MORSL represents skills through shared metadata and typed inputs/outputs, with composite subgraphs, atomic classes, and primitive gRPC methods composing through common types.The library’s component hierarchy supports reusable composite, atomic, and primitive skills, including model invocations and geometric routines.Source: D MORSL Library · p. 20

perception single / perception multi / perception subpart

  • Perception skills range from fast single-path detection to multi-detector disambiguation and hierarchical subpart localization for small affordances.The single detector fuses segmentation and depth into 3D geometry, the multi detector resolves disagreement among three detectors, and the subpart detector crops around a parent object.Source: perception single · p. 21, perception multi · p. 21, perception subpart · p. 21

track object

  • The tracking skill seeds a text-prompted tracker on the first frame, advances it through observations until termination, and frees tracker state afterward.Its output is a stream of tracked-mask snapshots.Source: track object · p. 21

grasp curobo obb / grasp moe / grasp short axis / grasp direct ik

  • Grasping primitives cover collision-aware OBB planning, mixture-of-experts candidate selection, geometry-locked short-axis alignment, and planner-free direct inverse-kinematics grasping.These variants target centroid grasps, off-center affordances, elongated objects, or uncluttered scenes without a motion planner.Source: grasp curobo obb · p. 21, grasp moe · p. 21, grasp short axis · p. 22

transport to drop

  • The transport skill moves a held object above a destination container and releases it, optionally grounding a named sub-region with a VLM.The optional localization step supports destinations such as “the left compartment.”Source: transport to drop · p. 22

bimanual crate lift

  • The bimanual crate-lift skill side-grasps handle bars with both arms and raises the crate synchronously through a fixed open-to-settle phase order.It uses baked waypoints and outputs completion plus final left- and right-arm world positions.Source: bimanual crate lift · p. 22

D.2 Primitive skills (one per gRPC method) … geometry svc.BuildWorldConfig

The primitive-skill library exposes modular perception, grasping, planning, geometry, tracking, and visual-question-answering methods through gRPC interfaces. Together, these skills transform images, masks, point clouds, poses, and world observations into detections, trajectories, grasps, and collision-aware planning configurations.

owlvit.Detect; sam3.SegmentText; sam3.SegmentPoint; sam3.SegmentBox; sam2.SegmentBox; sam2.SegmentPoint

  • Open-vocabulary detection returns labeled bounding boxes and scores from text queries, while text-, point-, and box-prompted segmentation returns masks, scores, and boxes.The library includes OWL-ViT detection plus SAM3 and lighter SAM2 segmentation variants, including optional foreground-point refinement and mask limits.Source: owlvit.Detect · p. 22, sam3.SegmentText · p. 23, sam3.SegmentPoint · p. 23

sam3 tracker.InitTracker; sam3 tracker.UpdateTracker

  • Tracker initialization seeds a stateful session from text, box, or point prompts, and tracker updates propagate masks, boxes, confidence, and object presence across frames.Initialization also returns a tracker identifier, initial mask, initial box, score, and presence status.Source: sam3 tracker.InitTracker · p. 23, sam3 tracker.UpdateTracker · p. 23

molmo.PointPrompt; molmo.Query; molmo.QueryYesNo; vlm.Query; vlm.QueryYesNo

  • Molmo and configurable VLM skills provide object-point prompts, free-form visual answers, and boolean visual answers from optional single- or multiview images.Point prompting returns pixel coordinates and a found flag; question answering returns either text or a boolean answer.Source: molmo.PointPrompt · p. 24, molmo.Query · p. 24, molmo.QueryYesNo · p. 24

graspgen.PlanFromPointCloud; graspgen.ScoreGrasps; graspnet.PlanFromDepth; graspnet.PlanFromPointCloud; m2t2.PlanFromPointCloud

curobo.PlanToGraspPoses; curobo.PlanWithGraspedObject; curobo.SolveIK; curobo.PlanToPose; curobo.ValidateJointTrajectoryRobot; pyroki.PlanLinear

curobo.ValidateJointTrajectoryGrasped; curobo.ValidateJointTrajectoryRobot

geometry svc.FilterAndComputeOBB; geometry svc.TopDownGraspCandidates; geometry svc.SelectTopDownGrasp; geometry svc.FrontGraspFromOBB; geometry svc.DepthToPointCloud; geometry svc.MaskToWorldPoints

geometry svc.PixelToWorldPoint; geometry svc.BuildWorldConfig

geometry svc.ComputeDropPosition

svc.ComputeDropPosition computes a drop point above a container while accounting for clearance and object height.

geometry svc.ComputeDropPosition

  • The service takes a container oriented bounding box, clearance (0.05), and object z extent, then outputs a Vec3 drop point.Its purpose is to determine a drop position above the container while accounting for clearance and object height.Source: geometry svc.ComputeDropPosition · p. 28

geometry svc.ComputeXYDistance … E.1 Pseudo-code

The appendix specifies MORSL geometry, robot-control, observation, and simulation services, alongside rehearsal-based graph-optimization pseudocode. The services expose motion, sensing, simulation, evaluation, and recording operations, while optimization samples scenes, rehearses graphs in parallel, analyzes failures, and updates the graph.

geometry svc.ComputeXYDistance / geometry svc.RotateQuatZ90

robot control.GoToPoseBoth / robot control.MoveToJointsBoth

robot control.ApplyPolicyAction / observation.GetObservation

sim bridge.Init / sim bridge.StepOnce / sim bridge.MoveToJointsBlocking

sim bridge.CheckTaskCompletion / sim bridge.SaveVideo

  • Simulation evaluation queries reward, task completion, and sub-goal completion rate, while video services encode captured frames as MP4 files.CheckTaskCompletion requires no inputs; SaveVideo accepts an output path, clearing option, and frame rate, returning success, file path, and frame count.Source: sim bridge.CheckTaskCompletion · p. 31, sim bridge.SaveVideo · p. 31

E.1 Pseudo-code

  • Rehearsal-based graph optimization initializes a language-derived graph, samples scene variations, performs parallel rollouts, analyzes failures, and updates the graph over iterations.The algorithm takes task specification T, iteration count M, and parallel rollout count N, returning an optimized task graph G∗.Source: §E.1 · p. 31

E.2 Sample Feedback · F Sample LLM Prompts and Outputs

This section presents sample feedback, prompts, and generated code for behavior and skill agents in a robotics workflow harness. It specifies structured graph construction, perception subgraphs, validation, error handling, and postcondition checks to support iterative refinement and valid workflow implementations.

E.2 Sample Feedback

  • The feedback records simulated-state and contact differences after each phase so the LLM can infer failures and revise graph parameters.The suggested update revises the placement target and release pose while improving grasp robustness for failed environments.Source: §E.2 · p. 32
  • Terminal coverage was 0.00 in iteration 1 because perception succeeded, but grasp instability and incorrect pan placement caused downstream failures.Perception succeeded in all sampled environments; grasp succeeded in 25/30 and placement in 17/30, with placement requiring at least 70% burner-footprint coverage.Source: §E.2 · p. 32

F.1 Behavior Agent Prompt

  • The behavior agent generates only the top-level workflow topology, while a universal subgraph agent separately fills each subgraph’s internal state machine.The top-level output specifies subgraph nodes, edges, conditional routing, inputs/outputs, exit values, and skill choices.Source: §F.1 · p. 32
  • The generated Python scaffold must bind a module-level spec, declare subgraphs and end nodes, wire conditional exits, and satisfy strict validation rules.Required checks include matching proto-typed inputs and outputs, mandatory exit_success_values and on_error arguments, reachable end nodes, and valid top-level routing.Source: §F.1 · pp. 33–34
  • Optional stage tags classify grasp, transport, and place subgraphs so iteration-1 refinement can compute per-stage pass rates and select canonical swaps.Unstaged perception or staging subgraphs are excluded from stage rollups, while staged pick-and-place subgraphs guide mechanical refinement.Source: §F.1 · p. 34
  • The behavior prompt offers single-path perception variants and a pick-and-place objective that locates, grasps, transports, and stably places an object.The perception pipeline may use Molmo alone or DINO, VLM, SAM3, and depth-to-3D fusion before producing a clean oriented bounding box and mask.Source: §F.1 · pp. 37–38

F.2 Behavior Agent Output

  • The behavior output instantiates perception, grasp, and transport subgraphs, wiring success-dependent transitions to completion and failures to abort.It declares target and container perception, grasp inputs and outputs, transport inputs, and conditional edges for found, grasped, and placed states.Source: §F.2 · pp. 38–39

F.3 Skill Agent Prompt

  • The skill prompt requires a minimal validated subgraph that consumes declared inputs, produces declared outputs, exposes success exits, and routes failure through a single raising error path.Postconditions are deferred to a checkpoint agent, while genuine mid-flow preconditions use raising script guards; streaming skills are source nodes without outgoing edges.Source: §F.3 · pp. 40–41
  • The universal agent reads each skill’s SKILL.md for node-flow guidance and emits tool, script, noop, or router nodes, replacing legacy domain-specific subagent classes and node types.Router nodes support static routing or dynamic fan-out through Send dictionaries, while streaming sources publish values consumed through $ref snapshots.Source: §F.3 · pp. 43, 46
  • Every subgraph must declare 1–6 checkpoints, including at least one validate=True postcondition with a non-empty rationale, because rehearsal feedback drives refinement.Checkpoints are evaluated against privileged World snapshots at subgraph exit; validate=False probes add diagnostics without blocking downstream evaluation.Source: §F.3 · p. 44
  • Checkpoint predicates must use exact scene-spec object IDs and privileged quantities rather than camera-derived features, avoiding circular validation and runtime BodyNotFoundError failures.The prompt gives w.body("alphabet soup").is_in(w.body("basket")) as the required form when those are the canonical scene-spec IDs.Source: §F.3 · p. 44
  • Workflow v3 uses top-level subgraph orchestration, declared nodes, static or conditional edges, explicit exits, and a single raising-based on_error failure exit.The validator rejects undeclared or unreachable nodes, invalid streaming edges, malformed routing, unknown references, and on_error used as a mapping target.Source: §F.3 · pp. 45, 47
  • The prompt specifies protobuf module and field conventions, including geometry types in geometry_pb2 and distinct Se3Pose.rotation versus OrientedBoundingBox.orientation fields.It identifies importing OrientedBoundingBox from sensor_pb2 as a leading source of script-load ImportErrors.Source: §F.3 · pp. 48–49
  • The single-path pipeline runs per camera, merges results through KD-tree intersection, and supports uncluttered scenes with visually distinct targets.It is intended for platforms deploying DINO, VLM, SAM3, and Geometry, and should be replaced by perception_multi for cluttered scenes with similar distractors.Source: §F.3 · p. 49
  • The prescribed subgraph is observe -> perceive -> filter_obb, using observation.GetObservation, perceive_dino_vlm.py, and geometry_svc.FilterAndComputeOBB.The perception script returns found, cloud, mask, and score, while the geometry service returns a bare OrientedBoundingBox.Source: §F.3 · pp. 49–50
  • The exit wiring must be linear from filter_obb through found to END, with set_on_error("not_found") handling perception failures instead of conditional edges.The not_found path routes to abort in normal pick-and-place or done in clean-all-items loops.Source: §F.3 · pp. 49–51
  • Every perception subgraph must emit <name>_obb, <name>_mask, and <name>_cloud so downstream skills such as grasp_moe can connect without anticipating their needs.The OBB references filter_obb directly, whereas mask and cloud reference fields of the perception output dictionary.Source: §F.3 · pp. 50–51
  • The concrete container_sg example locates a target and requires found and not_found end states, with outputs container_obb, container_mask, and container_cloud.A not_found result means the target is not visible in any view.Source: §F.3 · p. 51

G Sample Generation Outputs

This section illustrates sample graph-generation code using subgraph utilities, reference bindings, and explicit execution wiring. The example also includes success, output-binding, and error-handling hooks.

G Sample Generation Outputs

Related papers

Loading 2607.05369v1…