PaperlayerView exact revision on arXiv

Source-linked AI summary

CaP-X: A Framework for Benchmarking and Improving Coding Agents for Robot Manipulation

Letian Fu, Justin Yu, Karim El-Refai, Ethan Kou, Haoru Xue, Huang Huang, Wenli Xiao, Guanzhi Wang, Dantong Niu, Fei-Fei Li, Guanya Shi, Jiajun Wu, Shankar Sastry, Yuke Zhu, Ken Goldberg, Linxi "Jim" Fan

arXiv:2603.22435v2cs.ROcs.AI

TL;DR

Coding agents for robot control face challenges generalizing across environments, embodiments, and long-horizon tasks. CaP-X benchmarks and improves embodied coding agents, finding that abstractions and multi-turn execution feedback improve performance while raw visual interleaving can degrade it.

  • Problem

    Coding agents for robot control struggle to generalize across environmental changes, robot embodiments, and long-horizon tasks without additional data collection and retraining.

    Source: §1 · p. 2

  • Method

    CaP-X combines an interactive robot-control environment, controlled benchmarking across abstraction and interaction levels, and agentic and reinforcement-learning improvements.

    Source: §1 · p. 2, §2 · p. 3, §3 · p. 4

  • Results

    Performance improves with higher-level primitive abstractions and multi-turn execution feedback, while frontier models retain a gap versus human experts in single-turn evaluation.

    Source: §3.3 · p. 5

  • Takeaways & Limitations

    CaP-X provides a testbed for evaluating and advancing general-purpose embodied intelligence while motivating primitive-level evaluation and agent-recovered structure.

    Source: §3.3 · p. 5, §7 · p. 9

    Interleaving raw RGB observations at each turn degrades performance relative to text-only execution feedback, possibly because coding agents struggle to integrate software and visual reasoning.

    Source: §3.3 · p. 5

Abstract

from arXiv · show

"Code-as-Policy" considers how executable code can complement data-intensive Vision-Language-Action (VLA) methods, yet their effectiveness as autonomous controllers for embodied manipulation remains underexplored. We present CaP-X, an open-access framework for systematically studying Code-as-Policy agents in robot manipulation. At its core is CaP-Gym, an interactive environment in which agents control robots by synthesizing and executing programs that compose perception and control primitives. Building on this foundation, CaP-Bench evaluates frontier language and vision-language models across varying levels of abstraction, interaction, and perceptual grounding. Across 12 models, CaP-Bench reveals a consistent trend: performance improves with human-crafted abstractions but degrades as these priors are removed, exposing a dependence on designer scaffolding. At the same time, we observe that this gap can be mitigated through scaling agentic test-time computation--through multi-turn interaction, structured execution feedback, visual differencing, automatic skill synthesis, and ensembled reasoning--substantially improves robustness even when agents operate over low-level primitives. These findings allow us to derive CaP-Agent0, a training-free framework that recovers human-level reliability on several manipulation tasks in simulation and on real embodiments. We further introduce CaP-RL, showing reinforcement learning with verifiable rewards improves success rates and transfers from sim2real with minimal gap. Together, CaP-X provides a principled, open-access platform for advancing embodied coding agents.

1. Introduction

CaP-X studies Code-as-Policy agents for robot manipulation, addressing the limitations of classical, human-engineered control and data-dependent VLA models. It introduces CaP-Gym, CaP-Bench, CaP-Agent0, and CaP-RL to evaluate and improve coding-based robot control across abstraction, interaction, grounding, and embodiment.

Motivation

  • Classical robot control offers interpretability, geometric precision, and robustness through explicit structure, but requires costly human engineering and often produces task-specific solutions that generalize poorly.These paradigms manually decompose goals, compose perception and control modules, and encode failure handling and edge cases.Source: §1 · p. 1
  • VLA models achieve impressive contact-rich manipulation from large-scale visuomotor data, yet struggle with interpretability, environmental changes, new embodiments, and long-horizon tasks without retraining.Their limitations arise from dependence on training data and model design.Source: §1 · pp. 1–2
  • Code-as-Policy agents can synthesize executable mid- to low-level logic, but prior human-tuned primitives leave unclear how much performance comes from agents versus imposed scaffolding (Liang et al., 2023; Singh et al., 2023).CaP-X therefore studies whether iterative debugging, skill synthesis, ensembled reasoning, and multimodal grounding compensate for reduced scaffolding.Source: §1 · p. 2

CaP-Gym

  • CaP-Gym provides an interactive environment for generating and executing programs over shared perception and control primitives across 187 simulation tasks compatible with physical robots.It integrates RoboSuite (Zhu et al., 2020), LIBERO-PRO (Liu et al., 2023b), and BEHAVIOR (Li et al., 2024).Source: §1 · p. 2

CaP-Bench

  • CaP-Bench evaluates 12 open- and closed-source language and vision-language models across abstraction level, temporal interaction, and perceptual grounding using seven CaP-Gym environments.It compares human-crafted macros with atomic primitives, single-turn with multi-turn generation, and visual-feedback modalities.Source: §1 · p. 2

CaP-Agent0 and CaP-RL

  • CaP-Agent0 combines multi-turn interaction, text-based visual grounding, automatic task-agnostic skill synthesis, and parallelized multi-model generation, while CaP-RL improves success through on-policy rewards and transfers synthesized programs to real robots.CaP-Agent0 reaches human-comparable or sometimes higher performance on CaP-Bench tasks without task-specific training data.Source: §1 · p. 2, Figure 1 · p. 2

1. CaP-Gym, a unified suite of interactive robot coding

CaP-Gym provides environments spanning tabletop, bimanual, and mobile manipulation for evaluating and training code-generating multimodal embodied agents. The framework includes CaP-Bench for systematic performance measurement and CaP-Agent0, a training-free agentic harness.

CaP-Gym

  • CaP-Gym spans tabletop, bimanual, and mobile manipulation tasks for evaluating and training code-generating multimodal embodied agents.Source: §1 · p. 3

CaP-Bench

  • CaP-Bench measures robot-control performance across tasks and varying levels of primitives and modalities.Source: §1 · p. 3

CaP-Agent0

  • CaP-Agent0 is a training-free agentic harness combining multi-turn visual differencing, ensembled reasoning, and automatic skill-library synthesis.Source: §1 · p. 3

4. CaP-RL,

CaP-RL applies reinforcement learning to the coding agent using environment rewards.

4. CaP-RL

  • CaP-RL trains the coding agent with reinforcement learning based on environment rewards.Source: §4 · p. 3

2. CaP-Gym

CaP-Gym is a hierarchical, REPL-based control framework that connects coding agents to simulated or real robot tasks through stateful program execution. Its modular perception and control primitives expose structured observations and task-oriented actions while delegating execution feasibility to specialized services and controllers.

2. CaP-Gym

  • CaP-Gym connects a stateful Code Executor loop with simulator or real-world dynamics through Gymnasium and a REPL interaction model (Brockman et al., 2016).Each turn supplies observations, executes an agent-generated Python program to completion, and permits multiple primitive calls with internal simulator or controller updates.Source: §2 · p. 3

2.1. Low-Level Perception and Control Primitives

  • Modular perception primitives convert raw sensor data into structured semantic objects using tools including SAM3 (Carion et al., 2025), Molmo 2 (Clark et al., 2026), OpenCV, and Open3D (Zhou et al., 2018).Computationally intensive primitives are stateless services, enabling high-throughput parallel evaluation.Source: §2.1 · p. 3
  • Control primitives let agents reason in task-oriented Cartesian space while motion planners or inverse-kinematics solvers such as PyRoki (Kim et al., 2025) handle feasibility.These controllers support collision checking, reachability constraints, and action-space transformations instead of requiring direct joint-space commands.Source: §2.1 · p. 3

3. CaP-Bench: Evaluating Frontier Models

CaP-Bench evaluates 12 vision-language and language models across controlled manipulation tasks and tiers that vary abstraction, interaction, and perceptual grounding. Results show that human abstractions improve performance, while multi-turn feedback and visual differencing recover robustness over low-level primitives.

3. CaP-Bench: Evaluating Frontier Models

  • CaP-Bench evaluates 12 models on seven controlled manipulation tasks, using eight tiers that independently vary primitive abstraction, interaction, and grounding.The core suite uses 100 trials per tier, while the full CaP-Gym release contains 187 tasks for broader evaluation.Source: §3 · pp. 3–4

3.1. Single-Turn Benchmarks (S1-S4)

  • Single-turn results show that high-level human-designed primitives improve success by reducing the search space, whereas low-level tiers expose code-synthesis difficulty and enable more expressive behaviors.S1 uses privileged simulation state, S2 uses perception from raw RGB-D, S3 includes low-level usage examples, and S4 removes those examples.Source: §3.1 · p. 4, §3.3 · p. 5

3.3. Discussion

  • Human-written programs achieve 88.5% average single-turn success, yet no frontier model matches them in zero-shot Pass@1 evaluation.The comparison uses identical environments and perception-control primitives; closed-source and newer models generally outperform open-source and older alternatives.Source: §3.3 · p. 5

3.2. Multi-turn Benchmarks (M1-M4)

  • Execution traces improve performance across models, raw RGB interleaving degrades it, and visual differencing into text outperforms both direct images and execution-only feedback.M1 provides stdout/stderr traces, M2 supplies RGB observations, and M3 describes scene changes and task completion through a vision-language module.Source: §3.2 · pp. 4–5, §3.3 · pp. 5–6

3.3. Discussion

  • Low-level agents with multi-turn feedback can surpass high-level single-turn performance and reach parity with high-level multi-turn agents, supporting runtime scaling through reasoning and self-correction.Across multi-turn settings, coding and physical execution failures are often recoverable through iterative interaction and explicit state verification.Source: §3.3 · p. 6

4. CaP-Agent0: An Agentic Framework for Robot Control

CaP-Agent0 is a training-free agentic framework that combines visual differencing, synthesized persistent skills, and parallel reasoning to improve low-level robot control. It achieves expert-level reliability across simulation, extended benchmarks, real-world tasks, and reinforcement-learning transfer.

4. CaP-Agent0: An Agentic Framework for Robot Control

  • CaP-Agent0 integrates multi-turn visual differencing, an automatically synthesized persistent library of 9 task-agnostic primitives, and parallel candidate reasoning.Visual differencing structures observations as text; the library reuses successful low-level logic; parallel queries are synthesized into final code.Source: §4 · pp. 6–7

4.1. CaP-Agent0 Performance on CaP-Bench

  • Across 100 trials per task, CaP-Agent0 significantly outperforms single-turn baselines and matches or exceeds human-written programs on 4 of 7 CaP-Bench tasks.It uses low-level primitives while integrating visual differencing, a self-synthesized skill library, and parallel reasoning.Source: §4.1 · p. 7, Figure 8 · p. 7

4.2. CaP-Bench++

  • CaP-Agent0, despite being training-free, matches or exceeds post-trained VLA performance on LIBERO-PRO and remains robust under instruction perturbations.It achieves performance comparable to π0.5 under initial-position perturbations, while VLAs perform poorly under task perturbations.Source: §4.2 · p. 8, Table 2 · p. 8
  • On BEHAVIOR’s long-horizon mobile manipulation tasks, CaP-Agent0 substantially improves navigation and task success by repositioning for visibility, adapting search, and resampling grasps after disturbances.It achieves significantly higher task success than S3 and similar performance to the human expert on soda-can pickup.Source: §4.2 · p. 8

4.3. Let the Agent Experience the Real World

  • In zero-shot real-world tasks, CaP-Agent0 controls Franka Panda and AgiBot G1 robots without major cross-embodiment modifications, performing closed-loop search and embodied reasoning.Optional human feedback can correct behavior between turns.Source: §4.3 · p. 8
  • Simulation post-training with GRPO improves code compilation and strategic robustness, producing substantial gains over the base model under noisy S2 perception.CaP-Gym enables on-policy reinforcement learning with verifiable rewards directly on coding agents.Source: §4.3 · p. 9, Table 4 · p. 9
  • 84% cube-lifting and 76% cube-stacking success demonstrate that CaP-RL strategies transfer from simulation to a Franka Emika robot despite real-world perception noise.The transferable action space is composed of shared perception and control tools rather than raw visual-to-motor mappings.Source: §4.3 · p. 9, Table 4 · p. 9

6. Related Work

Related work spans executable code for robot control, structured planning, self-refinement, and inference-time sampling, but commonly depends on high-level human-crafted APIs. CaP-X addresses this gap through primitive-level evaluation, direct language-model fine-tuning with RLVR, and benchmarks combining executable control code with embodied perception across abstraction tiers.

Code as Policies

Skill Synthesis and RL with LLM-Generated Code

Benchmarks for robotics and embodied agents

7. Conclusion … B.2. Tools

CaP-X unifies benchmarking and improving coding agents for robot control, while its real-world CaP-Agent0 deployment combines interactive web-based supervision with perception, reasoning, and control tools. The paper identifies contact-rich limitations and future directions including hybrid CaP-VLA policies, stronger grounding, improved search, and expanded environments.

7. Conclusion

  • CaP-X unifies CaP-Gym, CaP-Bench, CaP-Agent0, and CaP-RL to study robot control across abstraction levels, interaction modes, and learning paradigms.It frames robot control as machine intelligence involving agent design, inference-time computation, perception, and control.Source: §7 · p. 9

A. Future Works

  • Programmatic control performs well on long-horizon reasoning tasks but remains brittle for contact-rich behaviors requiring tight visual servoing and continuous feedback.Future work includes hybrid CaP-VLA policies, embodiment-aware planning, improved visual grounding, test-time search, agent or prompt optimization, optimization-based control, and broader CaP-Gym environments.Source: A. Future Works · p. 17

B. Interactive Real-World Setup

  • CaP-Agent0 enables an AgiBot G1 to zero-shot complete novel real-world tasks through a chat-based interface supporting task proposals, stepwise tool-use inspection, and user feedback.The interactive setup centers on multi-turn improvement between the user and robot system.Source: B. Interactive Real-World Setup · p. 17

B.1. User Interface

  • A Viser 3D visualization displays the live robot URDF, trajectories, and depth pointcloud.This visualization is integrated into the web UI.Source: §B.1 · p. 17, Figure 10 · p. 19

B.2. Tools

  • CaP-Agent0 exposes APIs for observation, segmentation, point localization, gripper control, navigation, IK-based motion, transformations, visualization, and expert-VLM queries.The tools provide camera, depth, force-wrench, and joint data alongside task execution and multimodal reasoning capabilities.Source: §B.2 · pp. 17, 20

B.1. User Interface

  • The web UI lets users select tasks and models, inspect scene descriptions, generated code, tool-call progress, segmentation and point annotations, wrist-camera views, and provide feedback between turns.Figure 9 presents the interface structure, with selections at the top, responses and execution details in the middle, and instructions at the bottom.Source: §B.1 · p. 17, Figure 9 · p. 18

B.2. Tools

  • Providing rigid-body transformation helpers prevents recurring model-written mistakes, while overlay tools enable expert-VLM reasoning over end-effector axes and segmentation masks.The resulting visual feedback can condition code execution, such as selecting the appropriate segmentation mask.Source: §B.2 · p. 20

B.3. Real World Tasks … B.3.6. TOOL GENERALIZATION WITH DOMAIN KNOWLEDGE

CaP-Agent0 completes diverse real-world manipulation tasks zero-shot by combining visual perception, embodied reasoning, human feedback, and external computational tools. The demonstrations span cluttered-object retrieval, mechanical search, symbolic reasoning, adaptive correction, stable stacking, and geometric tool use.

B.3. Real World Tasks

  • All real-world experiments are zero-shot, with CaP-Agent0 generating detailed underlying code through the tools described in B.2.The section presents execution details as condensed pseudo-substeps, while raw code examples are available in Appendix F.Source: §B.3 · p. 20

B.3.1. NEEDLE IN A HAYSTACK

  • CaP-Agent0 locates and grasps an uncommon auto pencil refill holder in a cluttered scene using a pretrained VLM.The task tests “needles in a haystack” retrieval, illustrated in Figure 11.Source: §B.3.1 · p. 20, Figure 11 · p. 21

B.3.2. MECHANICAL SEARCH

  • CaP-Agent0 systematically explores inverted cups to retrieve a green lime concealed beneath one of them.This occluded-object retrieval problem is presented as mechanical search, with planning and physical execution shown in Figure 12.Source: §B.3.2 · p. 20, Figure 12 · p. 22

B.3.3. MULTIMODAL SYMBOLIC REASONING

  • CaP-Agent0 perceives numbered blocks, interprets their equation, and grasps and places the correct block to solve the task.Figure 13 illustrates the multimodal symbolic reasoning example of solving 59 plus 8.Source: §B.3.3 · p. 20, Figure 13 · p. 23
  • After human feedback identifies an excessively high grasp pose, CaP-Agent0 revises its generated code and successfully picks up the apple on the second attempt.The feedback is “grasped the apple too high,” and Figure 14 shows the failed and corrected attempts.Source: §B.3.3 · p. 21, Figure 14 · p. 24

B.3.5. EMBODIED REASONING

  • CaP-Agent0 stacks round objects on square ones to maximize stability when instructed to stack objects as high as possible.The strategy relies on reasoning about the physical properties of the mixed-shape objects.Source: §B.3.5 · p. 21, Figure 15 · p. 25

B.3.6. TOOL GENERALIZATION WITH DOMAIN KNOWLEDGE

  • CaP-Agent0 invokes SciPy’s RANSAC algorithm on a segmented wall point cloud to compute the surface normal needed to push an elevator button.This demonstrates tool generalization through arbitrary Python packages when the robot approaches the wall at an angle.Source: §B.3.6 · p. 21, Figure 16 · p. 26

C. Full Benchmark Table · D. Additional Takeaways

The full CaP-Bench results report code compilation, dense reward, and task success across models, while additional API-usage examples improve performance for almost all models. The benchmark also excludes single-turn Re-Stack evaluations without visual input because models failed to inspect initial conditions before acting.

C. Full Benchmark Table

  • Figure 17 reports CaP-Bench results using code compilation success, average task reward, and average task success rate.The figure presents these metrics from top to bottom.Source: Figure 17 · p. 27, C. Full Benchmark Table · p. 27
  • For Re-Stack without visual input and with a simple prompt, all models are evaluated only in the multi-turn setup because they fail to check initial conditions before acting.The reported failures occur when models do not write code to inspect the initial condition before performing the task.Source: Figure 17 · p. 27

D. Additional Takeaways

  • In-context API usage examples are added to supplement API descriptions because open-source models initially achieved low code compilation success.The examples accompany API documentation specifying descriptions, input and output datatypes, and shapes.Source: D. Additional Takeaways · p. 28
  • The API docstrings provide each API’s description, expected input and output datatypes, and expected input and output shapes.These specifications form the baseline context before adding examples of API use.Source: D. Additional Takeaways · p. 28
  • The usefulness of in-context API usage is also analyzed through a per-model breakdown.Figure 18 provides the per-model analysis of this added context.Source: Figure 18 · p. 28, D. Additional Takeaways · p. 28
  • Almost all models benefit from additional context showing how to use the low-level APIs.The results are presented in Figures 3 and 4, with a per-model breakdown in Figure 18.Source: D. Additional Takeaways · p. 28, Figure 18 · p. 28

E. Qualitative Analysis of CaP-RL Post-Training Effects · F. CaP-Agent0 Case Studies · F.1. Real-World Case Studies

CaP-RL transforms Qwen2.5-Coder-7B-Instruct behavior from step-skipping and hallucinated state toward causally sequenced, geometrically grounded manipulation. Despite privileged S1-only post-training, the resulting policies transfer zero-shot to non-privileged physical setups and task variants.

E. Qualitative Analysis of CaP-RL Post-Training Effects

  • The qualitative comparison studies CaP-RL on Stack Red Cube on Green Cube using Qwen2.5-Coder-7B-Instruct, with privileged S1 training used to stabilize learning by removing perception noise.The section contrasts code behavior before and after on-policy reinforcement learning with GRPO.Source: E. Qualitative Analysis of CaP-RL… · p. 29
  • Before post-training, the base model often skips prerequisites, identifies the target location, moves directly there, and opens the gripper while hallucinating that it holds the red cube.This failure reflects step skipping and hallucinated state rather than a failure to identify the intended placement location.Source: E. Qualitative Analysis of CaP-RL… · p. 29
  • After GRPO, the model executes the complete causal chain: Identify → Grasp → Transport → Release, learning that grasping must precede placement.Environment interaction and reward feedback correct the pre-RL tendency to move directly to the target and release without grasping.Source: E. Qualitative Analysis of CaP-RL… · pp. 29–30
  • RL replaces hard-coded placement offsets with dynamic stacking heights computed from red and green object bounding-box extents, indicating grounded geometric reasoning.The placement height uses red extent[2]/2 + green extent[2]/2 rather than a fixed offset.Source: E. Qualitative Analysis of CaP-RL… · pp. 29–30
  • Zero-shot transfer from privileged S1 post-training enables robust CaP-RL policies in non-privileged S2 physical setups, including the real-world task shown in Figure 19.The real deployment demonstrates effectiveness despite the training setup omitting perception noise.Source: E. Qualitative Analysis of CaP-RL… · p. 30, Figure 19 · p. 30
  • CaP-RL generalizes zero-shot to task variants, including different objects and randomized colors, because its policy relies on abstract bounding-box properties rather than entity names or fixed scales.Examples include putting a tennis ball on the green cube and handling randomized object colors.Source: E. Qualitative Analysis of CaP-RL… · p. 30, Figure 20 · p. 30

F.1.1. COMMON SENSE PHYSICS-AWARE TASK DECOMPOSITION … H. CaP-Agent0 Details

This section details CaP-Agent0’s components for robust manipulation, including multimodal grounding, closed-loop execution, fallback synthesis, and geometric reasoning, alongside the low-level primitives that support perception, grasp planning, inverse kinematics, motion, and gripper control.

F.1.1. COMMON SENSE PHYSICS-AWARE TASK DECOMPOSITION

  • CaP-Agent0 inferred the only stable stacking order for heterogeneous objects from an open-ended instruction without prescribed sequence or object semantics.Its auxiliary VDM described cubes as stackable and the tennis ball as requiring top placement or careful handling, enabling stability-based decomposition.Source: §F.1.1 · p. 31

F.1.2. IMPLICIT MULTI-STEP REASONING FOR OBSTRUCTED GOALS

  • For an obstructed restacking goal, CaP-Agent0 used multimodal grounding to infer that green and yellow had to be removed before placing blue atop yellow.The perception layer converted the underspecified tower state into an explicit multi-step manipulation strategy.Source: §F.1.2 · pp. 31–32

F.2.1. VERIFICATION AND RETRY BEHAVIOR ENABLED BY MULTITURN DESIGN

  • CaP-Agent0 transformed failed grasps into recovery actions by verifying lift success from subsequent observations, resetting its view, and autonomously retrying.This multi-turn behavior made the controller closed-loop and reactive rather than a static script generator.Source: §F.2.1 · pp. 32–33

F.2.2. PERCEPTION MODULE CASCADE FAILURE FALLBACKS

  • CaP-Agent0 cascaded perception fallbacks by grounding “brown spill” with Molmo, segmenting via SAM3 point prompts, and trying broader text prompts when needed.Fallback prompts included “stain” and “brown liquid.”Source: §F.2.2 · pp. 33–34

F.2.3. GEOMETRIC REASONING FOR GRASP SELECTION

  • CaP-Agent0 filtered grasp candidates for top-down alignment instead of relying solely on raw confidence scores, while retaining highest-score fallback behavior.The synthesized geometric wrapper bridged generic grasp planning and task-specific stability requirements, and the analogous filter entered its persistent skill library.Source: §F.2.3 · p. 34

F.2.4. CODE GENERATION ENSEMBLING INCREASES ROBUSTNESS

  • Model ensembles anticipated failure cases and preemptively inserted grasp fallbacks, whereas single-query generation added a fallback only after observing an invalid grasp.This comparison comes from cube-restack code-generation case studies.Source: §F.2.4 · pp. 35–36

G. Low-Level Perception and Control Primitives

  • CaP-Bench’s low-level Tier S3 primitives specify perception, grasp planning, inverse kinematics, motion, and gripper APIs, while Tier S4 removes in-context usage examples.Tiers S1 and S2 instead use higher-level abstractions composed from these primitives.Source: G. Low-Level Perception and Control… · pp. 36, 37, 38, 40
  • The Arm 1 IK primitive converts a target pose through world and base-frame transforms, applies a tool-center-point offset, and iterates until joint updates converge.It validates required environment capabilities and returns the seven arm-joint values from the converged configuration.Source: G. Low-Level Perception and Control… · p. 43

H.1. Synthesized Task-Agnostic Skill Library … I.2. Place the Blue Cube on Top of the Yellow Cube

The appendix documents CaP-Agent0’s synthesized skill library, ensemble-generation procedures, debugging variant, and generated manipulation programs. It reports reduced average turn count from model ensembling, while debugging prompts did not improve success empirically.

H.1. Synthesized Task-Agnostic Skill Library

  • CaP-Agent0 adds nine agent-synthesized task-agnostic function definitions, interfaces, documentation strings, and low-level implementations to its skill library.The library includes transformations, depth-based 3D geometry, point processing, waypoint interpolation, vector normalization, and top-down grasp selection.Source: §H.1 · pp. 43, 45, 46

H.2. Model Ensemble Temperature Details

  • Nine candidate responses are generated per setting: Gemini-3-Pro is sampled nine times for single-model ensembles, while three models are sampled three times each for multi-model ensembles.The single-model temperatures are 0.1 through 0.9; multi-model temperatures are 0.1, 0.5, and 0.9.Source: §H.2 · p. 47

H.3. Model Ensemble Prompt

  • The ensemble synthesizer critically compares candidates, prefers explicit checks and robust approaches, and emits exactly one complete Python solution in a fenced code block.For subsequent generations, it also combines ideas from REGENERATE candidates and chooses between REGENERATE and FINISH.Source: §H.3 · pp. 47–48

H.4. Multi-turn prompt incentivizing debugging

  • The debugging prompt did not empirically improve success rate, despite targeting inaccurate verification of task completion.Table 5 compares task completions for 3M and 3M + debug.Source: §H.4 · p. 48, Table 5 · p. 48
  • The debugging-oriented multi-turn prompt requires evidence-based verification of code, outputs, and task completion before either finishing or regenerating a revised program.It instructs the agent to distrust logs blindly, identify concrete failures, and try a fundamentally different strategy after repeated unsuccessful attempts.Source: §H.4 · pp. 48–49

H.5. Model Ensemble Decreases Average Turn Count

  • Model ensembling decreases average turn count relative to M4 by anticipating failures preemptively, whereas M4 tends to implement bug fixes and API fallbacks retroactively.Figure 23 presents the average-turn-count comparison.Source: Figure 23 · p. 49, §H.5 · p. 49

I.1. Stack These as High as You Can

  • The generated stacking program detects objects with segmentation and depth, selects top-down grasps, executes pick-and-place motions, and stacks a blue cube on yellow before placing a tennis ball on blue.It re-observes the scene after moving the blue cube to update the second stacking operation.Source: §I.1 · pp. 50–52

I.2. Place the Blue Cube on Top of the Yellow Cube

  • The blue-on-yellow program first removes green and yellow cubes, relocates yellow, then picks blue and places it at yellow’s center with an approximate height offset.It uses text-prompt segmentation, world-frame centroids, top-down grasp planning, refreshed observations, and scripted hover trajectories.Source: §I.2 · pp. 52–55

I.3. Additional Analysis Plots · J. LIBERO-PRO Evaluation Results · K.1. Human Expert Baseline Protocol

The analysis examines multi-turn recovery, LIBERO-PRO generalization, and the human-expert reference protocol. It shows that recovery supports robust execution, while LIBERO-PRO exposes perception and control limitations and human iteration reaches an 88.5% near-upper-bound.

I.3. Additional Analysis Plots

  • On cube stacking, Gemini 3 Pro, GPT 5.2, and Claude Opus 4.5 typically succeed on the first try and use subsequent turns for recovery.The analysis focuses on the three strongest CaP-Bench performers and examines success over turns and tokens.Source: §I.3 · p. 55, Figure 24 · p. 55, Figure 26 · p. 56
  • GPT 5.2 and Claude Opus 4.5 appear to achieve cube-stacking successes with shorter code than Gemini 3 Pro.This comparison is based on the plotted success distributions over tokens.Source: §I.3 · p. 55, Figure 25 · p. 55
  • Cube-stacking success is lower with too few or too many interaction turns, indicating an intermediate turn range is most effective.Figure 26 reports multi-turn success rate versus turns, averaged across M1, M2, and M3.Source: Figure 26 · p. 56

J. LIBERO-PRO Evaluation Results

  • CaP-Agent0’s LIBERO-PRO evaluation covers position and instruction perturbations across object, goal, and spatial benchmarks, with 50 trials per task.Results are reported for OpenVLA, π0, π0.5, and CaP-Agent0 in Tables 6–8.Source: J. LIBERO-PRO Evaluation Results · p. 56, Table 6 · p. 56, Table 7 · p. 57
  • LIBERO-PRO failures arise from perception, grasp generation, and control APIs, including segmentation confusion, angled grasps, and collisions caused by non-collision-aware motion planning.Occlusions and non-top-down cameras contribute to insecure grasps and objects being knocked over during execution.Source: J. LIBERO-PRO Evaluation Results · p. 56

K.1. Human Expert Baseline Protocol

  • 88.5% average success is achieved by iterated human-written code on single-turn tiers, serving as a near-upper-bound for development-time robotics expertise.The reference was produced by seven authors with at least two years of robotics programming experience, using the same API primitives and offline trial-and-error refinement as the model.Source: §K.1 · p. 57
  • The human baseline relies on extensive offline refinement, whereas CaP-Agent0 performs online multi-turn monitoring and recovery in approximately 2 minutes per Robosuite trial.Simple tasks took under a day to refine, while contact-rich and bimanual tasks required 2–3 weeks per task.Source: §K.1 · p. 58

K.2. Computational Cost Analysis · K.3. Visual Differencing Module: Model Choice

The computational analysis measures code-generation and total trial time under controlled cube-stacking conditions, while noting that CaP-Agent0’s task-level abstraction makes direct per-action latency comparisons misleading. The visual differencing module uses Gemini-3-Pro to upper-bound language-grounded perception effects and is designed to support backbone-independent multi-turn recovery.

K.2. Computational Cost Analysis

  • Code-generation time and total trial time were averaged over N=20 cube-stacking trials using Gemini-3-Pro with identical hardware, perception, and control stacks.The total trial time includes LLM inference and execution.Source: §K.2 · p. 58
  • The timing analysis reports per-trial breakdowns across CaP-Bench tiers for Gemini-3-Pro on cube stacking.Table 9 provides the scope of the reported timing breakdown, without supplying values in the passage.Source: Table 9 · p. 58
  • CaP-Agent0 generates complete manipulation sequences per iteration, so task-attempt cost—not per-action latency—is the relevant comparison axis.Each code-generation iteration takes 6.8–23.8 s, unlike VLA systems that emit single motor commands at high frequency.Source: §K.2 · p. 58
  • π0 reports ∼73 ms per action chunk, whereas OpenVLA (Kim et al., 2024) reports ∼167 ms per action.These VLA timings are substantially faster per inference step because they produce individual motor commands rather than full manipulation sequences.Source: §K.2 · p. 58

K.3. Visual Differencing Module: Model Choice

  • The Visual Differencing Module uses Gemini-3-Pro, selected as the strongest available VLM to ensure M3/M4 gains reflect auxiliary text grounding rather than a weak captioner.At the time of writing, Gemini-3-Pro achieved 81% on MMMU-Pro and 87.6% on Video-MMMU.Source: §K.3 · p. 58
  • VDM is model-agnostic: changing its backbone requires one CaP-Gym configuration change and no alterations to the coding-agent loop or benchmark.The paper demonstrates that VDM materially improves multi-turn recovery, while systematic backbone ablation remains an orthogonal supported study.Source: §K.3 · p. 58

Related papers

Loading 2603.22435v2…