PaperlayerView exact revision on arXiv

Source-linked AI summary

CaP-X: A Framework for Benchmarking and Improving Coding Agents for Robot Manipulation

Letian Fu, Justin Yu, Karim El-Refai, Ethan Kou, Haoru Xue, Huang Huang, Wenli Xiao, Guanzhi Wang, Dantong Niu, Fei-Fei Li, Guanya Shi, Jiajun Wu, Shankar Sastry, Yuke Zhu, Ken Goldberg, Linxi "Jim" Fan

arXiv:2603.22435v2cs.ROcs.AI

TL;DR

Code-based robot-control agents have limited evidence of reliable generalization across environments, embodiments, and long-horizon tasks. CaP-X benchmarks and improves them through interactive code execution, abstraction-level evaluation, and agentic computation, finding that abstractions and multi-turn feedback improve performance while skill synthesis supports primitive-level control.

  • Problem

    Existing robot-control approaches struggle to generalize across environmental changes, new embodiments, and long-horizon tasks without additional data, retraining, or both.

  • Method

    CaP-X combines interactive CaP-Gym, controlled CaP-Bench evaluations, training-free CaP-Agent0, and reinforcement-learning-based CaP-RL for code-based robot control.

  • Results

    Task success increases with higher-level abstractions, while multi-turn interaction and execution feedback consistently improve performance across models.

  • Takeaways & Limitations

    Evaluating primitive-level performance and enabling agents to synthesize reusable skills provides a scalable path beyond reliance on human-designed abstractions.

  • Takeaways & Limitations

    Interleaving raw RGB observations during multi-turn interaction degrades performance relative to text-only execution feedback.

Abstract

from arXiv · show

"Code-as-Policy" considers how executable code can complement data-intensive Vision-Language-Action (VLA) methods, yet their effectiveness as autonomous controllers for embodied manipulation remains underexplored. We present CaP-X, an open-access framework for systematically studying Code-as-Policy agents in robot manipulation. At its core is CaP-Gym, an interactive environment in which agents control robots by synthesizing and executing programs that compose perception and control primitives. Building on this foundation, CaP-Bench evaluates frontier language and vision-language models across varying levels of abstraction, interaction, and perceptual grounding. Across 12 models, CaP-Bench reveals a consistent trend: performance improves with human-crafted abstractions but degrades as these priors are removed, exposing a dependence on designer scaffolding. At the same time, we observe that this gap can be mitigated through scaling agentic test-time computation--through multi-turn interaction, structured execution feedback, visual differencing, automatic skill synthesis, and ensembled reasoning--substantially improves robustness even when agents operate over low-level primitives. These findings allow us to derive CaP-Agent0, a training-free framework that recovers human-level reliability on several manipulation tasks in simulation and on real embodiments. We further introduce CaP-RL, showing reinforcement learning with verifiable rewards improves success rates and transfers from sim2real with minimal gap. Together, CaP-X provides a principled, open-access platform for advancing embodied coding agents.

1. Introduction

CaP-X addresses the unclear role of agent capability versus human-designed scaffolding in code-based robot control by providing a unified evaluation and improvement framework. It combines CaP-Gym, CaP-Bench, and agentic methods to study abstraction, interaction, perceptual grounding, and training-free robot-control reliability.

  • Motivation: Classical robot-control programs combine perception, geometry, planning, and feedback, while newer systems struggle to generalize across environments, embodiments, and long-horizon tasks.These paradigms evolved toward geometric motion planning and Task-and-Motion Planning as robots entered continuous, higher-dimensional spaces.
  • Motivation: Human-tuned Code-as-Policy primitives simplify robotics tasks, leaving unclear how much performance comes from the agent versus imposed primitive structure.Prior work used task-specific abstractions such as stack_objs_in_order(), but did not systematically characterize agent performance across these conditions.
  • CaP-X framework: CaP-X introduces CaP-Gym, an interactive environment where agents generate executable programs composing perception and control primitives to directly control robots.CaP-Gym integrates 187 tasks from standard robot-manipulation simulators under a shared primitive design compatible with simulation and physical robot systems.
  • CaP-Bench: CaP-Bench evaluates coding agents along abstraction level, temporal interaction, and perceptual grounding, from human-crafted macros and zero-shot generation to atomic primitives and multi-turn recovery.The benchmark is designed to quantify how agents handle failure recovery, iterative reasoning, and differing visual modalities.
  • Agentic methods: CaP-Agent0 is a training-free framework that combines multi-turn interaction, text-based visual grounding, automatic task-agnostic skill synthesis, and parallelized multi-model generation.It achieves performance comparable to, and sometimes exceeding, human expert baselines on CaP-Bench tasks; CaP-Gym also supports reinforcement learning for coding agents.

1. CaP-Gym, a unified suite of interactive robot coding

CaP-Gym is an interactive suite for evaluating and training code-generating multimodal embodied agents across diverse robot-manipulation environments. CaP-Bench measures control performance across primitive and modality levels, while CaP-Agent0 provides a training-free agentic harness for interaction and skill synthesis.

  • CaP-Gym: CaP-Gym spans tabletop, bimanual, and mobile manipulation tasks for code-generating multimodal embodied agents.The environments are designed for both evaluation and training.
  • CaP-Bench: CaP-Bench systematically measures robot-control performance across tasks and levels of primitives and modalities.Its evaluation tiers vary environment-state access, primitive abstraction, in-context primitive examples, and visual-grounding modality.
  • CaP-Agent0: CaP-Agent0 is a training-free agentic harness combining multi-turn visual differencing, ensembled reasoning, and automatic skill-library synthesis.These components support agentic interaction and reusable skill construction.

4. CaP-RL,

CaP-RL applies reinforcement learning to the coding agent using environment rewards.

  • CaP-RL: CaP-RL trains the coding agent with reinforcement learning driven by environment reward.

2. CaP-Gym

CaP-Gym is a hierarchical Code-as-Policy control framework that connects a stateful code executor with physics simulation or real-world robot environments through a Gymnasium-compatible interface. It equips agents with modular perception and control primitives while delegating execution feasibility to specialized controllers.

  • Architecture: CaP-Gym combines a Low-Level Environment loop with a stateful Code Executor loop under the standard Gymnasium interface.It preserves the native dynamics of simulators including Robosuite, LIBERO-PRO, and BEHAVIOR while exposing them through a REPL paradigm.
  • Implementation: Stateless perception and control services enable high-throughput parallel evaluation of CaP-Gym agents.The services contain the computationally intensive perception and control primitives.
  • Perception Primitives: Perception primitives transform raw sensor data into structured semantic objects that agents can use for environment interaction.Examples include language-conditioned segmentation, open-vocabulary pointing, and standard computer-vision libraries.
  • Control Primitives: Control primitives replace direct joint-space commands with motion planning or inverse-kinematics calls that support collision checking and reachability constraints.This lets agents reason in task-oriented Cartesian space while controllers handle execution feasibility and action-space transformations.

3. CaP-Bench: Evaluating Frontier Models

CaP-Bench evaluates 12 vision-language and language models across seven manipulation tasks and controlled tiers that vary abstraction, interaction, and perceptual grounding. Results show that human-designed abstractions improve single-turn performance, while multi-turn textual feedback and visual differencing recover robustness over low-level primitives.

  • Benchmark setup: CaP-Bench evaluates 12 state-of-the-art vision-language and language models across seven core manipulation tasks, each with 100 trials per tier.The tasks span single-arm manipulation through bimanual coordination, including Cube Lift, Cube Stack, Spill Wipe, Peg Insertion, Cube Re-stack, Two-Arm Lift, and Two-Arm Handover.
  • Benchmark setup: The benchmark varies primitive abstraction, interaction mode, and feedback or grounding signal across privileged, perception-based, low-level, and multi-turn settings.S1 uses ground-truth simulation state; S2 uses perception from raw RGB-D; S3 provides low-level usage examples; S4 removes those examples; M1–M4 add interactive feedback and grounding.
  • Single-turn evaluation: Success increases monotonically with primitive abstraction, revealing a persistent single-turn gap between frontier models and human expert reference solutions.Higher-level primitives improve performance, while the human reference is constructed under identical environments and APIs as a near-upper-bound comparison.
  • Abstraction and expressivity: Higher-level abstractions boost performance but constrain the action space, whereas lower-level primitives enable expressive behaviors that fixed primitives cannot represent.The lower-level performance drop reflects code-synthesis difficulty, while abstraction imposes a generality ceiling through human priors.
  • Interactive grounding: Multi-turn execution feedback consistently improves performance, while direct raw-RGB interleaving degrades performance relative to text-only feedback.Textual stdout/stderr feedback supports diagnosis, debugging, and recovery; the reported degradation for M2 is attributed to a cross-modal alignment gap.
  • Interactive grounding: Low-level primitives with multi-turn feedback surpass high-level single-turn performance and can reach parity with high-level multi-turn performance.Across multi-turn settings, agents often recover coding and physical execution failures through iterative interaction and explicit symbolic-state checks.

4. CaP-Agent0: An Agentic Framework for Robot Control

CaP-Agent0 is a training-free robot-control framework that combines multi-turn visual differencing, an automatically synthesized persistent skill library, and parallel reasoning. It achieves human-comparable performance on several simulated tasks, transfers zero-shot to real robots, and supports reinforcement learning with verifiable rewards and sim-to-real transfer.

  • Framework design: CaP-Agent0 combines multi-turn visual differencing, a persistent task-agnostic skill library, and parallel candidate reasoning to address grounding, low-level coding, and test-time exploration failures.The system grounds observations in structured textual scene changes, reuses synthesized utilities, and has a central coding agent synthesize candidate solutions.
  • Framework design: 9 verified, task-agnostic primitives are synthesized from successful S3-tier rollouts pooled across 12 models and 7 Robosuite tasks.The resulting library is model-independent and extracts recurring implementation patterns for reuse across trials.
  • Simulation results: Across 100 trials per task, CaP-Agent0 matches or exceeds human-written programs on 4 out of 7 tasks despite using only low-level primitives.Visual differencing, the self-synthesized skill library, and parallel reasoning significantly outperform single-turn baselines.
  • Extended evaluation: CaP-Agent0 achieves comparable or higher performance than three training-based VLA methods on LIBERO-PRO while remaining training-free.The comparison includes OpenVLA, π0, and π0.5 under initial-position and instruction perturbations.
  • Real-world evaluation: Zero-shot deployment on Franka Panda and AgiBot G1 demonstrates real-world transfer without major cross-embodiment modifications, aside from single-arm to bimanual primitive changes.The CaP-Gym loop interfaces directly with real-world perception and control interfaces.
  • CaP-RL: CaP-RL applies GRPO with verifiable rewards to coding agents, improving compilation and strategic robustness after 50 iterations per task and retaining 84% cube-lifting and 76% stacking success sim-to-real.The transferred action space is code composed from shared perception and control tools rather than raw visual-to-motor mappings.

6. Related Work

Related work spans Code-as-Policy methods, structured planning and refinement, LLM-generated skill synthesis with reinforcement learning, and benchmarks for robotics and embodied agents. Existing benchmarks generally do not combine executable program synthesis, tiered APIs, multi-turn debugging, and embodied perception in one evaluation setting.

  • Code as Policies: Code-as-Policy research uses language models to generate executable programs that compose robot perception and control modules.Work covers affordance-grounded planning, closed-loop behaviors through APIs, Python programs over perception APIs, and modular vision-language agent pipelines.
  • Code as Policies: Structured representations, persistent state tracking, executable actions, and iterative self-refinement are reported to improve planning reliability.Examples include PDDL, Signal Temporal Logic, persistent state tracking, executable code as an action representation, and iterative self-refinement.
  • Skill Synthesis and RL with LLM-Generated Code: Prior work uses frozen language models to generate reward functions, curricula, and skills for separately trained policies, while RL with verifiable rewards improves models across several settings.The settings named include reasoning, code, and agentic tasks.
  • Benchmarks for robotics and embodied agents: Robotics benchmarks evaluate fixed policy interfaces, code benchmarks lack embodied perception, and embodied-agent benchmarks do not provide the combined executable-program evaluation targeted here.The comparison concerns executable program synthesis with tiered APIs and multi-turn debugging alongside embodied perception.

7. Conclusion … B.2. Tools

CaP-X unifies benchmarking and improvement of coding agents for robot control, while the real-world CaP-Agent0 setup combines interactive feedback, visualization, perception, reasoning, and manipulation tools. The paper also identifies brittleness in contact-rich behaviors and motivates hybrid CaP-VLA policies.

  • 7. Conclusion: CaP-X unifies CaP-Gym, CaP-Bench, CaP-Agent0, and CaP-RL to evaluate robot-control agents across abstraction levels, interaction modes, and learning paradigms.The framework treats agent design, inference-time computation, perception, and control as jointly studied aspects of machine intelligence.
  • A. Future Works: Programmatic control performs well on long-horizon reasoning tasks but remains brittle for contact-rich behaviors requiring tight visual servoing and continuous feedback.Insertion and pouring are examples of the behaviors for which this limitation is identified.
  • A. Future Works: Hybrid CaP-VLA policies are proposed to let coding agents manage high-level task logic and recovery while VLA policies handle low-level execution.This direction is motivated by the complementary strengths of programmatic control and VLA-based execution.
  • B. Interactive Real-World Setup: CaP-Agent0 uses a chat-based web UI in which users propose novel tasks, inspect step-by-step tool-use results, and provide feedback for multi-turn improvement on an AgiBot G1.The setup is intended to demonstrate zero-shot completion of novel real-world tasks.
  • B.1. User Interface: The interface displays task and model selections, scene descriptions, generated code, tool-call progress, segmentation masks, Molmo 2 points, and wrist-camera views during execution.Users can provide additional feedback through the chatbox between turns.
  • B.1. User Interface: Viser adds a 3D visualization of the live robot URDF, trajectories, and depth pointcloud to the interactive interface.This visualization is presented as a dedicated section of the web UI.

B.3. Real World Tasks … B.3.6. TOOL GENERALIZATION WITH DOMAIN KNOWLEDGE

Across zero-shot real-world tasks, CaP-Agent0 demonstrates code-as-policy capabilities in visual retrieval, mechanical search, multimodal reasoning, feedback-driven correction, embodied planning, and tool use. These examples show it can combine pretrained vision-language models, physical reasoning, human feedback, and external Python tools to execute manipulation tasks.

  • B.3. Real World Tasks: CaP-Agent0 completes zero-shot real-world manipulation tasks by writing detailed underlying code with tools available in B.2.Execution details are presented as condensed pseudo sub-steps, while raw code is available in Appendix F.
  • B.3.1. NEEDLE IN A HAYSTACK: In cluttered scenes, CaP-Agent0 localizes and grasps an uncommon auto pencil refill holder, leveraging pretrained VLMs for needle-in-a-haystack retrieval.The task targets an object that is relatively uncommon and challenging for end-to-end learning policies such as VLAs.
  • B.3.2. MECHANICAL SEARCH: For mechanical search, CaP-Agent0 systematically explores three inverted cups to retrieve a green lime concealed beneath one of them.The task requires locating an occluded object through physical exploration.
  • B.3.3. MULTIMODAL SYMBOLIC REASONING: CaP-Agent0 solves multimodal symbolic reasoning tasks by perceiving numbered wooden blocks, interpreting an equation, and placing the correct block.The illustrated example asks the robot to solve 59 plus 8.
  • B.3.4. LEARNING FROM HUMAN FEEDBACK: After failing with an excessively high grasp pose, CaP-Agent0 incorporates the feedback “grasped the apple too high,” modifies its code, and succeeds on the second attempt.This demonstrates code-generation adjustment from human feedback.
  • B.3.5. EMBODIED REASONING: To stack objects as high as possible, CaP-Agent0 reasons about physical properties and places round objects on square ones for stability.The scene contains both square and round objects, requiring a stable stacking strategy.
  • B.3.6. TOOL GENERALIZATION WITH DOMAIN KNOWLEDGE: For the elevator task, CaP-Agent0 uses arbitrary Python-package tools and invokes SciPy RANSAC on segmentation to address angled button-pushing geometry.The task involves locating and pressing the button before moving into the elevator.

C. Full Benchmark Table · D. Additional Takeaways

The full CaP-Bench table reports code compilation, dense reward, and task success, while additional results show that in-context API usage examples improve performance for almost all models.

  • C. Full Benchmark Table: CaP-Bench reports code compilation success rate, average dense reward, and average task success rate.These metrics are presented in Figure 17 as the benchmark’s full results.
  • C. Full Benchmark Table: For Re-Stack under a simple prompt without visual input, models failed because they did not first check the initial condition before acting.All models were therefore evaluated only in the multi-turn setup for this condition.
  • D. Additional Takeaways: Adding in-context API usage examples was motivated by low initial code compilation success among open-source models.The examples supplemented API descriptions of inputs, outputs, datatypes, and shapes in S4.
  • D. Additional Takeaways: The API usage examples provided brief demonstrations of how to use the low-level APIs.Their usefulness was analyzed through aggregate results and a per-model breakdown.
  • D. Additional Takeaways: Almost all models benefited from additional context showing examples of low-level API usage.The results are reported in Figures 3 and 4, with per-model results in Figure 18.
  • D. Additional Takeaways: The additional API context improved performance across nearly the entire model set.This conclusion is based on the aggregate results and their per-model breakdown.

E. Qualitative Analysis of CaP-RL Post-Training Effects · F. CaP-Agent0 Case Studies · F.1. Real-World Case Studies

CaP-RL post-training transforms the model’s manipulation behavior from skipping prerequisite actions and hallucinating state to executing causally ordered actions with geometry grounded in object dimensions. Although trained on privileged S1 observations, the resulting policy transfers zero-shot to real-world S2 setups and related task variants.

  • E. Qualitative Analysis of CaP-RL Post-Training Effects: CaP-RL uses the privileged S1 tier during training to remove perception noise and stabilize on-policy reinforcement learning with GRPO.The qualitative analysis focuses on Qwen2.5-Coder-7B-Instruct and the Stack Red Cube on Green Cube task.
  • F.1. Real-World Case Studies: In real deployment, the finetuned model successfully performs the red-cube-on-green-cube task, while the Human Oracle achieves 21/25 task successes.Before reinforcement learning, the model could approach the green cube first instead of sensibly approaching the red cube.
  • E. Qualitative Analysis of CaP-RL Post-Training Effects: CaP-RL replaces pre-training step skipping with the complete Identify → Grasp → Transport → Release manipulation chain.Before RL, the model moved directly toward the placement target without grasping the red cube, effectively hallucinating the required state.
  • E. Qualitative Analysis of CaP-RL Post-Training Effects: Training on privileged S1 produces robust zero-shot transfer to non-privileged S2 physical setups.The resulting policies function effectively in real-world embodiments, including the task of putting the red cube on the green cube.
  • F. CaP-Agent0 Case Studies: The finetuned policy generalizes zero-shot to related instructions, including putting a tennis ball on the green cube and randomized object colors.Its logic relies on abstract bounding-box properties rather than specific entity names or training-instance scales.

F.1.1. COMMON SENSE PHYSICS-AWARE TASK DECOMPOSITION … H. CaP-Agent0 Details

CaP-Agent0 combines multimodal perception-to-text grounding, iterative verification, fallback strategies, geometric reasoning, and ensemble generation to perform robust manipulation under underspecified goals and execution failures. The framework exposes these capabilities through low-level perception and control primitives used by CaP-Bench.

  • F.1.1. COMMON SENSE PHYSICS-AWARE TASK DECOMPOSITION: On a real Franka Panda, CaP-Agent0 inferred a stable stacking order for heterogeneous objects from the open-ended instruction “stack these as high as you can.”The prompt specified neither a stacking sequence nor semantic object descriptions.
  • F.1.1. COMMON SENSE PHYSICS-AWARE TASK DECOMPOSITION: An auxiliary VDM multimodal agent described cubes as stackable and the tennis ball as requiring top placement or careful handling, grounding subsequent code generation.The coding agent used this textual grounding to decompose the task around stability constraints.
  • F.1.2. IMPLICIT MULTI-STEP REASONING FOR OBSTRUCTED GOALS: For an obstructed cube-restacking goal, CaP-Agent0 recognized that the blue cube supported the yellow cube beneath a green cube and planned removal and rearrangement steps.The goal prompt omitted both the green obstruction and the blue-yellow support relationship.
  • F.1.2. IMPLICIT MULTI-STEP REASONING FOR OBSTRUCTED GOALS: This perception-to-text grounding enabled complex manipulation strategies that would traditionally require extensive task-specific imitation learning data or manual state-machine design.The auxiliary multimodal layer supplied a high-level strategic prior before code generation.
  • F.2.1. VERIFICATION AND RETRY BEHAVIOR ENABLED BY MULTITURN DESIGN: In the Robosuite “pick up the red cube” task, multi-turn CaP-Agent0 verified whether a grasp lifted the object and retried after physics stochasticity or poor contact caused failure.The generated logic checks segmentation and depth-derived lifting before either completing the task or retrying from a home pose.
  • F.2.2. PERCEPTION MODULE CASCADE FAILURE FALLBACKS: For “wipe up the brown spill,” the agent cascaded Molmo point grounding into SAM3 segmentation, then fell back to broader prompts such as “stain” or “brown object.”This hierarchy addresses generic or imprecise text-prompt failures.
  • F.2.3. GEOMETRIC REASONING FOR GRASP SELECTION: CaP-Agent0 filtered grasp candidates by downward approach geometry instead of raw Contact-GraspNet scores, while retaining a global-score fallback and synthesizing a reusable grasp filter.The geometric wrapper selected vertical candidates to improve stability and avoid execution stagnation.
  • F.2.4. CODE GENERATION ENSEMBLING INCREASES ROBUSTNESS: Model ensembles anticipated failure cases and added fallbacks, unlike single-query generation that retroactively addressed an invalid grasp pose after execution.The primitive interface supplies perception and control APIs, including Molmo, SAM3, grasp planning, inverse kinematics, and joint motion.

H.1. Synthesized Task-Agnostic Skill Library … I.2. Place the Blue Cube on Top of the Yellow Cube

The appendices detail CaP-Agent0’s synthesized skill library, ensemble-generation procedures, debugging prompt, reduced-turn behavior, and generated manipulation programs for stacking and cube placement. They also show that debugging incentives did not improve success empirically, while model ensembling reduced average turn count through preemptive robustness.

  • H.1. Synthesized Task-Agnostic Skill Library: CaP-Agent0’s skill library adds nine agent-synthesized function definitions, interfaces, documentation strings, and low-level implementations.The library includes utilities for rotation conversion, mask-to-world point projection, transforms, and grasp processing.
  • H.2. Model Ensemble Temperature Details: The ensemble generates 9 candidate responses per setting, using nine Gemini-3-Pro queries or three queries each from Gemini-3-Pro, Claude-Opus-4.5, and GPT-5.2.Single-model temperatures span 0.1 through 0.9; multi-model temperatures are 0.1, 0.5, and 0.9.
  • H.3. Model Ensemble Prompt: The model-ensemble prompt synthesizes candidate Python solutions by critically checking assumptions, combining robust ideas, and outputting one program.Subsequent generations additionally use regenerate-versus-finish votes and multi-turn robot-control context.
  • H.4. Multi-turn prompt incentivizing debugging: The debugging-oriented multi-turn prompt did not empirically improve success rate despite targeting inaccurate verification of task completion.The experiment compared task completions between 3M and 3M + debug.
  • H.4. Multi-turn prompt incentivizing debugging: The modified debugging prompt requires explicit task-completion verification, evidence-based diagnosis, alternative strategies after repeated failure, and a synthesized fix.It treats observed state differences as evidence but warns that they may be inaccurate.
  • H.5. Model Ensemble Decreases Average Turn Count: Model ensembling decreases average turn count by preemptively anticipating failures, whereas M4 tends to implement bug fixes and API fallbacks retroactively.The appendix points to a case study contrasting retroactive and preemptive behavior.
  • I.1. Stack These as High as You Can: The generated stacking program segments objects, computes 3D metrics and top-down grasps, lifts the first object, places it, and then stacks a second object on the new base.The code uses a relaxed grasp threshold when no strictly vertical grasp is found and adds a 0.20 lift offset.
  • I.2. Place the Blue Cube on Top of the Yellow Cube: The blue-on-yellow program segments objects, plans grasps from depth and camera geometry, removes an obstructing green cube, refreshes observations, and then picks and places the blue cube.It uses a fixed top-down quaternion and recomputes the yellow cube’s location before the blue-cube placement.

I.3. Additional Analysis Plots · J. LIBERO-PRO Evaluation Results · K.1. Human Expert Baseline Protocol

The paper analyzes multi-turn success dynamics, evaluates CaP-Agent0 under LIBERO-PRO perturbations, and clarifies that the human baseline reflects extensive expert iteration rather than single-shot attempts. The iterated human reference reaches 88.5% average success on single-turn tiers, while CaP-Agent0’s deployment-time recovery takes approximately 2 minutes per Robosuite trial.

  • I.3. Additional Analysis Plots: The cube-stack analysis examines success distributions over turns and tokens, plus success rate as a function of turns, for Gemini 3 Pro, GPT 5.2, and Claude Opus 4.5.The three models are the strongest CaP-Bench performers, and results are averaged across M1, M2, and M3 where specified.
  • I.3. Additional Analysis Plots: Trials with too few or too many turns have lower cube-stack success rates.Figure 26 reports results averaged across M1, M2, and M3.
  • J. LIBERO-PRO Evaluation Results: LIBERO-PRO tests OpenVLA, π0, π0.5, and CaP-Agent0 under initial-position and instruction perturbations across object, goal, and spatial benchmarks.Each task was executed over 50 trials.
  • J. LIBERO-PRO Evaluation Results: CaP-Agent0 encounters failures in perception, grasp generation, and control APIs on LIBERO-PRO.The passage gives SAM 3 segmentation errors for queries such as “alphabet soup can” as one example.
  • K.1. Human Expert Baseline Protocol: The human baseline was produced by N=7 paper authors with 2+ years of robotics programming experience, each iterating a single Python script per task and tier until high reliability.Scripts used exactly the API primitives available to models at the corresponding tier and incorporated execution traces, bug fixes, and script updates.
  • K.1. Human Expert Baseline Protocol: 88.5% average success is achieved by the iterated human reference on single-turn tiers and is treated as a near-upper-bound for hand-written static code with development-time iteration.This is not a single-shot human attempt.
  • K.1. Human Expert Baseline Protocol: Human development ranges from <1 day for simple pick-and-place tasks to 2–3 weeks per task for contact-rich and bimanual tasks.The longer estimate includes comparing alternative primitives and updating implementations as the toolchain evolved.
  • K.1. Human Expert Baseline Protocol: CaP-Agent0 completes a Robosuite evaluation in approximately 2 minutes per trial, enabling deployment-time multi-turn monitoring and recovery.The comparison is between extensively iterated static human code and an interactive agent recovering online at evaluation time.

K.2. Computational Cost Analysis · K.3. Visual Differencing Module: Model Choice

K.2 reports code-generation and total trial times for cube stacking and cautions that CaP-Agent0’s sequence-level latency is not directly comparable to VLA per-action inference. K.3 uses Gemini-3-Pro for VDM while keeping the module model-agnostic and configurable for backbone ablations.

  • K.2. Computational Cost Analysis: Code-generation time and total trial time were averaged over N=20 cube-stacking trials with Gemini-3-Pro.Measurements used identical hardware, perception, and control stacks.
  • K.2. Computational Cost Analysis: CaP-Agent0’s code-generation iterations take 6.8–23.8 s and produce entire manipulation sequences rather than single actions.The passage states that each iteration operates at a different abstraction level from VLA inference.
  • K.2. Computational Cost Analysis: π0 reports ∼73 ms per action chunk, whereas OpenVLA reports ∼167 ms per action.These VLA figures are presented as reference points for per-inference-step latency.
  • K.2. Computational Cost Analysis: Direct latency comparison is misleading because VLAs emit single motor commands at high frequency while CaP-Agent0 generates complete manipulation sequences.The passage frames cost per action or sequence as the relevant comparison axis, though its final wording is truncated.
  • K.3. Visual Differencing Module: Model Choice: The VDM uses Gemini-3-Pro, described as state-of-the-art on major multimodal benchmarks when selected for the study.Reported reference scores are MMMU-Pro: 81% and Video-MMMU: 87.6%.
  • K.3. Visual Differencing Module: Model Choice: The strongest available VLM was chosen to upper-bound the effect of language-grounded perception in M3/M4.This design aims to ensure observed gains reflect text grounding via an auxiliary VLM rather than weakness of an alternative model, with the passage truncated before completion.
  • K.3. Visual Differencing Module: Model Choice: VDM is model-agnostic: changing its backbone requires one CaP-Gym configuration change without modifying the coding-agent loop or benchmark.CaP-Bench supports systematic VDM-backbone ablations as an orthogonal study axis.
  • K.3. Visual Differencing Module: Model Choice: The paper’s contribution is demonstrating that VDM materially helps multi-turn recovery, not conducting a systematic backbone ablation.The authors encourage the community to explore backbone substitutions using the supported framework.
Loading 2603.22435v2…