PaperlayerView exact revision on arXiv

Source-linked AI summary

RACaP: Agentic Reasoning, Acting, and Coding as Policies for Evolvable Robot Learning

Zexi Li, Yehang Zhang, Haojian Huang, Bohan Zhou, Wenqian Li, Chenxu Wang, Yifan Chang, Yangkai Wei, Tianyi Zhang, Ying-Cong Chen, Kaiwen Zhou, Yinchuan Li, James Cheng

arXiv:2609.29394v1cs.RO

TL;DR

General-purpose robot policies must reuse experience across changing tasks, while runtime code generation introduces latency and mixes reusable mechanisms with scene-specific choices. RACaP moves coding into evolution and uses ReAct with frozen Policy APIs at deployment; across LIBERO benchmarks, it improves transfer and achieves 2.5× CaP success on LIBERO-PRO.

  • Problem

    Robot agents need to reuse experience across changing physical tasks, while runtime code generation adds latency and entangles reusable mechanisms with task-specific choices.

    Source: §1 · pp. 1–2

  • Method

    RACaP builds competence through a capability curriculum, evolves typed Policy APIs, the ReAct harness, and long-term memory, then deploys ReAct over frozen APIs.

    Source: §1 · p. 3, §3.1 · p. 4

  • Results

    Across in-domain, zero-shot, and long-horizon LIBERO benchmarks RACaP improves transfer; on LIBERO-PRO, it achieves 2.5× CaP success and 1.9× median policy-time speedup.

    Source: §1 · p. 3, §6 · p. 10

  • Takeaways & Limitations

    Separating reusable code from runtime decisions supports continued evolution, effective transfer, and efficient long-horizon control.

    Source: §6 · p. 10

    At deployment, RACaP generalizes within its Policy API closure rather than inventing new physical skills online.

    Source: §5 · p. 10

Abstract

from arXiv · show

General-purpose robot agents must learn from experience, transfer to new tasks, and act efficiently. Code as Policies (CaP) methods generate and repair programs at runtime, incurring latency and entangling reusable mechanisms with task-specific decisions. We introduce RACaP, an agentic framework that moves coding to evolution and uses a Reasoning-and-Acting (ReAct) loop to call frozen, typed Policy APIs at deployment. A two-phase strategy combines capability curriculum learning with autonomous self-evolution to improve the APIs, the ReAct harness, and experience memory. The APIs encode reusable physical mechanisms while exposing arguments for runtime adaptation. ReAct combines task-specific working memory, long-term experience memory, and visual feedback to select actions, verify outcomes, and recover from failures without modifying source code. RACaP achieves 54.4% success on LIBERO-90, 45.0% on zero-shot LIBERO-PRO, and 46.0% on LIBERO-Long, compared with at most 4.0% for CaP baselines on long-horizon tasks. On LIBERO-PRO, it achieves 2.5 times the success rate of CaP baselines and a 1.9-fold speedup in median policy time. For efficient on-robot deployment, rejection-sampled fine-tuning distills GPT-5.6 ReAct decisions into Qwen3-VL-8B-Instruct, yielding a 13.2-fold per-decision inference speedup and reducing repeated physical calls from 16 to 4. These results show that separating reusable code from runtime decisions supports continued evolution, effective transfer, and efficient long-horizon control.

1 Introduction

RACaP shifts coding from task execution to pre-deployment evolution, then uses a ReAct loop over frozen APIs to adapt actions to the current scene. It reports transfer across LIBERO settings and faster decisions than CaP baselines.

1 Introduction

  • CaP puts coding on the physical execution path: one CaP-X coding iteration takes 6.8 to 23.8 seconds, and a multi-turn trial takes 60.8 to 113.6 seconds.Runtime code can also select the wrong object or repeat an ineffective action during operation.Source: §1 · pp. 1–2
  • Runtime programs can mix reusable physical mechanisms with task-specific setups and offsets, making prior experience difficult to transfer when tasks change.The paper frames this as an unresolved evolution-to-execution problem for current CaP methods.Source: §1 · p. 2
  • A capability curriculum first establishes reliable execution, recovery, skill breadth, and orchestration; autonomous paired evolution then improves the APIs, harness, and memory.Candidates are promoted only when paired gains exceed paired regressions.Source: §1 · p. 3
  • At deployment, frozen Policy APIs encode reusable geometry and contact mechanisms, while ReAct steers exposed arguments and decisions using working memory, experience memory, and visual feedback.ReAct can change action sequence, recovery strategy, and stopping without generating source code.Source: §1 · p. 3
  • RACaP reaches 54.4% on LIBERO-90, 45.0% on zero-shot LIBERO-PRO, and 46.0% on LIBERO-Long; on LIBERO-PRO it achieves 2.5× CaP success and 1.9× median policy-time speedup.The evaluation also covers in-domain, zero-shot, long-horizon, and cross-embodiment settings.Source: §1 · p. 3
  • A compact model provides a 13.2× per-decision speedup and reduces repeated physical calls by 75%; the distilled ReAct model preserves accepted teacher decisions.Source: §1 · p. 3

2 Related Work

Prior work connects language instructions and visual feedback to robot actions, and increasingly retains programs or execution experience. RACaP differs by evolving both its harness and callable policies from robot rollouts.

Agentic Robot Harnesses

  • Agentic harnesses progress from language planning to feedback-driven orchestration, but callable skills usually remain fixed; RACaP extends both harness and Policy APIs from robot rollouts.The related-work discussion contrasts this with harnesses connected to learned visuomotor policies.Source: §2 · p. 3

Code as Policy (CaP)

3 Method

RACaP evolves reusable Policy APIs, a ReAct harness, and experience memory before freezing them for deployment. Runtime decisions adapt through tool choices and arguments, while coding and autonomous improvement occur during evolution.

3.1 Problem setting and objective

  • Deployment represents the policy as Policy APIs, a ReAct runtime policy, and long-term memory; runtime may change tools and arguments but cannot edit source code.The runtime combines current-episode working memory with retrieved lessons from long-term memory.Source: §3.1 · p. 4

3.2 Overview

  • Evolution begins with a fixed capability curriculum, then clusters failures to choose changes; APIs, the harness, and memory are frozen for deployment.Both phases can update these persistent components before deployment.Source: §3.2 · p. 5

3.3 Phase 1: capability curriculum learning

  • On the 41-task transport cohort, Phase 1 recovery raises success from 13/41 (31.7%) to 28/41 (68.3%).Recovery adds retries, failure feedback, and visual correction.Source: §3.3 · p. 5

3.4 Phase 2: autonomous self-evolution

  • 45.0% on LIBERO-PRO and 46.0% on LIBERO-Long, up from 32.8% and 32.0%, respectively, while LIBERO-90 rose to 49/90 (54.4%).Phase 2 starts from the 48/90 Phase 1 system and evaluates candidates against the parent under identical tasks, seeds, and budgets.Source: §3.4 · pp. 5–6

3.5 What evolution produces

  • Runtime pairs atomic Policy APIs with Full and Transport ReAct loops, working and long-term memories, and visual reports; source artifacts are frozen before evaluation.The APIs execute physical operations and return status, measurements, and fresh images for subsequent decisions.Source: §3.5 · p. 6

3.6 Faster ReAct through rejection-sampled fine-tuning

  • 13.2× speedup (0.750 versus 9.895 seconds per call) accompanied a reduction in repeated physical calls from 16 to 4 across 60 paired held-out episodes.Rejection-sampled fine-tuning retains teacher decisions that succeed, satisfy executable schema constraints, and contain valid visual evidence.Source: §3.6 · p. 7

4 Experiments

RACaP is evaluated for in-domain learning, zero-shot transfer, long-horizon execution, cross-embodiment adaptation, and compact ReAct deployment. Results include stronger LIBERO transfer and long-horizon performance than runtime-coding baselines, alongside lower runtime costs and gains from calibration and distillation.

4.1 Experimental setup

  • Evaluation covers LIBERO-90 at seed 0, six LIBERO-PRO suites across 180 paired episodes, and LIBERO-Long across 10 tasks and five seeds.Zero-shot LIBERO-PRO permits no target-domain rollouts, failure logs, or labels to update artifacts; methods share an 8000-step horizon and GPT-5.5 evaluation endpoint.Source: §4.1 · p. 7, §4 · p. 7

4.2 In-domain growth and zero-shot transfer

  • 46.0% Phase 2 success on LIBERO-Long exceeds Phase 1's 32.0% and code-generation baselines' 2.0% to 4.0%.Mean ReAct turns increase from 8.22 to 8.72, while the reported paired test yields p = 0.118.Source: Figure 4 · p. 8, §4.2 · p. 9
  • 49/90 (54.4%) Phase 2 tasks succeed on LIBERO-90, compared with 48/90 (53.3%) for evolution-time Phase 1 and 42/90 (46.7%) for Phase 1 replay.The passage characterizes Phase 2's additional in-domain gain as modest.Source: Figure 4 · p. 8, §4.2 · p. 8
  • 81/180 (45.0%) Phase 2 zero-shot LIBERO-PRO episodes succeed, versus 15/180 (8.3%) to 32/180 (17.8%) for runtime-coding CaP baselines.The result aggregates six perturbation suites; the strongest gains include 73.3% for position changes and 60.0% for task changes.Source: Table 1 · p. 8, §4.2 · p. 8
  • 380 seconds median policy time and 15.9 API calls on LIBERO-PRO for Phase 2 compare with RATS-90's 954 seconds and 45.2 calls.The RATS-90 audit traces failures to interface drift, including mismatched keys, data types, and arguments.Source: §4.2 · p. 8
  • A single calibration trial per task family improves Phase 1 by 8.3 percentage points and Phase 2 by 10.0 points on the seed-1 subset.Code-generation baselines show mixed or negative responses.Source: §4.2 · p. 8, Figure 5 · p. 9

4.3 Cross embodiment evolution and compact ReAct

  • 0.750 seconds per call for the compact student compares with 9.895 seconds for the hosted teacher, while repeated physical calls fall from 16 to 4.Rejection-sampled fine-tuning raises exact teacher-decision agreement from 83.3% to 97.6%.Source: Figure 6 · p. 9, §4.3 · p. 10
  • 11/35 (31.4%) success after domain evolution is achieved by both RACaP-RS and RATS-RS, while both solve 1/10 held-out task types (10.0%).RACaP-RS takes 32.9 seconds and costs $0.089 per episode, versus 90.7 seconds and $0.159 for RATS-RS.Source: §4.3 · p. 9

5 Limitations and Broader Impact

At deployment, frozen source code limits RACaP's generalization to the skills within its Policy API closure.

5 Limitations and Broader Impact

  • Frozen deployment restricts RACaP to generalizing within its Policy API closure, rather than inventing new physical skills online.The authors suggest routing novel failures to a sandboxed coding pipeline as a possible future approach.Source: §5 · p. 10

6 Conclusion

RACaP separates reusable physical mechanisms encoded in code from scene-dependent choices made at runtime. Its architecture uses typed APIs, hierarchical ReAct scopes, visual feedback, and paired evolution to support that boundary.

6 Conclusion

  • RACaP decouples reusable physical mechanisms learned as code during offline evolution from scene-dependent choices managed at runtime.The conclusion characterizes this separation as addressing the evolution-to-execution trade-off in Code as Policy systems.Source: §6 · p. 10

A Formal Objective

  • The interface objective makes the responsibility boundary between evolution and deployment explicit across system layers.Table 2 details architectural ownership across those layers.Source: A Formal Objective · p. 13, Table 2 · p. 13

A.1 ReAct Scopes and Roles

  • Full ReAct orders goals from the complete instruction, while child Transport ReAct handles individual source-to-destination subgoals.Both scopes maintain task-objective-driven working memory and can retrieve read-only lessons from long-term memory.Source: §A.1 · p. 13, Figure 7 · p. 14
  • After physical execution, API reports, measurements, and fresh RGB-D frames inform visual judgments and are logged for closed-loop control.The six typed prompt roles specify visual inputs and structured outputs for planning, mechanism control, and transport diagnosis.Source: §A.1 · p. 13, Table 3 · p. 14

B Evolution Details

  • Phase 2 proposals undergo compilation, basic tests, and simulator validation, and promotion requires a strictly positive success delta over the parent.The capability curriculum is organized across stages S0 to S6.Source: B Evolution Details · p. 15

C.1 Closed-Loop Visual Recovery

A LIBERO-PRO recovery trace shows RACaP adjusting subsequent execution parameters using visual checks.

C.1 Closed-Loop Visual Recovery

  • In the bowl-transfer trace, visual checks guide iterative parameter changes, including a minor push after spatial error falls to 3.2 cm.The trace concerns transferring a bowl from a stove to a plate.Source: §C.1 · p. 15, Table 5 · p. 16

C.2 Interface Abstraction vs. Code Generation

CaP generates robot programs with task-specific coordinates, while RACaP places reusable physical mechanisms behind typed Policy APIs and leaves scene-dependent adjustments to structured calls.

C.2 Interface Abstraction vs. Code Generation

  • CaP scripts segment objects, hardcode explicit spatial coordinates, and issue low-level goto_pose directives.An example hardcodes place_pos as [basket_x, basket_y, basket_top + 0.16].Source: §C.2 · p. 16
  • CaP repairs generate entirely new code blocks, leaving the approach vulnerable to syntax errors and state mismatches.Source: §C.2 · p. 16
  • RACaP encapsulates contact geometry in tested, versioned Policy APIs, letting ReAct adjust grasp strategies and nudges through typed parameters.The APIs contain shape reasoning, clearance calculations, and motion generation.Source: §C.2 · p. 16

D Experimental Details

Experiments span LIBERO and Robosuite evaluations under specified episode and repair limits. An audit finds that interface mismatches account for many RATS-90 failures on LIBERO-PRO.

D Experimental Details

  • Evaluations cover 90 LIBERO-90 tasks, 180 LIBERO-PRO episodes, and 50 LIBERO-Long episodes, with one reset and limits of 8000 simulator steps and 1000 seconds per standard episode.Code-generation baselines receive at most 10 visual feedback repairs; hosted LLM calls use GPT-5.5 via a unified VAPI endpoint at zero temperature.Source: D Experimental Details · p. 16
  • Reporting uses Wilson 95% confidence intervals, paired McNemar tests, and Holm corrections.Source: D Experimental Details · p. 17
  • 181 of 321 RATS-90 failures across the evaluation grid stem from interface mismatches, indicating unconstrained self-play can aggravate failure rates.RATS-90 reaches 8.3% (15/180) on LIBERO-PRO versus 17.8% (32/180) for RATS-base; its largest regression is on spatial perturbations, 0/30 versus 4/30.Source: D Experimental Details · p. 17

D.1 Rejection-Sampled Fine-Tuning

Rejection-sampled fine-tuning distills hosted GPT-5.6-sol ReAct decisions into a local Qwen3-VL-8B-Instruct student, which reduces repeated calls and decision latency in held-out episodes.

D.1 Rejection-Sampled Fine-Tuning

  • 54 teacher decisions were retained and 40 rejected after filtering 24 selected LIBERO-90 training trajectories for execution success, valid schemas, and image-reference integrity.Source: §D.1 · p. 18
  • Training uses role-specific LoRA adapters, freezes base-model weights and the vision encoder, and applies loss only to teacher response tokens.Source: §D.1 · p. 18
  • 0.750 seconds per ReAct decision is the distilled student's average latency versus 9.895 seconds for the hosted teacher proxy; repeated physical calls fall from 16 to 4 across 60 paired held-out episodes.Episodes containing redundant calls decrease from 20.0% (12/60) to 6.7% (4/60).Source: §D.1 · p. 18

Related papers

Loading 2609.29394v1…