PaperlayerView exact revision on arXiv

Source-linked AI summary

CaPtain: A Pipeline for Post-Training Code-as-Policy Robot Agents

Yijia Fan, Xilin He, Yilei Jiang, Dongming Wu, Jiaxu Wang, Xiangyu Yue

arXiv:2610.40799v1cs.RO

TL;DR

CaPtain addresses the lack of a complete post-training recipe for small open VLMs that control robots through low-level code rather than frontier models or delegated skills. It combines an execution harness, synthetic-trajectory SFT, and multi-turn RL guided by a privileged simulator critic; CaPtain-4B improves held-out LIBERO performance and reports transfer across benchmarks, perturbations, and a physical robot.

  • Problem

    Existing CaP agents rely on closed frontier models or delegate manipulation to pretrained VLAs and skills, while small open VLMs fail at direct robot control.

    Source: §1 · p. 2

  • Method

    CaPtain trains a small open VLM over low-level robot APIs with synthetic-trajectory SFT and multi-turn RL using a privileged simulator critic for per-request credit.

    Source: §1 · p. 2

  • Results

    73.8% held-out LIBERO success after RL versus 55.4% after SFT; CaPtain-4B also reports gains across three perturbation suites and transfer to other benchmarks and a physical robot.

    Source: §1 · p. 3, §4.1 · p. 8, §6 · p. 13

  • Takeaways & Limitations

    CaPtain shows that a 4B code-writing policy can be post-trained over low-level manipulation primitives and deployed without privileged state, a VLA, or frontier API calls at test time.

    Source: §6 · p. 13

    RL training uses only 40 LIBERO tasks with a single 7-DoF arm, and tasks requiring bimanual coordination or unfamiliar 6-DoF arm kinematics remain unaddressed.

    Source: §5 · p. 13

Abstract

from arXiv · show

Across coding, web, and tool-use tasks, language-model agents are now routinely trained with the same recipe: supervised fine-tuning (SFT) to learn the interface, then multi-turn reinforcement learning (RL) to master it. Code-as-policy (CaP) casts robot manipulation in the same form, as the policy writes code against robot APIs and reads back the results, so the same recipe should apply. Yet existing CaP agents sidestep training: they either prompt closed frontier models or delegate manipulation to pre-trained vision-language-action models (VLAs) and high-level skills, while small open vision-language models (VLMs) fail outright. We present CaPtain, an open post-training pipeline that turns a small open VLM into a capable CaP agent, with no VLA, frontier model, or privileged state at test time. CaPtain trains and deploys the policy inside CaPtain-Harness, which exposes low-level perception, geometry, and guarded-motion primitives across simulation and real hardware. We first align the base model via SFT on synthetic trajectories generated with GPT-6 Astra, then apply multi-turn RL. Multi-turn RL must credit tens of agent requests, whose physical effects often surface turns later, from a single end-of-episode outcome. We therefore introduce a physical critic that reads privileged simulator state, never seen by the policy, and credits each request by how much it changes the predicted chance of success. Starting from Qwen3.5-4B (0% zero-shot), CaPtain-4B reaches 55.4% held-out LIBERO success after SFT and 73.8% after RL. On LIBERO-PRO, it reaches 78.5%, surpassing several frontier-model agents, and its RL gains persist across three perturbation suites (+17.6 to +21.2 points). Without target-domain training, it attains 83.3% on SimplerEnv WidowX, exceeds frontier models on CaP-Bench, and runs on a real robot after simulation-only adaptation. We release all code, data, and model weights.

1 Introduction

CaPtain addresses the gap between capable CaP agents built on frontier models or delegated skills and small open VLMs that fail at direct robot control. It post-trains a small open model over low-level robot primitives, using a simulator critic for request-level credit, and reports gains across benchmarks and perturbations.

1 Introduction

  • Existing CaP agents rely on closed frontier models or delegate manipulation to pretrained VLAs and skills, while small open VLMs fail at direct robot control.These alternatives limit fine-tuning, onboard deployment, or the agent’s ability to improve manipulation beyond skill sequencing.Source: §1 · p. 2
  • Low-level manipulation spreads consequential decisions across tens of requests, so a single end-of-episode outcome makes credit assignment difficult.A simulator critic can use privileged physical state to estimate how each request changes success likelihood while the policy still acts from cameras at test time.Source: §1 · p. 2
  • CaPtain-Harness exposes 22 low-level perception, geometry, and guarded-motion primitives, with structured receipts, and uses the same policy-facing API across simulation and a physical robot.The policy is trained first on synthetic in-harness trajectories generated with GPT-6 Astra, then with multi-turn RL; the critic is removed at test time.Source: §1 · p. 2
  • 73.8% held-out LIBERO success after RL, up from 55.4% after SFT and 0% zero-shot; CaPtain-4B reaches 78.5% on LIBERO-PRO, above several frontier-model agents.Its gains persist across three perturbation suites, and it transfers to SimplerEnv WidowX, CaP-Bench, and a physical robot after simulation-only adaptation.Source: §1 · p. 3
  • CaPtain provides an open SFT-and-RL pipeline that trains a small open VLM to act as a standalone CaP agent, with released data, training code, and weights.The contribution also includes a portable harness and a simulator critic for dense per-request credit.Source: §1 · pp. 2–3

2 Related Work

Prior robot agents use code with frontier models or richer harnesses, often leaving manipulation to pretrained skills or VLAs. CaPtain instead post-trains a small open VLM directly over low-level perception and motion primitives, while adapting privileged-critic ideas to language-model policies that write code.

Code-as-policy robot agents

  • CaPtain post-trains a small open VLM end-to-end over low-level perception and motion primitives, using camera observations and execution receipts at test time.Unlike systems built around frozen VLAs or reusable skills, its policy makes the manipulation decisions itself.Source: §2 · p. 3
  • Figure 2 contrasts the policy’s multi-request code-and-feedback loop with training that applies SFT followed by RL using a simulator critic.The caption specifies that the critic’s privileged-state credit is used during training and removed at test time.Source: Figure 2 · p. 4

End-to-end robot policies

  • End-to-end robot policies can be sensitive to object, layout, and instruction changes; π0.5 falls from 97% on LIBERO to 12.8% on LIBERO-PRO.The paper studies whether RL gains for a code-writing policy persist across LIBERO-PRO, LIBERO-Plus, and LIBERO-MAX.Source: §2 · p. 4

Credit assignment and privileged critics

  • Prior LLM credit methods use game solvers, acceptance checks, distance estimators, or simulator predicates, while language-model privileged critics do not read physical state.CaPtain adapts asymmetric actor-critic training to a language-model actor that writes code, deriving per-request credit from changes in privileged-physics value.Source: §2 · p. 4

3 CaPtain

CaPtain couples a portable low-level execution harness with in-harness supervised fine-tuning and multi-turn RL. The harness supplies a code-and-receipt interaction loop; a simulator critic uses hidden physical state to assign request-level credit from changes in predicted success.

3 CaPtain

  • CaPtain combines a fixed portable harness, synthetic-trajectory SFT, and simulator-critic-guided multi-turn RL in one interaction environment.The harness defines the action space and feedback loop; SFT aligns the model to them, and RL optimizes closed-loop manipulation and recovery.Source: §3 · p. 4

3.1 CaPtain-Harness

  • Each episode is a ReAct-style sequence of up to 60 requests, with the policy observing the accumulated task and interaction history before emitting code.The harness appends cell results, receipts, and images to the history; the episode ends when the agent stops calling tools or a request, action, or time budget is exhausted.Source: §3.1 · p. 5
  • Structured receipts report physical outcomes such as motion stop reasons, distance, force, jaw opening, detections, positions, and offsets, grounding the policy’s next decisions.Persistent Python variables let the policy retain detections, poses, and helper functions across requests; a host gate checks recorded measurements before accepting finish().Source: §3.1 · pp. 5–6
  • The harness keeps the same policy-facing API across LIBERO, RoboTwin 2.0, RoboDojo, SimplerEnv, CaP-Bench, and a real robot.The API offers 22 low-level capabilities plus finish(), with no grasp, place, or open-drawer skills or VLA; the agent explicitly chooses grasp poses, approach directions, and jaw commands.Source: §3.1 · pp. 5–6

3.2 Stage 1: SFT in the harness

  • SFT uses synthetic in-harness trajectories rendered in the same format as deployment, supervising agent-produced reasoning and code while treating harness outputs as context.The training masks repeated-failure or recovery-loop turns and randomly masks earlier coordinates with probability 0.2 to encourage grounding in the current image.Source: §3.2 · p. 6

3.3 Stage 2: RL with the simulator critic

  • The RL problem is assigning a sparse terminal outcome to requests whose physical effects may appear several turns later.The critic predicts host success from hidden simulator state, task, and request index; per-request temporal-difference credit is the predicted-success change at a fixed time index plus immediate cost.Source: §3.3 · pp. 6–7
  • Because the critic compares values at the same request index, time passing alone earns no credit, and requests that only perceive or verify receive approximately zero immediate credit.The critic reads 36 hidden-state features, is co-trained to track the improving policy, and is not available to the policy at test time.Source: §3.3 · pp. 6–7
  • RL combines within-rollout critic credit with a group-relative outcome term, while costs penalize failed or unnecessary actions and the terminal return distinguishes success, failure, and missed or false finishes.The policy update uses a streaming single-pass procedure; up to eight requests per episode are sampled for the policy gradient, while the critic uses every request.Source: §3.3 · p. 7

4 Experiments

Experiments evaluate CaPtain-4B on held-out LIBERO, perturbation suites, published systems, unseen benchmarks, real-robot tasks, and simulator-critic behavior. RL improves LIBERO and perturbation performance, while transfer results vary across benchmarks and hardware settings.

4 Experiments

  • Evaluations use LIBERO training tasks and held-out or perturbed test sets, with the simulator critic removed at test time and object coordinates estimated online.Episodes allow up to 60 requests, 150 physical actions, and 1,080 seconds; CaPtain-4B is the round-300 checkpoint initialized from the SFT model.Source: §4 · p. 8

4.1 Main results

  • RL raises held-out LIBERO host success from 55.4% to 73.8% and strict success from 52.1% to 67.5%, with gains across perturbation suites.Gains are +19.4 host points on four LIBERO-PRO suites, +17.6 on LIBERO-Plus, and +21.2 Dynamic and +18.8 Base on LIBERO-MAX; six of seven LIBERO-Plus dimensions survive Bonferroni correction.Source: §4.1 · pp. 8–9, Figure 3 · p. 9

4.2 Comparison with published systems

  • On LIBERO-PRO, CaPtain-4B scores 78.5% on the three-suite average, exceeding several frontier-model agents and open systems that do not delegate manipulation to a LIBERO-trained VLA.Its multi-turn RL gain on the four-suite average is 19.4 points, compared with 5.9 points for AgenticRoboRL; the methods start from different baselines.Source: §4.2 · pp. 9–10, Table 2 · p. 10

4.3 Zero-shot transfer

  • On RoboTwin 2.0, CaPtain-4B succeeds in 22.6% of episodes across 50 tasks and 30.0% on the single-arm subset; on RoboDojo it scores 1.7 without target-benchmark training.RoboDojo transfer is from LIBERO’s single Panda arm to dual ARX X5 arms, and the reported primary bottleneck is kinematic reach.Source: §4.3 · pp. 10–11
  • CaPtain-4B averages 83.3% success across four SimplerEnv WidowX tasks and, evaluated zero-shot under CaP-Gym’s standard step horizon, scores 75.0 on CaP-Bench, outperforming cited frontier models.At full episode horizon, it completes 20/20 Cube Lift and 19/20 Cube Stack trials; it does not solve Nut insertion and cannot execute Wipe because guarded-contact primitives prevent sustained downward pressing.Source: §4.3 · p. 11

4.4 Transfer to a real robot

  • After simulation-only adaptation to the XR1 embodiment, the policy completes two real-robot tasks with new objects and layouts using the same low-level primitives as in simulation.The adaptation uses a coarse twin simulation and no real-world data; the hardware backend retains the policy-facing API and receipts, while omitting the critic.Source: §4.4 · pp. 12–13, Figure 5 · p. 12

4.5 Analysis of the simulator critic

  • The simulator critic’s predicted success has 0.970 overall AUC, and its request-level credit concentrates on physical actions rather than perception-only turns.Successful grasps receive median ΔV of +0.058, compared with −0.0002 for perception-only requests; restricting critic inputs to privileged physical state and fixing the time index prevents credit leakage.Source: §4.5 · p. 13

5 Limitations •

CaPtain’s scope is bounded by single-benchmark, single-arm RL training, privileged simulator state for the critic, and the harness capabilities and behaviors covered in SFT.

5 Limitations •

  • RL training uses 40 LIBERO tasks and a single 7-DoF arm; bimanual coordination and tight-reach tasks on unfamiliar 6-DoF arms remain unaddressed.Zero-shot transfer to unseen single-arm tasks and embodiments is reported, but cross-benchmark performance remains below in-domain accuracy.Source: §5 · p. 13
  • Replacing exact simulator poses with noisy estimates lowers critic correlation with privileged ΔV from 0.49 at 2 cm noise to 0.21 using policy perception.The authors state that online RL on physical hardware would require an external state estimator.Source: §5 · p. 13
  • The 22-capability harness and synthetic SFT behaviors bound the action space; guarded Cartesian position primitives cannot express sustained force-controlled wiping.Wipe on CaP-Bench is given as an example requiring sustained downward pressing.Source: §5 · p. 13

6 Conclusion

CaPtain combines a portable low-level harness, synthetic-trajectory SFT, and simulator-critic-guided multi-turn RL to produce a standalone 4B code-as-policy agent. It improves across held-out LIBERO and perturbation suites, transfers to unseen benchmarks and embodiments, and runs on a physical robot after simulated embodiment adaptation.

6 Conclusion

  • CaPtain-4B improves substantially over SFT across held-out LIBERO and three perturbation suites, transfers zero-shot to unseen benchmarks and embodiments, and deploys on a physical robot after simulated embodiment adaptation alone.The conclusion also reports matching or surpassing several frontier-model agents at a small fraction of their inference cost.Source: §6 · p. 13
  • The simulator critic converts sparse episode outcomes into dense per-request credit concentrated on physical manipulation decisions, while the policy uses no privileged information at test time.The pipeline pairs a portable execution harness with synthetic SFT trajectories and multi-turn RL.Source: §6 · p. 13
  • Extending the simulator critic to value termination, bimanual coordination, and multi-embodiment workspaces is presented as a path toward closing the remaining gap to frontier systems.These are proposed extensions, not demonstrated capabilities.Source: §6 · p. 14

A Implementation details

The implementation pairs a persistent code-execution harness and low-level robot API with synthetic SFT and multi-turn RL, using privileged simulator state for per-request credit. It also describes training-data diversity, reward design, infrastructure, and lessons for stabilizing RL.

A.1 Policy and harness

  • The persistent Python session exposes low-level perception, measurement, and motion capabilities; guarded Cartesian actions use IK and collision checks, and receipts report physical outcomes.The same API is used across environments, while a completion gate checks the agent’s own recorded measurements.Source: Robot API · pp. 20–21, Figure 7 · p. 20, §A.1 · p. 20

A.2 RL

  • The physical critic predicts host success from hidden simulator state and assigns request-level temporal-difference credit from changes in predicted success, with perception-only requests receiving near-zero one-step credit.The critic is co-trained online; the same request index is held fixed inside each value difference to prevent elapsed time alone from producing credit.Source: §A.2 · p. 21
  • RL draws fresh starts from a pool of 7,360 states across 40 LIBERO tasks, excludes evaluation states, and trains the policy on at most eight sampled requests per trajectory while the critic uses all requests.The pool includes 5,600 perturbed wide-layout starts and 1,760 official initial states.Source: §A.2 · p. 22

C SFT details

  • Synthetic SFT trajectories from frontier coding agents were rewritten onto four shared helper functions, and the final model was trained for 2,435 steps with the vision tower frozen.Call-site rewrites were checked against a mock robot; about 3.6% of requests were masked from the loss for repeated failures or recovery loops.Source: C SFT details · pp. 22–23

D Diversity of tasks and layouts during post-training

  • Diverse layouts preserve visual grounding: narrow official layouts led models to memorize object positions, whereas wide-layout starts and randomized RL scenes support grounding in current images.Per-object position standard deviation was about 1.4 cm in the earlier data and about 15 cm for wide-layout starts.Source: D Diversity of tasks and layouts… · pp. 23–24

E Reward, costs and early stopping

  • Post-success costs were added after missed finishes rose to about 20% per round, while staged one-shot subgoal rewards provide bounded partial credit on eight hard tasks.The subgoal ladder is judged from hidden snapshots, with partial rewards below the missed-finish return.Source: §A.2 · p. 22, E Reward, costs and early stopping · p. 24

F RL infrastructure

  • Infrastructure optimizations reduced training-round time from 1,812 s to 1,246 s and increased throughput from 520 to 824 episodes per hour.Optimizations were retained only if they passed numerical gates intended to keep the trained objective unchanged.Source: F RL infrastructure · pp. 24–25

G Lessons from RL on a multi-turn agent

  • The authors identify large batches, fresh initial states, layout diversity, excluding policy-behavior counters from the critic, and a dedicated termination signal as lessons for multi-turn RL.In particular, intermediate-state starts after successful demonstrations led the policy to learn only to place already-held objects.Source: §A.2 · p. 21, G Lessons from RL on a multi-turn agent · p. 25

H Gradient-noise analysis

Gradient-noise estimates motivate accumulating four RL rounds per update, while late training reveals a divergence between host and strict success. The resulting over-verification drift leads the authors to select the round-300 checkpoint.

H Gradient-noise analysis

  • The 4B model’s early-checkpoint noise scale is about 9,500 episodes, motivating accumulation of K=4 rounds, or 1,024 episodes, per update.Without accumulation, consecutive updates had near-zero cosine similarity.Source: H Gradient-noise analysis · p. 25, Table 7 · p. 25

I Host and strict success late in RL

  • After about round 340, host success stays flat while strict success falls; the authors therefore select the round-300 checkpoint.Host success remains about 65%, while strict success drops from about 60% to 44% between rounds 300 and 400.Source: I Host and strict success late in RL · p. 25, Figure 8 · p. 26
  • After solving a task, the policy over-verifies instead of calling finish(), while the critic gives the stop decision no credit.A missed finish still earns 0.25, the post-success cost is capped, and the critic assigns V ≈1 to post-success states.Source: I Host and strict success late in RL · p. 25
  • An earlier run showed the same failure, doubling missed finishes and returning strict LIBERO-Plus success to the SFT level.The authors suggest giving the decision to finish its own target, such as success-and-finished.Source: I Host and strict success late in RL · pp. 25–26

J Failure taxonomy for zero-shot transfer

The first-blocker analysis identifies coordination, reach, and request-budget failures as prominent transfer barriers. A training-aligned RoboTwin variant did not outperform the official layout.

J Failure taxonomy for zero-shot transfer

  • 0/35 RoboTwin bimanual tasks succeed; RoboDojo has 21/46 reach or inverse-kinematics first blockers, while request-cap exhaustion often follows re-verification.In an early evaluation, 7 of 11 host successes never called finish; the round-400 checkpoint called it in only 13–15 of 250 episodes.Source: J Failure taxonomy for zero-shot… · p. 26
  • A training-aligned RoboTwin variant was not better than the official layout (p=0.86), suggesting viewpoint and lighting are not the main constraint.The variant used a LIBERO-like camera, reachable layouts, and LIBERO brightness.Source: J Failure taxonomy for zero-shot… · p. 26

K Additional LIBERO results

Additional LIBERO results show paired gains across suites and improved success on a clean LIBERO-PRO subset using an initial state absent from training. A task predicate and request cap also constrain interpretation of some results.

K Additional LIBERO results

  • +18.3 spatial, +15.0 object, +26.7 goal, and +13.3 long are CaPtain-4B’s per-suite paired gains over SFT (host).The corresponding p-values are 10−3, 0.035, 1.4×10−4, and 0.13, respectively.Source: K Additional LIBERO results · p. 27
  • 74.6% (3 suites) and 66.4% (all 4 suites) host success compare with SFT’s 57.8% and 47.4% on 78 LIBERO-PRO Pos+Task episodes.The subset uses initial state 11, which was never used in training.Source: K Additional LIBERO results · p. 27
  • On the correct flat placements, every model scores 0/6; raising the request cap from 60 to 120 on libero_10 leaves success unchanged, at 17 versus 18/60 for an earlier checkpoint.Source: K Additional LIBERO results · p. 27
  • Raising the request cap from 60 to 120 did not change libero_10 success: 17 versus 18/60 for an earlier checkpoint.Source: K Additional LIBERO results · p. 27
  • LIBERO-PRO Pos+Task baseline protocols vary, including oracle signals, play experience, privileged geometry or depth, and different trial budgets.Source: K Additional LIBERO results · p. 27

L Harness adaptation per transfer benchmark

The transfer appendix distinguishes generic from benchmark-specific harness changes, reports substantial training compute, and illustrates policy behavior through simulation and real-robot traces.

L Harness adaptation per transfer benchmark

M Compute accounting

  • RL to CaPtain-4B used about 1,290 H100-hours, compared with 120 H100-hours for SFT and about 16 GPU-hours for critic warm start.A round took 812 seconds before staged reward and 991–1,053 seconds afterward on 16 H100s.Source: M Compute accounting · p. 28

N Example episode

  • In a 14-request bowl-placement episode, the policy switches to the wrist camera under occlusion and checks the final relation against the task tolerance before finishing.Fresh measurements showed an offset of about 1.7 cm, within the required 2 cm tolerance.Source: N Example episode · pp. 28–29

O Real-robot episode traces

P Zero-shot transfer episodes

The section traces a cup-placement episode and a block-stacking episode, showing geometry-based manipulation and recovery from a guarded stop.

P Zero-shot transfer episodes

  • CaPtain-4B grasps the cup at its rim using measured geometry, carries it by the measured cup-to-coaster offset, and checks placement from a fresh image.The episode includes distractors around the cup and coaster.Source: Figure 9 · p. 31, P Zero-shot transfer episodes · p. 31
  • 6.7 mm: CaPtain-4B’s measured cup-to-coaster offset is below the task’s 2 cm tolerance after placement.It remeasures from a fresh image before calling finish().Source: P Zero-shot transfer episodes · p. 31
  • After a guarded descent stops short on the arm’s force limit, CaPtain-4B recovers, changes the gripper angle, descends more slowly, and stacks the blocks.Fresh images and measurements verify the final stack before finish().Source: P Zero-shot transfer episodes · pp. 31–32

Q Full LIBERO-family comparison

The LIBERO-family comparison reports strong standard-benchmark results for many demo-trained VLAs and WAMs, but lower perturbation retention; CaPtain-4B retains 90% without LIBERO demonstrations.

Q Full LIBERO-family comparison

  • 90%: CaPtain-4B retains 90% of standard LIBERO success on LIBERO-Plus without LIBERO demonstrations, versus 20–87% retained by most demo-trained VLAs and WAMs.The table defines Plus/LIBERO as the percentage of standard LIBERO success retained under perturbation.Source: Q Full LIBERO-family comparison · p. 32, Table 14 · p. 33
  • 94–99%: most VLAs and WAMs reach this success range on standard LIBERO after fine-tuning on LIBERO demonstrations.Source: Q Full LIBERO-family comparison · p. 32
  • The comparison covers standard LIBERO, LIBERO-PRO, LIBERO-Plus, and LIBERO-MAX, with baseline protocols differing in episode counts and evaluation scope.For example, the supplied protocols use a stratified 700-case LIBERO-Plus sample for CaPtain-4B and the full 10,030 instances for baselines.Source: Q Full LIBERO-family comparison · p. 33
Loading 2610.40799v1…