PaperlayerView exact revision on arXiv

Source-linked AI summary

RE-0: Verified Recursive Improvement of Embodied Code-as-Policy Agents through Local On-Policy Distillation

Jiawei Zhang, Xiangrong Zhang, Rui Song, Huanbin Zhou, Chengye Song, Hongzhou Wang

arXiv:2609.32416v1cs.ROcs.AIcs.LG

TL;DR

Code-as-Policy agents need corrective credit on student-induced histories because execution errors alter later states and observations. RE-0 verifies bounded teacher repairs and distills positive-credit interventions into a standalone student. Across tasks and physical deployment, verified rows improve performance, with verified corpus selection identified as the load-bearing mechanism.

  • Problem

    Code-as-Policy errors alter later environment states and observable histories, leaving open where corrective credit should be assigned along the student’s actual execution history.

    Source: §1 · p. 1

  • Method

    RE-0 localizes student failure boundaries, verifies bounded teacher repairs against student behavior from the same snapshot, and recursively distills verified interventions.

    Source: §1 · p. 2

  • Results

    Across six long-horizon tasks and physical deployment, 3–67 verified rows lift the base from 4–68% to 62–100%, and verified corpus selection—not weighting alone—is load-bearing.

    Source: §5 · p. 9

  • Takeaways & Limitations

    Verified corpus selection, rather than weighting alone, is the load-bearing mechanism identified by the paper’s selection ablations.

    Source: §1 · p. 2

    The improvement guarantee is conditional on bounded returns, valid contracts, simultaneous coverage, and bounded occupancy-weighted projection error.

    Source: §3.2 · p. 5, §3.4 · p. 6

Abstract

from arXiv · show

Code-as-Policy agents accomplish long-horizon embodied tasks by generating and executing code, yet continually improving them with teachers that are stronger but not globally reliable remains a key challenge. Existing distillation methods typically treat the teacher's complete behavior as the supervision target and thus misassign training credit on states where the teacher fails. We propose RE-0, a recursively verified policy improvement framework: rather than assuming that the teacher globally outperforms the student, RE-0 requests local corrections from the teacher on the student's own failure histories and checks in the environment whether each correction is genuinely beneficial; verified corrections yield immediate improvement. Building on this, we propose RE-OPD, which turns verified interventions into supervision for on-policy distillation. Only counterfactually verified teacher interventions provide distribution-level supervision, weighted by their measured local benefit, and the improvement they induce is projected back into the standalone student, so both where supervision is applied and how much credit the teacher receives co-evolve with the student policy. We further prove that the student's per-round gain is lower-bounded by its verified intervention gain up to verification and projection error terms. Experiments across multiple Code-as-Policy embodied tasks show that RE-0 improves both teacher-assisted execution and the standalone student, and generalizes to novel robots and scenes.

1 INTRODUCTION

Code-as-Policy errors depend on execution history, so improvement requires assigning corrective credit on the student’s own visited histories. RE-0 verifies local teacher repairs and recursively distills accepted improvements into a standalone student.

1 INTRODUCTION

  • Code-as-Policy errors are history-dependent: a harmful segment can alter the environment state, program namespace, and subsequent observable history.The central question is where along the student’s execution history corrective credit should be assigned.Source: §1 · p. 1
  • Global teacher strength does not determine whether a local intervention deserves credit: even a strong teacher can err, while a weaker one may decide better at a student failure boundary.Existing approaches do not establish whether a concrete intervention on a real student failure made a positive causal contribution.Source: §1 · p. 1
  • The framework couples localization of harmful boundaries, same-state comparison of student and correction, and absorption of verified assistance into a teacher-free student.These challenges motivate verifier-mediated local teacher credit assignment.Source: §1 · pp. 1–2
  • RE-0 localizes the earliest candidate harmful boundary, verifies a bounded teacher correction against the student from the same snapshot, and distills only positive-credit interventions.The next student regenerates trajectories, so supervision locations are re-localized as the policy changes.Source: §1 · p. 2
  • The paper contributes verifier-mediated local credit, RE-OPD with a finite-sample improvement guarantee, and evidence that verified corpus selection—not weighting alone—is load-bearing.The evidence spans six long-horizon tasks and physical deployment.Source: §1 · p. 2

2 RELATED WORK

Prior embodied code-agent work emphasizes planning, repair, or program search, while interactive imitation and on-policy distillation learn on student-induced states. RE-0 instead verifies local teacher credit on those states.

2 RELATED WORK

  • Embodied code-agent methods use affordance-constrained planning, environment feedback, code structure, or execution traces, while newer work repairs, searches, or evolves programs.RE-0 focuses on amortizing execution-time corrections into a teacher-free student.Source: §2 · p. 2
  • Interactive imitation and on-policy distillation train on learner- or student-generated states, but RE-0 does not presume teacher reliability on queried histories.It estimates local counterfactual advantage against student behavior from the same simulation snapshot.Source: §2 · p. 2
  • Agentic correction methods commonly revise whole answers, select candidates by outcomes, or feed execution traces back as prompts; RE-0 localizes a harmful boundary and returns control after a minimal verified macro-action.The student regenerates the suffix after the intervention.Source: §2 · pp. 2–3
  • Prior selective-supervision methods query experts or weight demonstrations, whereas RE-0 defines credit as a teacher’s conditional behavior on a student history evaluated by paired same-state counterfactual advantage.The verifier determines whether the teacher distribution earns local training credit.Source: §2 · p. 3

3 RE-0: RECURSIVE VERIFIED POLICY IMPROVEMENT

RE-0 verifies bounded teacher repairs on student-induced histories, admits only positive local credit, and distills the resulting assistance into a standalone student. The recursion’s guarantee is conditional on verification and projection assumptions.

3 RE-0: RECURSIVE VERIFIED POLICY IMPROVEMENT

  • RE-0 assigns credit to a teacher’s local correction relative to the current student on a specific student-induced history, without assuming global policy dominance.The correction takes over only a minimal executable segment before control returns to the student.Source: §3 · p. 3

3.1 LOCAL TEACHER CREDIT ON STUDENT-INDUCED HISTORIES

  • The ideal allocation accepts supervision only on histories with positive counterfactual advantage, but exhaustive verification is infeasible, so diagnostics localize candidate boundaries for testing.The advantage compares intervention with continuing the student at that history.Source: §3.1 · p. 4

3.2 VERIFIED LOCAL ASSISTANCE

  • RE-0 finds the earliest candidate failure boundary, saves its environment snapshot, and tests bounded repair contracts in increasing scope before selecting a verified repair.The smallest verified scope is minimal by scope, not necessarily by candidate-string length; paired runs use shared randomness and return control to the student.Source: §3.2 · pp. 4–5

3.3 RE-OPD: AMORTIZING LOCAL CREDIT INTO THE STUDENT

  • RE-OPD projects the hybrid policy into an independent student using a retention term and teacher-projection term weighted by verified local credit.The recursion’s per-round gain is lower-bounded by verified intervention gain, subject to verification and projection error; the outer criterion guards against retaining a regressed generation.Source: §3.3 · pp. 5–6

3.4 COLD START AND RECURSIVE REFINEMENT

  • Cold start verifies teacher-seat failure histories to form a training corpus, while refinement uses the student’s occupancy; recursion halts when the gate closes or the target performance is reached.Useful refinement requires both remaining headroom and verifiable teacher effect; the guarantee also relies on bounded returns, valid contracts, simultaneous coverage, and bounded projection error.Source: §3.2 · p. 5, §3.4 · p. 6

4 EXPERIMENTS

Across six long-horizon robosuite tasks, RE-OPD uses verified teacher interventions to improve student success, while verification also determines where training occurs. Experiments assess credit selection, ablations, transfer, and physical deployment.

4.1 EXPERIMENTAL SETUP

  • The six robosuite tasks span single- and dual-arm manipulation, with held-out evaluation on 50 seeds per task.The base model ranges from 4% to 100% success, and collection seeds are disjoint from held-out seeds.Source: §4.1 · pp. 6–7

4.2 MAIN RESULTS

  • 3–67 verified rows lift base success from 4–68% to 62–100%, with gains varying by task and row count.Restack reaches 62% with 3 rows, lift 76% with 7, and nut 88% with 27; improvement is not monotone in row count.Source: §4.2 · p. 7
  • When projection regresses on lift, the outer acceptance criterion halts recursion; spill abstains when already at 100%, while restack refines toward its 80% target.Lift falls from 76% to 58% after a thinly credited projection; the regression replicates at 76%→46% in another training seed.Source: §4.2 · pp. 7–8

4.3 VERIFICATION AND CREDIT ANALYSIS

  • Admissions depend on verified local benefit rather than headroom alone: admitted events separate from abstentions, and the threshold sweep leaves the admitted set unchanged.The gate also rejects harmful interventions, including an API patch that dropped baseline success from 8/16 to 1/16.Source: §4.3 · p. 8, Figure 3 · p. 8
  • GRPO with terminal rewards reaches 0–1/50 on non-saturated tasks and undergoes entropy collapse, while weight-scale and normalization changes cannot recover verified-corpus gains.The weight-scale and normalization ablations can also induce catastrophic forgetting.Source: §4.3 · pp. 8–9

4.4 ABLATIONS

  • On lift, LCB admission outperforms unfiltered and sign-only selection across three trainer seeds, while success-filtered restack demonstrations remain at base versus 46/50 for verified patches.The restack comparison uses three verified event patches; outcome filtering selects the same corpus on that task.Source: §4.4 · p. 9, Table 2 · p. 9
  • Removing the student prefix or retention term collapses the restack gain, while student self-bootstrap yields zero admissions on four tasks.The objective-composition comparison uses a separate 50-seed cohort.Source: §4.4 · p. 9

4.5 GENERALIZATION AND REAL-ROBOT DEPLOYMENT

  • Transfer is selective: restack training improves nut, spill wiping dynamics do not transfer, and pooled multi-task training causes substantial interference.Teacher-free policies also run zero-shot on a dual-arm AgileX PiPER platform without hardware-specific policy edits, with successes ranging from 2/20 on lift to 20/20 on spill and cubestack.Source: §4.5 · p. 9

5 CONCLUSION

The conclusion presents RE-0 as a locate–verify–weight recursion and reports results across six tasks and physical deployment. It also records AI-use, responsibility, and reproducibility disclosures.

5 CONCLUSION

  • 3–67 verified rows lift the base from 4–68% to 62–100% across six tasks, while the same admission ledger supports self-termination.The conclusion identifies verified corpus selection, rather than weighting alone, as the load-bearing mechanism, based on selection ablations across three trainer seeds.Source: §5 · p. 9
  • RE-0 verifies minimal teacher patches through same-snapshot counterfactual experiments, admits them with finite-sample confidence, and returns them to the student after one bounded macro-action.The conclusion describes this as a locate–verify–weight recursion.Source: §5 · p. 9
  • The paper reports using generative AI for methodology feedback, implementation, result interpretation, and translation, but not to generate data or prove mathematical claims.Authors report re-verifying AI-assisted result interpretation against raw artifacts and extracting every reported number from logged evaluation runs.Source: AI USE STATEMENT · p. 10
  • The authors take responsibility for the final content and report cross-checking numbers against raw logs and testing AI-assisted code.They also state that AI-assisted text was reviewed and revised by the authors.Source: AI USE STATEMENT · p. 10
  • The authors plan to release the pipeline, six-task environment configurations, 110 audited events, verified training data, evaluation JSONs, and training configurations.They state that reported numbers are traceable to the released artifacts through a manifest.Source: REPRODUCIBILITY STATEMENT · p. 10

A LIMITATIONS

The authors identify unresolved supervision allocation, verification cost, limited evaluation scope, experimental-protocol caveats, and a validation-cohort reuse in stage-entry decisions.

A LIMITATIONS

  • The allocation layer remains open: LCB weighting suppresses patch weights, normalization can fail to flip and destroy composition, and unprotected distillation forgets.The authors call for an allocation theory under composition constraints and contribute a stress test and η/ν criteria.Source: A LIMITATIONS · p. 11
  • The teacher is black-box, calibration uses an API frontier model, and paired-counterfactual cost grows linearly with audited events.The reported conclusions cover six robosuite families and one physical platform, with selective transfer.Source: A LIMITATIONS · p. 11
  • One physical platform and six robosuite families bound the evaluation scope; transfer is selective.The authors do not claim broader platform or scene coverage in this limitation statement.Source: A LIMITATIONS · p. 11
  • The π1.5 row has a collection-seat artifact, ablation pools differ from the main-table protocol, and the normalization arm is a single run.Selection-ablation arms are single runs per seed, with three seeds on lift.Source: A LIMITATIONS · p. 11
  • Immediate handback to the student after each verified patch rests on a design argument; RFT/STaR remains inferred from ablations.The authors cite the gate-off ablation and self-imitation arm for that inference.Source: A LIMITATIONS · p. 11
  • Stage-entry decisions reused the evaluation cohort as validation, so validation and test were not fully separated for those decisions.The authors say a disjoint-cohort re-evaluation reproduces base/cold conclusions; the reuse mattered for thin margins, including lift at 76% against ρ=80%, which was rejected on remeasurement.Source: A LIMITATIONS · p. 11

B PROOFS AND STATISTICAL GUARANTEES

The appendix develops finite-sample guarantees for admitting local teacher interventions and projecting their gains into the student, alongside conditions for safe abstention and the scope of the analyzed process.

B PROOFS AND STATISTICAL GUARANTEES

  • The trajectory-return bound assumes a T-step horizon and rewards bounded by R; the induction proof uses the Markov property of the macro-action history process.The proof proceeds by induction on T.Source: B PROOFS AND STATISTICAL GUARANTEES · pp. 12–13
  • Contract selection chooses the smallest-scope candidate with a lower confidence bound above ε, subject to execution-safety, program-context, snapshot-restoration, and execution-identity checks.The paired estimates compare teacher candidates with student resamples under shared randomness.Source: B PROOFS AND STATISTICAL GUARANTEES · p. 12
  • Simultaneous confidence coverage across audited histories preserves the per-history bound despite admission depending on the measured lower confidence bound.The deployed verifier uses an anytime-valid e-process; the theorem’s analytical Hoeffding radius is unaffected by that operational choice.Source: B PROOFS AND STATISTICAL GUARANTEES · p. 12
  • Theorem 1 lower-bounds each projected student gain by the verified intervention gain minus verification and projection error terms, under bounded returns, valid contracts, simultaneous coverage, and bounded projection error.The proof decomposes the student’s per-round change into the execution-time gain and the cost of projecting the mixture into the student.Source: B PROOFS AND STATISTICAL GUARANTEES · p. 13
  • When teacher and student share a tokenizer, the trajectory-level KL divergence decomposes into per-token conditional KL terms, including termination.The appendix also invokes Pinsker’s inequality and Cauchy–Schwarz to bound projection error.Source: B PROOFS AND STATISTICAL GUARANTEES · p. 13
  • Cold-start distillation targets completion-conditioned occupancy, whereas later allocation uses local counterfactual credit rather than trajectory outcomes.The completion-conditioned target is the cold-start case of credit weighting; the ideal allocation is a fractional-knapsack problem, with localization and LCB admission serving as a practical proxy.Source: B PROOFS AND STATISTICAL GUARANTEES · pp. 13–14
  • If local advantage is at most ε almost everywhere, the gate closes, leaving both execution and training without teacher contribution.In this case the mixture matches the student per history and the teacher training weight is zero.Source: B PROOFS AND STATISTICAL GUARANTEES · p. 14
  • The guarantee analyzes a two-pass post-rewind process: both paired continuations start from the same snapshot, but it does not guarantee a causal online policy in the original environment.The diagnostic evidence selecting an intervention is observed after the boundary in the original rollout; snapshot-free hardware deployment is outside this guarantee.Source: B PROOFS AND STATISTICAL GUARANTEES · p. 14

C EXPERIMENTAL DETAILS AND REPRODUCIBILITY

The appendix specifies the training and evaluation configuration and reports the auditing workload across six tasks.

C EXPERIMENTAL DETAILS AND REPRODUCIBILITY

  • Students use Qwen3.8-27B with LoRA for 120 training steps; refinement warm-starts from the prior generation at a lower learning rate.Evaluation uses 50 held-out seeds per task, disjoint from collection, and admission requires the two-layer LCB to exceed ε under the simultaneous budget.Source: C EXPERIMENTAL DETAILS AND… · p. 14
  • 110 audited events yielded 19 admitted events; selected candidates consumed 4,192 suffix rollouts, with a median of 12 shared-randomness pairs per event.Rejected candidates and the live first pass add further environment rollouts, so verified rows are not rollout-free.Source: C EXPERIMENTAL DETAILS AND… · p. 14

D MAIN-RESULT ACCOUNTING AND ROBUSTNESS

Per-round sign accounting aligns admitted intervention credit with standalone performance changes in all four decided rounds, including a rejected round. Replications support most cold-start gains while identifying spill as seed-sensitive.

D MAIN-RESULT ACCOUNTING AND ROBUSTNESS

  • The base row uses a 50-seed cohort disjoint from collection, while the restack refine cell uses an intermediate teacher-seat recollection.A collection-seat artifact affecting that refine cell is disclosed in the Limitations.Source: D MAIN-RESULT ACCOUNTING AND ROBUSTNESS · p. 14
  • 76%→58% followed one admitted lift event (w=0.069) plus replay, with spill falling from 28 to 0 on the branch.Re-running the rejected lift round with training seed 2 reproduced a regression from 76% to 46%.Source: D MAIN-RESULT ACCOUNTING AND ROBUSTNESS · p. 15
  • On restack composition arms, plain SFT on eight success-filtered teacher-seat demonstrations scored 3/50, versus 46/50 for three unweighted outcome-selected patches.All nine patches rejected by the outcome rule had intervention success 0.0; both arms were single runs.Source: D MAIN-RESULT ACCOUNTING AND ROBUSTNESS · p. 15
  • All four retain-or-halt rounds show matching signs between admitted credit and realized standalone change, including the rejected round.This is sign-level consistency evidence, not an estimate of the projection-error term; the retained generation was measured better in every round.Source: Table 3 · p. 15, D MAIN-RESULT ACCOUNTING AND ROBUSTNESS · p. 15
  • Restack cold-start replicates scored 31/50, 44/50, and 36/50 against a 4/50 base, while spill remained seed-sensitive.Spill scored 50/50, 33/50, and 33/50 against a 34/50 base; lift, nut, and door replicates were reported far above their respective bases.Source: D MAIN-RESULT ACCOUNTING AND ROBUSTNESS · p. 15
  • On a disjoint never-used 50-seed cohort, base and cold checkpoint results matched reported cells within binomial noise.The reported base values were 3/3/5/47/50 and cold values 28/45/42/50/49 for the listed tasks.Source: D MAIN-RESULT ACCOUNTING AND ROBUSTNESS · p. 15

E MECHANISM AND ABLATION DETAILS

The ablations distinguish admission rules and corpus selection from weighting: the LCB gate abstains on unsupported interventions, while teacher capability varies sharply at the contract layer.

E MECHANISM AND ABLATION DETAILS

  • Four lift events admitted by the sign rule but rejected by the LCB had positive mean advantages indistinguishable from zero.Their b∆ values were +0.21, +0.06, +0.04, and +0.00004 at M values 48, 48, 48, and 12, respectively.Source: E MECHANISM AND ABLATION DETAILS · p. 16, Table 5 · p. 16
  • The spill sign arm scored 0/50 with zero reward as all rollouts repeated the same three-block program without wipe coverage.The LCB admitted nothing on spill or nut; true base-seat RFT was infeasible, and the nearest self-imitation analogue reached 2/50.Source: E MECHANISM AND ABLATION DETAILS · p. 16
  • The strong API teacher earned 4 admissions on 13 events with rescue rates of 19–47%, while local experts failed at the contract-format layer.Those narrow task experts had own-task win rates of 62–100% but a ∼10% rescue ceiling.Source: E MECHANISM AND ABLATION DETAILS · p. 16, Table 6 · p. 16

F TRANSFER, POOLING, AND DEPLOYMENT

The section reports cross-task transfer, a restack own-task result, pooled-training failure modes, and physical deployment on a dual-arm robot platform.

F TRANSFER, POOLING, AND DEPLOYMENT

  • The transfer table reports cross-task success counts out of 50 at 120 steps, but the supplied passage gives no cell values.The table caption identifies the scope and evaluation horizon only.Source: Table 7 · p. 16
  • The restack own-task transfer cell for π1/π′2 is 31/40.Source: F TRANSFER, POOLING, AND DEPLOYMENT · p. 16
  • Zero-shot deployment uses a dual-arm AgileX PiPER rig, and the deployed policy code needs no hardware-specific edits.The execution stack exposes the simulation API surface unchanged and uses SAM3-based open-vocabulary perception.Source: F TRANSFER, POOLING, AND DEPLOYMENT · p. 16, Figure 5 · p. 17
  • Pooled joint training fails through two stated mechanisms: task interference and quota starvation.The table caption reports successes out of 50, without supplied cell values.Source: Table 8 · p. 17

G BASELINE DETAILS

The GRPO control was tuned to train within the available stack, but the comparison is resource-matched rather than maximally tuned.

G BASELINE DETAILS

  • GRPO used a 3072-token response limit, disabled thinking mode and KL regularization, and required thread-safety and liveness fixes to train in this stack.Entropy collapse from 0.30 to 0.002 persisted across these fixes.Source: G BASELINE DETAILS · p. 17
  • On the shared 50-seed protocol, GRPO scored 1/50 on restack, 0/50 on spill, 0/50 on lift, and 50/50 on cubestack.The corresponding base results were 4/34/2/50; nut was omitted because of a serving-stack incompatibility.Source: G BASELINE DETAILS · p. 17
  • The authors cannot rule out different behavior with a larger-memory configuration and KL regularization, so this is not a maximally tuned GRPO comparison.Source: G BASELINE DETAILS · p. 17

Related papers

Loading 2609.32416v1…