PaperlayerView exact revision on arXiv

Source-linked AI summary

Co-Evolving Robot Orchestrators and Policies through Deployment

Xilun Zhang, Maggie Wang, Erik Bauer, Hong-Xing Yu, Huang Huang, Jiajun Wu, Marco Pavone

arXiv:2610.09228v1cs.ROcs.SYeess.SY

TL;DR

Robot orchestrators can route around VLA failures but cannot overcome a policy bottleneck, and policy updates can leave the orchestrator’s memory out of date. Robo-COP co-evolves policy and orchestrator during deployment by curating skill demonstrations, training when warranted, and verifying candidate policies. Mean held-out success rises in simulation and on real-world tasks compared with a frozen-policy harness.

  • Problem

    An orchestrator built around a limited-steerability, frozen VLA can route around its failures but cannot fix them, while policy updates alone leave its memory tuned to old behavior.

    Source: §1 · p. 1

  • Method

    Robo-COP curates skill demonstrations from deployment attempts, trains when data warrants it, verifies candidate policies before adoption, and revises policy-dependent memory.

    Source: §1 · p. 2, §3.3 · p. 5

  • Results

    Across ten simulated tasks, mean held-out success was 73.8% versus 64.8% with a frozen-policy harness; across three real-world tasks, it was 50.0% versus 38.3%.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Deployment experience can support co-evolution while verification allows the system to reject candidate updates rather than adopt every fine-tuned checkpoint.

    Source: §5 · p. 9

    Robo-COP learns only skills its policy and scripted tools already demonstrate at least partially, and scalability remains to be validated on larger experiments.

    Source: §5 · p. 9

Abstract

from arXiv · show

Vision-language-action (VLA) policies trained on large datasets are capable within their training domains, yet they still fail to generalize to the variety of situations a robot meets in real-world deployment. Agentic robot systems complement the policy with a vision-language model (VLM) orchestrator that learns when to call the policy, how to instruct it, and when to use scripted skills instead. However, because the harness is built around a frozen policy that has limited language steerability, the orchestrator can avoid the policy's failures but never overcome them. The policy becomes the bottleneck of the whole system. Fine-tuning the policy can remove this bottleneck, but updating it alone decouples it from an orchestrator tuned to its old behavior. We propose Robo-COP, in which the orchestrator and policy co-evolve during deployment. Robo-COP curates skill demonstrations from its own executions, fine-tunes the policy when this data can address recurring failures, and adopts each new policy only after it improves the skills it was trained for. Across ten simulated RoboLab tasks, Robo-COP raises mean held-out success from 64.8% to 73.8% over the same harness with a frozen policy, while fine-tuning on a fixed schedule without verification reaches only 65.8%. On three real-world tasks, Robo-COP raises held-out success from 38.3% to 50.0%. Robo-COP turns deployment into a self-improving flywheel in which robots learn by doing, with each improvement in execution producing better data for the next round of learning. Videos and code are available at https://robo-cop.pages.dev/.

1 Introduction

Robo-COP addresses the policy bottleneck in deployment-time robot orchestration by co-evolving the orchestrator and VLA policy. It curates skill demonstrations from task attempts, trains when data warrants it, and verifies candidates before adoption; held-out success improves in simulation and on real-world tasks.

1 Introduction

  • Limited language steerability means an orchestrator can route around VLA failures but cannot fix them, leaving the policy as the system’s performance bottleneck.Fine-tuning could raise that ceiling, but failed episodes may contain useful correct skills, candidates may regress, and updating policy alone leaves memory tuned to old behavior.Source: §1 · pp. 1–2
  • Robo-COP co-evolves the orchestrator and policy during deployment, using scripted primitives and VLA calls while adapting policy-dependent memory.It learns during task attempts rather than using a separate exploration phase.Source: §1 · p. 2
  • The evaluation spans ten simulated tasks across three capability regimes and three real-world tasks.The simulated regimes cover tasks solved by the pretrained VLA, scripted primitives, or both.Source: §1 · p. 2
  • Deployment executions become skill demonstrations, retaining successful skills even from failed episodes and training only when the data warrants it.Each candidate is evaluated on target skills, overall task success, and previously admitted skills before adoption; memory is revised after policy updates.Source: §1 · p. 2
  • Robo-COP raised mean held-out success over a frozen-policy harness from 64.8% to 73.8% in RoboLab simulation and from 38.3% to 50.0% on real-world DROID.Source: §1 · p. 2

2 Related Work

Prior work adapts how fixed capabilities are used, gathers or curates policy-training data, or improves policies during deployment. Robo-COP combines adaptive orchestration with policy updates and revises harness memory when the policy changes.

2 Related Work

3 Method

Robo-COP logs deployment actions, curates skill-level demonstrations, decides when training is warranted, and verifies candidate policies before adoption. After policy selection, it revises memory so the orchestrator can reassess policy-dependent advice.

3 Method

  • Robo-COP updates both the VLA policy and adaptive harness, curating logged policy and scripted actions into skill demonstrations and revising memory after policy verification.The method couples deployment execution, experience curation, candidate verification, and memory revision.Source: §3 · p. 3

3.1 Adaptive policy orchestration

  • The orchestrator combines scripted primitives, perception functions, and VLA calls, reflects on outcomes, and credits successful skills even when a later step fails.It records task memory, and scripted and VLA actions can be unified into a continuous action stream for training.Source: §3.1 · pp. 3–4

3.2 Experience curation and policy improvement

  • Skill curation keeps segments only when text and video judges confirm success or clear partial progress, while the controller waits for sufficient data before training.Segments are assessed independently of episode outcome, so a successful pickup may be retained even if the object is later dropped; the controller reviews demonstrations, performance, memory, failures, and earlier attempts.Source: §3.2 · pp. 4–5

3.3 Policy verification and integration

  • Candidate verification checks target skills and admitted-skill regressions while retaining the incumbent as fallback, then adopts only candidates meeting all requirements.Insufficient calls make verification starved; failed checks revert to the incumbent. After adoption or reversion, memory advice against the VLA is marked unverified for the current checkpoint rather than deleted.Source: §3.3 · pp. 5–6

4 Experiments

Experiments compare Robo-COP with policy and orchestration baselines across ten simulated tasks and three physical tasks. Robo-COP improves mean held-out success in both settings, while verification guides when updates are adopted.

4.1 Experimental setup

  • Experiments cover ten randomized RoboLab tasks across three capability regimes, with matched deployment trials and shared held-out initializations for baseline comparisons.Frozen-policy, fixed-schedule, and Robo-COP each run 100 deployment trials per task and are evaluated on 50 held-out initializations; the baselines isolate memory adaptation and unverified scheduled updates.Source: §4.1 · pp. 6–7, §4 · p. 6

4.2 Simulation results

  • 73.8% mean held-out success for Robo-COP versus 64.8% for the frozen-policy baseline and 65.8% for fixed-schedule training.The fixed schedule recovers only 1.0 of the 9.0 points over the frozen-policy baseline.Source: Figure 5 · p. 7, §4.2 · p. 7
  • By regime, Robo-COP reaches 80.0% where the VLA suffices, 72.0% where scripted primitives suffice, and 70.0% where neither suffices.ASPIRE leads on primitive-sufficient tasks at 73.5%, while the fixed and frozen harnesses reach 50.0% and 55.0%; on tasks needing both, Robo-COP is 16.7 points above the frozen-policy baseline.Source: Table 1 · p. 7, §4.2 · p. 7
  • Robo-COP fails 131 of 500 held-out trials versus 176 for the frozen-policy baseline, with grasp-or-drop failures accounting for 28 of 45 fewer failures.Every first training request targeted a pickup skill, and the analysis attributes more reliable grasps to fine-tuning on curated examples, including examples from failed episodes.Source: §4.2 · pp. 7–8
  • The controller trained on seven tasks with frequent deployment failures and waited on three with infrequent failures; verification rejected 5 of 12 candidates.On four tasks whose first candidate was reverted, the fixed schedule fell below the frozen-policy baseline on three, and a later candidate was adopted on three of those four tasks.Source: §4.2 · p. 8
  • On Recycle cup clutter, the second candidate-evaluation window reaches 76% success over the final 25 deployment trials, versus 60% for the frozen-policy baseline and 52% for fixed schedule.Training was requested after trials 45 and 75, and the candidates were adopted at trials 70 and 100.Source: Figure 6 · p. 8, §4.2 · p. 8
  • Robo-COP succeeds on 73.2% of 100 deployment trials, versus 68.7% for the frozen-policy baseline and 70.3% for fixed schedule.The difference persists after the systems diverge after trial 30 and over the final 25 trials.Source: §4.2 · p. 8

4.3 Real-world experiments

  • 50.0% mean held-out success on three real-world tasks for Robo-COP, versus 38.3% for the frozen-policy baseline.On the 60-second task, held-out success rose from 35% to 70% after verification adopted an update; on the 40-second task, verification reverted the candidate and held-out success stayed at 40%.Source: §4.3 · pp. 8–9, Table 2 · p. 9

5 Conclusion

Robo-COP co-evolves a robot’s orchestrator and policy during deployment. The conclusion reports gains across simulation and real-world tasks, alongside limits and directions for further development.

5 Conclusion

  • 9.0 points: Robo-COP improves held-out success over a frozen-policy harness across ten simulated tasks, while fixed-schedule fine-tuning without verification recovers only 1.0 point.The controller leaves the policy unchanged on three tasks with 80–96% deployment success and rejects 5 of 12 candidates on the other seven tasks.Source: §5 · p. 9
  • On real-world tasks, Robo-COP improves performance where it adopts an update, loses little where it reverts, and performs worse when forced to use a rejected update.The conclusion does not give per-task values for these outcomes.Source: §5 · p. 9
  • Robo-COP learns only skills its policy and scripted tools already demonstrate at least partially; physical deployment requires human resets, and each update adds curation, training, and evaluation cost.The conclusion also says ASPIRE outperforms Robo-COP on tasks already covered by scripted primitives and that larger experiments are needed to validate scalability.Source: §5 · p. 9
  • Robo-COP updates the policy and orchestrator memory, while its scripted skills, judges, and reasoning remain fixed.The conclusion proposes deployment-time structured skill exploration as a way to provide better tools and align the orchestrator more closely with reinforcement learning from deployment outcomes.Source: §5 · p. 9
  • Pooling skill-level demonstrations from many tasks is presented as a natural path toward training one policy for cross-task generalization.The conclusion frames this as a future direction, not a demonstrated result.Source: §5 · p. 10

A Constants

The appendix specifies simulation loop settings and the orchestrator’s sandbox, execution monitoring, memory updates, and recording fields.

A Constants

  • Simulation results use the settings defined in Table 3.The supplied passage identifies Table 3 as defining loop constants but does not state their values.Source: A Constants · p. 14, Table 3 · p. 14

B Orchestrator interface

  • The orchestrator runs in a sandbox limited to perception and motion tools, with no ground-truth poses or simulator imports, and each episode has at most twenty turns.The permitted functions include observation, object-pose, grasp/place-pose, motion, gripper, and policy-call tools.Source: B Orchestrator interface · p. 14
  • A policy call ends on a completion check, step limit, or VLM monitor stop; a stationary arm holding an object does not count as a completed placement.Calls use execute_vla(instruction, max_steps, policy) and return arm and gripper commands.Source: B Orchestrator interface · p. 14
  • After each episode, reflection updates memory about what works and known failures, while recipe deletions require turn verification and policy-discouraging advice is unverified after adoption or reversion.Memory is capped at 8000 characters with a fixed allocation per section; other entries remain unchanged after reconciliation.Source: B Orchestrator interface · pp. 14–15
  • The recorder logs each actuated step with both camera views, end-effector pose, measured joint state, commanded joint target, gripper channel, and turn index.Source: B Orchestrator interface · p. 15

C Curation

Curation converts execution segments into skill demonstrations through text-based skill identification and video-based outcome checks, then filters and segments the accepted data.

C Curation

  • A text judge proposes and labels skills, and a visual judge checks them against video.Source: C Curation · p. 15
  • For each turn, the text judge reads code, console output, and outcome assessment to return an instruction and verdict; policy-call instructions are kept verbatim.Each instruction names one skill using a canonical verb and its required arguments.Source: C Curation · p. 15
  • For scripted turns, primitive calls are grouped into physically meaningful skills, and function names are excluded from instructions.Source: C Curation · p. 15
  • The visual judge samples six native-resolution frames and labels each segment as success, useful partial progress, no progress, failure, or unclear.Source: C Curation · p. 15
  • A segment is kept only when the text judge returns ok and the visual judge finds success or useful partial progress; gripper-only segments are dropped.Accepted rows are split at instruction changes, rejected rows, or joint-state jumps above 0.3 rad; segments shorter than two steps are discarded and remaining segments receive fifteen held frames to teach stopping.Source: C Curation · p. 15

D Controller

The controller applies fixed guards before asking a model whether to train, then supplies deployment evidence for a train-or-wait decision and target-skill selection.

D Controller

  • Training waits for at least thirty episodes with a kept segment and ten such episodes since the last attempt, with at most three attempts per task.It also waits if the incumbent has fewer than five task-success reference trials or the deployment budget is spent; overall runs and episodes are capped.Source: D Controller · p. 15
  • After the guards pass, the model receives task memory, trial statistics, prior attempts, curated instructions and segment counts, plus code and judge output from up to three recent failures.It returns train or wait with at most four target instructions; unparsable verdicts default to wait, and unparsable instruction lists become empty.Source: D Controller · p. 15
  • Table 4 reports each task’s first training request, targets and curated-data counts, with arrows for later decisions; six tasks end with an updated policy.A dash indicates the controller never requested training.Source: Table 4 · p. 16

E Adoption arithmetic

The appendix details how Robo-COP evaluates candidate checkpoints, fine-tunes policies on curated data, and uses separate model calls for curation, training control, and adoption review.

E Adoption arithmetic

  • Candidate evaluation spans up to 25 episodes, requires exposure to relevant skills, and may stop early only after repeated attributable primary-skill failures.A round with fewer than five episodes containing a candidate call that commits an action is starved; stopping prevents adoption.Source: E Adoption arithmetic · p. 16
  • Adoption compares the candidate with a reference from up to 30 recent incumbent episodes and checks target skills, overall success, and previously admitted skills.Prior-skill groups need at least five candidate calls and an incumbent finish rate to block adoption; exact instruction matches are checked numerically, with paraphrases reviewed semantically.Source: E Adoption arithmetic · p. 16

F Fine-tuning

  • The candidate is fine-tuned on curated data with the policy’s flow-matching objective, using joint-target offsets from observed state while keeping gripper commands absolute.LoRA adapters are trained in the language backbone and action expert; the vision encoder, action projections, and time-conditioning MLP are fully fine-tuned.Source: F Fine-tuning · pp. 16–17

G Agent prompts and decision contracts

  • Curation, training control, and adoption review are separate model calls, and the adoption reviewer checks the candidate against its stated expectations and target instructions.Mechanical completion alone is not verified skill success, and ambiguous attribution is classified as uncertain.Source: G Agent prompts and decision contracts · pp. 17–18
  • Curation groups executed primitives into low-level skills, then judges each skill’s outcome using instructions, execution records, and sampled frames.A skill completed before a later episode failure can be retained; uncertain judgments are not counted as success, and scripted recovery is not credited to the candidate.Source: G Agent prompts and decision contracts · pp. 17–18
  • The training controller weighs curated-data counts, recurring instructions, trial statistics, recent failures, and task memory when deciding whether to fine-tune.Its prompt frames training as worthwhile when skills are demonstrable and the current approach is failing or slow, but wasteful when orchestration succeeds cheaply or data is thin or stale.Source: G Agent prompts and decision contracts · p. 17

H Additional simulation results

The additional simulation results compare deployment windows and trajectories with held-out performance, including task-level differences and the frozen baseline’s decline.

H Additional simulation results

H.1 Successful episodes and randomized resets

The figures document successful simulated episodes across ten tasks and show the randomized initial object positions used for evaluation.

H.1 Successful episodes and randomized resets

  • Figure 10 presents five frames from one successful episode for each of tasks 1–5, sampled at roughly 2%, 25%, 50%, 75%, and 98% of trial duration.Source: Figure 10 · p. 20
  • Figure 11 presents successful episodes for tasks 6–10 using the same frame-sampling scheme as Figure 10.Source: Figure 11 · p. 21
  • Figure 12 plots initial object positions in simulator world coordinates after 50 seeded resets per task, with colors distinguishing objects within each task.Source: Figure 12 · p. 21

I Real-world executions

Figure 13 shows successful real-world executions of the three tasks, with frames arranged in temporal order.

I Real-world executions

Loading 2610.09228v1…