PaperlayerView exact revision on arXiv

Source-linked AI summary

FastHarness: Experience-Driven Review Scheduling for Vision-Language-Action Policies

Tao Lin, Yuxin Du, Haoxiao Wang, Zile Zhong, Bo Zhao

arXiv:2610.31427v1cs.RO

TL;DR

Reviewing every robot action chunk adds overhead even in familiar stages, while errors may require repeated correction. FastHarness uses accumulated experience to schedule GPT-6 Astra review of a fixed π0.5 policy and focus supervision on recovery. Across four simulated tasks, review demand and episode time decrease as experience accumulates, and mean success is higher than with GPT-as-Policy.

  • Problem

    Reviewing every action chunk repeats overhead in reliably completed stages, yet errors can require several correction rounds, creating a need to reduce redundant review without losing recovery supervision.

    Source: §1 · p. 1

  • Method

    FastHarness uses reviewed stage outcomes to adapt supervision of a fixed π0.5 policy, with a fast path for routine execution and a reviewed recovery path for errors.

    Source: §1 · p. 1

  • Results

    Across four simulated tasks, reviewer token use decreases by an average of 57.2% per task and mean episode time falls from 85.5 to 52.0 minutes; mean success is 97.19% with FastHarness versus 80% for GPT-as-Policy.

    Source: §1 · pp. 1–2, §6 · p. 7

  • Takeaways & Limitations

    Adaptive supervision is a practical approach to improving agentic robot execution efficiency and reliability without additional model training.

    Source: §6 · p. 7

    In one Fold clothes execution, 20 reviews did not initiate recovery, and the task ended in failure with a native score of 0.2.

    Source: §5.5 · p. 7

Abstract

from arXiv · show

Agentic robot systems combine language-model reasoning with robot policies to monitor execution and recover from failures. However, reviewing every action chunk incurs substantial overhead, even in familiar situations. We introduce FastHarness, an experience-driven execution harness that adaptively schedules GPT-6 Astra supervision of a {\pi}0.5 policy. A fast path uses stage memory and reviewed outcomes to reduce review frequency as experience accumulates, while a complementary recovery path focuses supervision on correcting errors. Once recovery is verified, routine execution resumes, with failure experience retained to guide future interventions. The policy remains frozen and generates fresh actions from current observations. Across four simulated manipulation tasks, reviewer token use decreases by an average of 57.2% per task from the first evaluated episode to the mean of the final five episodes. Mean episode time decreases from 85.5 to 52.0 minutes over the same interval. On the same tasks, mean success increases from 80% for GPT-as-Policy ({\pi}0.5 + GPT-6 Astra) to 97.19% with FastHarness. These results highlight adaptive supervision as a practical route to more efficient agentic robotics without additional model training.

1 Introduction

Agentic robot systems pair learned policies with language-model reasoning, but reviewing every action chunk adds overhead even in reliable stages. FastHarness uses experience to schedule review and focus supervision on recovery; across four simulated tasks, it reduces review demand and time while improving success over GPT-as-Policy.

1 Introduction

  • Agentic robot systems combine learned policies that act on observations and instructions with language models that assess progress and guide corrections.GPT-as-Policy pairs π0.5 with GPT-6 Astra.Source: §1 · p. 1
  • Reviewing every action chunk repeats overhead even in reliably completed stages, while errors such as missed grasps or dropped objects may require several correction rounds.Source: §1 · p. 1
  • FastHarness uses reviewed outcomes to reduce review frequency as successful stage experience accumulates, while keeping the π0.5 policy fixed and generating actions from current observations.The figure contrasts this adaptive supervision with GPT-as-Policy review at every action chunk.Source: §1 · p. 1, Figure 1 · p. 2
  • Errors trigger a recovery path with continued review; after the reviewer verifies recovery, routine scheduling resumes and the failure record is retained for future interventions.Source: §1 · p. 1
  • Across four simulated manipulation tasks, FastHarness mean success is 97.19%, up from 80% for the baseline; across repeated runs, reviewer token use falls 57.2% per task and mean episode time drops from 85.5 to 52.0 minutes.Source: §1 · pp. 1–2

2 Related Work

Related work spans language-model-guided robot execution, selective supervision, and experience-based recovery. FastHarness builds on hybrid execution by scheduling review from recurring task-stage experience while the policy remains fixed.

Language-model-guided robot execution

  • SayCan grounds skill selection in robotic affordances, Inner Monologue uses execution feedback for language-based planning, and VLA policies such as π0.5 map observations and instructions to actions.GPT-as-Policy combines π0.5 with GPT-6 Astra for execution review and correction.Source: §2 · p. 2

Selective supervision and intervention

  • Selective-supervision work studies when systems should request help, including uncertainty alignment and intervention strategies based on human input, novelty, or risk.FastHarness instead schedules model supervision during repeated execution with a fixed policy, using reviewed outcomes from recurring stages.Source: §2 · p. 2

Experience and failure recovery

  • Prior experience methods use feedback to guide later trials, accumulate reusable skills, or explain robot failures; FastHarness retains reviewed failure evidence to allocate supervision across routine execution and recovery.Source: §2 · p. 2

3 Problem Setting

The setting is repeated manipulation-task execution with a fixed VLA policy and persistent task-specific experience across episodes. The harness schedules review and incorporates feedback, with the study examining overhead and task performance.

3 Problem Setting

  • At each decision step, a pretrained VLA policy maps the current observation and language instruction to a multi-step action chunk, which a separate reviewer may approve or correct.Source: §3 · p. 2
  • FastHarness adds experience-driven scheduling and recovery paths around the GPT-as-Policy execution backbone, retaining reviewed evidence across episodes.Ordinary scheduling resumes after verified resolution.Source: Figure 2 · p. 3
  • Episodes with varying scene layouts form a memory chain when linked by persistent task-specific experience, while the VLA policy weights remain fixed throughout.Adaptation occurs in harness memory and the supervision schedule.Source: §3 · pp. 2–3
  • The study asks whether accumulated task experience can reduce model-review overhead while maintaining performance; task success follows the environment’s completion criterion.Source: §3 · p. 3

4 FastHarness

FastHarness augments GPT-as-Policy with a scheduler that uses task memory to decide when to review policy proposals and how to incorporate feedback. Its fast path adapts routine review frequency, while its recovery path maintains supervision until faults are verified as resolved.

4 FastHarness

  • The experience-driven fast path uses stage-specific reviewed outcomes to adjust review frequency, while the recovery path maintains review during detected-error correction.Reviewed outcomes update the memory used for later scheduling decisions.Source: §4 · p. 3

4.1 Experience-Driven Fast Path

  • Stage labels organize stored observation examples and outcome statistics; only explicit judgments on the immediately preceding ordinary policy transition count, while corrective and recovery actions are excluded.Source: §4.1 · p. 3
  • Limited evidence or too few consecutive successes after an error keeps review intervals at one chunk; stronger stage scores and more reviewed samples permit longer intervals.Ambiguous contexts, uncertainty, alerts, remembered errors, stage transitions, random audits, and a control-step limit can also trigger review.Source: §4.1 · p. 4
  • When review is skipped, the executor runs a prefix of the current policy proposal within the remaining control-step allowance, then obtains a fresh observation and proposal rather than replaying stored trajectories.Source: §4.1 · p. 4

4.2 Error Recovery Path

  • Recovery begins with reviewed local repair; if unresolved, the harness can select an earlier verified waypoint and have the policy generate new actions toward it in the current environment.Ordinary scheduling resumes only after the reviewer verifies resolution, and later observations near the remembered error still require review.Source: §4.2 · p. 4

5 Experiments

Across four shared tasks, FastHarness reaches a 97.19% mean success rate versus 80% for the hybrid GPT-as-Policy baseline. Repeated execution also reduces reviewer demand and execution time, while recovery examples show how normal scheduling resumes after local correction.

5.1 Experimental Setup

  • The evaluation uses π0.5 as the student policy and GPT-6 Astra as reviewer across four simulated manipulation tasks with persistent task memory.Efficiency comparisons use the first evaluated episode and final-five-episode mean; success uses the native task evaluator.Source: §5.1 · p. 4

5.2 Reviewer Use and Execution Efficiency

  • Across the four simulated tasks, recorded GPT-6 tokens decrease by 57.2% on average and mean episode time falls from 85.5 to 52.0 minutes.These comparisons use the first evaluated episode and the mean of the final five episodes.Source: §5.2 · p. 4, Figure 4 · p. 5

5.4 Review Allocation and Runtime Analysis

  • Unknown-context triggers fall from 35 to 16 for Fold clothes and 142 to 64 for Bottle disposal; median automatic spans grow from 30 to 60 control steps in both.Reviewer interaction accounts for 82.0% of wall-clock time in Fold clothes and 89.7% in Bottle disposal across their 20-episode chains.Source: §5.4 · p. 5

5.3 Task Completion

  • 97.19% mean success with FastHarness compares with 80% for the hybrid π0.5 + GPT-6 Astra baseline, a 17.19-point difference.Build tower improves from 60% to 93.75% (15/16), and Classify objects from 60% to 100% (9/9); Bottle disposal stays at 100%, while Fold clothes reaches 95% versus 100%.Source: §5.3 · pp. 4–5, Figure 5 · p. 6

5.5 Recovery Behavior

  • During Bottle disposal recovery, the robot re-grasps a dropped bottle, resumes normal scheduling, and completes the episode with a score of 100/100.In Classify objects, a saved waypoint confirms that two rabbits remain placed, but the original fault still requires correction.Source: §5.5 · p. 6, Figure 7 · p. 7, Figure 8 · p. 7
  • All 16 recovery events in Classify episodes 1–8 return to normal scheduling, but this does not guarantee native task success.One Fold clothes episode ended with a native score of 0.2 after 20 reviews without recovery being initiated.Source: §5.5 · pp. 6–7

6 Conclusion

Across four simulated manipulation tasks, FastHarness reduces review demand and execution time as experience accumulates, while achieving higher mean task success than the hybrid GPT-as-Policy baseline. Its two-path harness uses stage experience to reduce redundant review and guides error correction before ordinary scheduling resumes.

6 Conclusion

  • The fast path uses accumulated stage experience to reduce redundant review, while the recovery path guides error correction and restores routine scheduling after verification.Source: §6 · p. 7
  • 97.19% mean task success with FastHarness compares with 80% for the hybrid GPT-as-Policy baseline.The reported results also show review demand and execution time decreasing as experience accumulates.Source: §6 · p. 7
  • Across four simulated manipulation tasks, review demand and execution time decrease as experience accumulates, supporting adaptive supervision without additional model training.Source: §6 · p. 7

A Implementation Details

The appendix specifies how FastHarness represents stage experience, attributes outcomes, allocates reviews, and handles recovery, alongside the evaluation runtime configuration.

B.1 Runtime and Controller Configuration

A.1 Stage Recognition and Outcome Attribution

  • Stage memory stores normalized visual, joint-angle, and gripper features under reviewer-assigned stage labels, retaining the latest 128 distinct examples per stage.Each review labels the immediately preceding transition; merged error records preserve failure and verified-recovery evidence.Source: §A.1 · p. 9

A.2 Reliability and Review Allocation

  • The scheduler combines a conservative score from reviewed successes and failures with labeled experience to set review intervals, while uncertainty, warnings, error proximity, and other triggers can prompt review.The evaluated configuration uses count decay 1.0; review triggers follow a stated priority order, with contact gating disabled.Source: §A.2 · p. 9

A.3 Control Loop and Recovery State

  • After reviewed execution errors or intent mismatches, local repair proceeds through bounded attempts and may escalate to an eligible waypoint; verified return requires a further successful chunk before normal execution.Eligible waypoints are saved outside recovery after an ok outcome, with a reviewer-supplied goal and prerequisites.Source: §A.3 · p. 9

B.2 Evaluation Protocol

Evaluation uses four task chains with memory persisting across episodes, and measures native task outcomes separately from review and runtime usage. Results are descriptive at the chain level because episodes and recovery events share memory.

B.2 Evaluation Protocol

  • Memory persists across episodes within each task chain, with 20 episodes for Fold clothes and Bottle disposal, 16 for Build tower, and nine for Classify objects.The first evaluated Build tower episode follows earlier task experience; Classify objects begins with forced review.Source: §B.2 · p. 10
  • Task success uses the native evaluator’s native_success flag at termination or truncation; infrastructure interruptions are excluded, while task failures remain in the denominator.Reviewer approval and partial scores do not alter native outcomes.Source: §B.2 · p. 11

B.3 Metrics and Aggregation

  • Efficiency compares the first evaluated episode with the mean of the final five, while token usage sums recorded input and output tokens and wall-clock time includes policy, robot, reviewer, and runtime components.Review calls, accepted reviews, and executed chunks are counted separately, so skip rate cannot be inferred as one minus calls per chunk.Source: §B.3 · p. 11
  • Because episodes share task memory and recovery events may contain several attempts, results are descriptive chain-level findings rather than independent-trial estimates.Returning to normal scheduling marks the end of an intervention, separately from task success.Source: §B.3 · p. 11

C Recovery Trace Analysis

The traces show recovery returning to normal scheduling in the reported Classify objects events, while also documenting a task failure that was not detected by recovery triggers. Individual traces illustrate local repair and waypoint-guided correction.

C Recovery Trace Analysis

  • In Bottle disposal episode 19, four completed recovery chunks and five accepted reviews used 161,222 recorded tokens over 480.15 seconds before automatic execution resumed.The recovery window excludes the triggering review and the replan after recovery exit; the dropped-bottle failure record remained in memory.Source: C Recovery Trace Analysis · p. 11
  • In Classify objects episode 8, waypoint-guided correction preserved a saved subgoal in the current scene without resetting physical state, before a separate later recovery event.Source: C Recovery Trace Analysis · p. 11
  • All 16 recovery events across eight Classify objects episodes returned to normal scheduling, an intermediate outcome distinct from native task success.Events within episodes and the shared memory chain are not independent trials.Source: Table 4 · p. 11, C Recovery Trace Analysis · p. 12
  • Fold clothes episode 17 failed at the native evaluator despite 20 reviews without recovery initiation, showing that segment reviews did not ensure detection of every task-level failure.These traces do not estimate outcomes without recovery.Source: C Recovery Trace Analysis · p. 12

D Supplementary Numerical Results

FastHarness achieves a higher mean success rate than GPT-as-Policy on the four tasks, while reviewer token use also declines across repeated execution. The numerical comparisons use the first evaluated episode and the mean of the final five episodes.

D Supplementary Numerical Results

  • 57.2% is the unweighted mean per-task reduction in recorded GPT-6 token use from the first evaluated episode to the final-five-episode mean.Counts include input and output tokens, with cached input counted once; Build tower’s first evaluated episode follows earlier task experience.Source: Table 5 · p. 12
  • The GPT-as-Policy baseline report reweights official π0.5 RoboDojo leaderboard results for the selected tasks and scene mix.Its local evaluations use five cases per task, with two standard and three random cases for Fold clothes and five standard cases for each other task.Source: D Supplementary Numerical Results · p. 12
  • 97.19% mean success with FastHarness compares with 80% for the GPT-as-Policy hybrid across the four tasks.The unweighted task means are reported for FastHarness and GPT-as-Policy (π0.5 + GPT-6 Astra), respectively.Source: D Supplementary Numerical Results · p. 12, Table 6 · p. 12, Table 7 · p. 12

E Supplementary Fast-Path Analysis

Supplementary analysis reports longer automatic execution spans in later episodes and running token-use averages that retain the initial experience-acquisition cost. The evaluation is limited to four simulated RoboDojo tasks, and sensor or hardware differences may affect deployment.

E Supplementary Fast-Path Analysis

  • In both Fold clothes and Bottle disposal, median automatic-span length rises from 30 to 60 control steps between episodes 1–5 and 16–20.Figure 9 pools 21 early and 19 late spans for Fold clothes, and 18 early and 26 late spans for Bottle disposal; episodes without automatic execution contribute no spans.Source: E Supplementary Fast-Path Analysis · p. 12, Figure 9 · p. 13
  • After 20 episodes, running mean token use is 0.249 million for Fold clothes and 0.734 million for Bottle disposal.The running mean includes the initial cost of acquiring experience and all intervening executions.Source: E Supplementary Fast-Path Analysis · p. 13, Figure 10 · p. 13

F Limitations

  • The evaluation covers four simulated RoboDojo tasks using π0.5 and GPT-6 Astra.Different sensor interfaces may require adapting the RGB-and-proprioception memory representation, while runtime savings also depend on reviewer latency and execution hardware.Source: F Limitations · p. 13

Related papers

Loading 2610.31427v1…