Source-linked AI summary
Recursive Harness Distillation across Agents for Robot Manipulation
Seungyeon Kim, Junhoo Lee, Minkyu Kim, Baekseung Kim, Nojun Kwak
TL;DR
VLA policies can struggle to diagnose failures and adapt as execution changes, while transferring intervention expertise to lighter models is costly. Recursive Harness Distillation has a capable agent create a reusable playbook and refine it using a light agent’s execution feedback. On SimplerEnv Bridge, the light agent achieves 66.7% success with the playbook, versus 41.7% for GR00T alone; real-world success rises from 37.3% to 64.0%.
Problem
Agents must connect physical observations, policy behavior, and intervention effects to elicit intended VLA behavior, while relying on capable models throughout execution is costly.
Source: §1 · p. 1Method
Results
66.7% success on SimplerEnv Bridge is achieved by the light agent with the playbook, versus 41.7% for the GR00T-only baseline.
Source: §1 · p. 2Takeaways & Limitations
The results demonstrate the feasibility of accumulating, refining, and reusing VLA intervention experience across agents to improve robotic manipulation.
Source: §6 · p. 9
Abstract
from arXiv · showhide
A central goal in robotics is to enable manipulation across changing tasks and environments. Vision-language-action (VLA) models provide broad manipulation capabilities but can struggle when execution requires diagnosing failures and adapting behavior. Strong agents can discover effective interventions through interaction with these policies. We propose Recursive Harness Distillation to accumulate this experience as reusable guidance across agents. A strong agent distills its experience into a playbook for a light agent, then recursively refines the playbook using the light agent's execution feedback. The resulting playbook enables agents to reuse accumulated intervention knowledge in new task instances without updating model parameters. In real-world manipulation, the harness improves success from 37.3% to 64.0%. On SimplerEnv Bridge, the light agent with the playbook achieves 66.7% success, compared with 41.7% for the GR00T-only baseline, and outperforms the strong agent without a playbook. The same playbook also benefits the strong agent, which reaches 79.2% success. These results demonstrate the feasibility of harness distillation for robotics: intervention experience can be accumulated, refined through execution, and reused across agents to improve manipulation.
1 INTRODUCTION
Operating a VLA effectively requires connecting physical observations and policy behavior to interventions, and acquiring that expertise through capable models is costly. Recursive Harness Distillation transfers interaction experience into a playbook for a lighter agent, then refines it using that agent’s execution feedback.
1 INTRODUCTION
- Effective intervention requires linking physical observations, policy behavior, and intervention effects; experience helps agents learn which adjustments work in which situations.Source: §1 · p. 1
- Transferring expertise is itself resource-intensive: the capable model must explore the task, analyze experience, and prepare useful guidance.The paper asks how to distill that expertise for a lighter model under a limited token budget.Source: §1 · p. 1
- Recursive Harness Distillation transfers a capable agent’s experience with a VLA into a reusable harness for a lighter agent.The capable agent records effective strategies, then revises the playbook after reviewing the lighter agent’s execution experience.Source: §1 · pp. 1–2
- The harness combines an interface for observing and intervening in VLA execution with a playbook that guides its use.Repeated refinement captures both what works for the VLA and what the lighter agent needs to apply it.Source: §1 · pp. 1–2
- 66.7% success on SimplerEnv Bridge is achieved by the light agent with a playbook, versus 41.7% for the GR00T-only baseline; the same playbook also benefits the strong agent.In real-world manipulation, the harness improves success from 37.3% to 64.0%.Source: §1 · p. 2
2 RELATED WORK
Related work provides robot policies and mechanisms for guiding execution, and studies ways to transfer or retain knowledge. The paper positions RHD as reusable intervention guidance refined through recipient-agent experience.
2 RELATED WORK
- VLA policies combine pretrained vision-language representations with diverse robot demonstrations for language-conditioned tasks, but unfamiliar visual conditions remain a generalization challenge (Zhou et al., 2025; Liu et al., 2026).Source: §2 · p. 2
- Harness approaches generate robot programs or coordinate learned policies with planning, monitoring, and recovery, but do not explain how to accumulate and transfer effective interventions (Liang et al., 2023; Chen et al., 2026; Galanti et al., 2026).Source: §2 · p. 2
- Knowledge distillation, student feedback, and reusable agent reflections motivate adapting taught knowledge; RHD applies these ideas to VLA manipulation through a teacher–playbook–student process (Hinton et al., 2015; Liu et al., 2021; Zhou et al., 2022; Chen et al., 2025; Shinn et al., 2023; Zhao et al., 2024; Zhang et al., 2026).The guidance concerns when and how to intervene, and is refined using the physical outcomes of recipient-agent decisions.Source: §2 · p. 2
3 METHOD
Recursive Harness Distillation keeps the VLA and agent parameters fixed, using a playbook to guide interventions and recipient execution feedback to refine that guidance. Its objective is to improve the task success of the light agent’s composed policy.
3.1 PROBLEM FORMULATION
- The task is to improve success for a light agent working with a frozen VLA, using execution histories and intervention decisions to guide its composed policy.The history may include goals, observations, prior interventions, and actions; the decision record includes selected operations and proposed and executed actions.Source: §3.1 · pp. 2–3
3.2 INTERVENING ON VLA EXECUTION
- The agent can intervene at instruction, attention, and output sites, choosing any admissible subset and checking later observations for the intended effect.Instruction and attention interventions affect candidate generation; output interventions transform candidate actions, and identity settings leave the VLA unchanged.Source: §3.2 · pp. 3–4
3.3 POLICY OPTIMIZATION VIEW: IMPROVING THE RECIPIENT’S DECISIONS
- Playbook revisions target better decisions at histories the recipient actually encounters, so the objective is evaluated through complete recipient rollouts rather than teacher-history imitation alone.The light agent’s choices change later situations, making its execution the relevant experience distribution.Source: §3.3 · p. 4
3.4 RECURSIVE HARNESS DISTILLATION (RHD)
- The strong agent initializes guidance from its experience, then proposes revisions using recipient rollouts; the first revision meeting the teacher’s empirical success rate on shared development instances is accepted.The playbook specifies when and how to intervene and what to check afterward; further recipient experience informs later revisions.Source: Figure 1 · p. 3, §3.4 · pp. 4–5
4 EXPERIMENTS
The experiments test whether a refined playbook improves use of frozen VLAs across simulated and real-world manipulation, and examine refinement, teacher capability, and intervention sites. Results show substantial gains with the playbook, while iterative refinement and a more capable teacher are important to those gains.
4.1 EXPERIMENTAL SETUP
- Evaluation covered four fixed-GR00T Bridge tasks and three Franka Panda tasks, using held-out configurations in simulation and randomized object positions in real-world trials.Playbooks were developed on development instances and frozen for evaluation.Source: §4.1 · pp. 5–6
4.2 MAIN RESULTS
- 66.7% success for Luna with the refined playbook versus 43.8% without one on Bridge; real-world success reaches 64.0% versus 37.3% for π0.5 alone.The real-world comparison also reports 0% for Luna with the policy but no playbook.Source: Figure 3 · p. 6, §4.2 · p. 6, Table 1 · p. 7
4.4 CONTRIBUTION OF INTERVENTION SITES
- Removing instruction editing caused the largest performance drop among the tested intervention sites, alongside direct action correction’s contribution.Each ablation disabled one site while retaining the other two and keeping the refined playbook fixed.Source: Figure 3 · p. 6, §4.4 · p. 7
4.3 RECURSIVE HARNESS DISTILLATION
- 22.9% success with a Luna-derived playbook versus 66.7% with an Astra-derived one shows that self-refinement by the light agent did not match the stronger teacher.Both recipients operated frozen GR00T on the same development split and under the same stated limits.Source: Table 2 · p. 7, §4.3 · p. 7
- Refinement raised Luna’s Bridge success from 31.3% with the initial playbook to 66.7%, improving success across four tasks without increasing average tokens per episode.The initial playbook had reduced success from 43.8% without a playbook to 31.3%.Source: §4.3 · p. 7
5 ANALYSIS
The analysis turns playbook experience into concrete recovery rules and shows how those rules affect execution. It also compares deployment inference costs for light and strong agents using the refined playbook.
5 ANALYSIS
- Figure 5 tracks success rate and mean agent tokens per episode across playbook revisions, using the same 12 held-out instances per task.Token usage is averaged across episodes, including failures.Source: Figure 5 · p. 8
5.1 UNDERSTANDING POLICY BEHAVIOR THROUGH THE PLAYBOOK
- Figure 6 presents selected playbook excerpts covering identity after occlusion, weak lifts, instruction–action mismatches, attention-based recovery, preserved corrections, and placement evidence.The examples come from interactions with GR00T on SimplerEnv Bridge tasks.Source: Figure 6 · p. 8, §5.1 · p. 8
- The playbook addresses observation and action failures with object re-identification, bounded lifts, action inspection and correction, and checks that preserve grasp geometry and verify placement.A small upward motion alone does not prompt reopening when a grasp remains plausible; before release, the playbook checks object–target geometry and stable support.Source: §5.1 · p. 8
5.2 HOW GUIDANCE CHANGES EXECUTION
- Local instruction, gripper, and spatial interventions return stalled spoon, eggplant, and carrot executions to states where GR00T can continue.For the spoon, the robot regrasped at the source; for the eggplant, a gripper correction enabled lifting and carrying; for the carrot, the intervention focused on object–finger contact.Source: §5.2 · p. 9
5.3 DEPLOYMENT COST
- $0.63 per episode for Luna versus $3.43 for Astra on the held-out Bridge test set, with Astra approximately 5.5 times more expensive.The full-test-set estimates are $30.03 for Luna and $164.61 for Astra.Source: §5.3 · p. 9
6 CONCLUSION
The conclusion presents Recursive Harness Distillation as a way to accumulate a strong agent’s VLA experience into reusable guidance, refined using a light agent’s execution feedback. Experiments show improved light-agent manipulation across simulation and real-world tasks, with Bridge results also supporting use of the playbook by the strong agent.
6 CONCLUSION
- Recursive Harness Distillation turns a strong agent’s VLA interaction experience into a playbook that a light agent can reuse and refine from execution feedback.The guidance adapts to how the recipient applies it without updating model parameters.Source: §6 · p. 9
- Experiments on SimplerEnv Bridge and three real-world manipulation tasks show that the playbook improves the light agent’s ability to operate the deployed policy.Source: §6 · p. 9
- On Bridge, the light agent with the playbook outperforms the strong agent without one, and the same playbook also improves the strong agent’s performance.Source: §6 · p. 9