Source-linked AI summary
Skill-Space Shooting for Autonomous Robot Policy Improvement
Zihang Rui, Renhao Wang, Haoxu Huang, Yang Gao
TL;DR
Robots need to improve after deployment without requiring people to demonstrate every correction, but composing skills to complete tasks does not itself teach a task policy to overcome its failures. Skill-space shooting uses foundation-model guidance to try reusable skills as corrections and trains policies on successful repairs. Across four real-world tasks, unaided performance improves over successive updates, and sharing skill data reduces teaching needed for new tasks.
Problem
Robots need to improve beyond initial training as they encounter new situations, while relying on people for each correction makes improvement across tasks costly.
Source: §1 · p. 1Method
Skill-space shooting uses foundation-model guidance to select reusable skills as candidate continuations at policy failures, then trains the task policy on accepted repairs.
Source: §1 · p. 2Results
Across all four tasks, unaided performance improves over successive updates without new human demonstrations; Drawer rises from 0/20 initial successes to 7/20 after one update.
Source: §4.1 · p. 7Takeaways & Limitations
Sharing skill data reduces the additional human effort needed to prepare corrections for another task and can facilitate improvement on a new task.
Source: §1 · p. 2, §6 · p. 9The failure detector requires per-task training, although skill selection and repair verification use a common procedure.
Source: §5 · p. 9
Abstract
from arXiv · showhide
Robots deployed in the physical world must be able to improve beyond their initial training as they encounter new situations and failures. For this improvement to scale across tasks, it must make effective use of experience without requiring human demonstration of each correction. Recent agentic systems offer a way to reduce this reliance on human effort by using foundation models to autonomously compose learned behaviors to complete tasks. Yet completing tasks this way does not itself teach a task policy to overcome its own failures; that requires turning these behaviors into learnable corrections for the policy. Our insight is that many such corrections are familiar short behaviors, or skills: they recur across tasks and describe actions that foundation models can reason about from a scene. We introduce skill-space shooting, which uses foundation model guidance to explore corrections through these reusable skills and turn successful trials into policy improvement. Real-world experiments show repeated improvement in policies acting autonomously, while skills can also be shared to reduce the teaching needed to improve on new tasks. By making reusable skills a source of corrective supervision, skill-space shooting enables scalable and generalizable policy improvement within and across tasks. Additional results and videos at https://skill-space-shooting.github.io.
1 INTRODUCTION
Robots need to improve after deployment without relying on repeated human corrections. Skill-space shooting turns reusable skills into autonomous corrective supervision, so successful repairs can train task policies and skill data can be reused across tasks.
1 INTRODUCTION
- Repeated human corrections bottleneck improvement as tasks grow, while sparse-reward exploration can make physical learning on long-horizon tasks costly.Agentic skill composition can guide execution but does not, by itself, improve the task policy that needed assistance.Source: §1 · p. 1
- Reusable short-horizon skills can provide local corrections for long-horizon tasks, including at states shaped by the policy’s earlier actions.Foundation models can select and monitor a skill trial; successful trials supply both a continuation and actions that teach the policy.Source: §1 · p. 2
- Sharing skill training data can reduce the teaching needed to prepare corrections for another task.The paper also reports repeated unaided policy improvement across four real-world tasks, including from a policy with no observed initial task successes.Source: §1 · p. 2
- Skill-space shooting trials learned skills as candidate continuations at policy failures, then uses accepted repairs to update the task policy.A failure detector decides when to intervene, and foundation-model guidance selects skills and checks repairs.Source: §1 · p. 2
2 RELATED WORK
The paper situates skill-space shooting among work on agentic skill composition, learning from corrections, and autonomous improvement from experience. Its focus is using skill executions as corrective supervision so a task policy can retain the benefit of assistance.
2 RELATED WORK
- The related-work framing distinguishes three bodies of research: skill composition, learning from corrections, and autonomous improvement from experience.The paper’s stated learning problem is to make execution assistance improve the complete-task policy.Source: §2 · pp. 2–3
Agentic composition of primitives
- Agentic robotics makes learned behaviors available to model guidance during execution, while skill-space shooting uses those executions to improve the complete-task policy itself.Related approaches include skill composition, programmatic behavior sequencing, feedback-based repair, and online recovery.Source: §2 · p. 3
Learning from corrections
- Interactive imitation learning trains on corrections at states an imperfect policy visits, but obtaining those corrections from people can impose recurring effort.Skill-space shooting instead lets demonstrated primitives supply corrective actions autonomously as the task policy improves.Source: §2 · p. 3
Autonomous improvement from experience
- Prior autonomous-improvement methods use collected experience to retrain policies, fit critics or world models, or learn recovery behaviors.Skill-space shooting uses language-described skills to direct semantically guided physical repair trials, then trains a policy that can act without assistance.Source: §2 · p. 3
3 METHOD
Skill-space shooting uses a seeded library of learned short-horizon skills to repair likely policy failures and train the task policy on successful repairs. The method repeats this cycle and reuses skill data to reduce teaching for new tasks.
3 METHOD
- The method starts with an imperfect behavior-cloned task policy and a library of independently taught short-horizon skills, while focusing on turning those skills into corrective experience.Skills and their training data may come from previous tasks; acquiring skills from human video or autonomous experience is identified as further work.Source: §3 · pp. 3–4
3.1 DECIDING WHEN AND HOW TO SHOOT
- A value-based detector triggers a skill trial when the policy’s proposed action chunk indicates likely failure, and a foundation model selects a skill using the scene and task context.The detector is trained per task and then frozen; the foundation model also judges whether the physical trial repaired the stage.Source: §3.1 · pp. 4–5
3.2 REPAIRING ROLLOUTS AND IMPROVING POLICIES
- The selected skill takes control from the policy’s actual state, repairs the stage, and then hands control back so the policy can continue the rollout.This exposes skills to the state produced by earlier policy actions, rather than a clean-start demonstration state.Source: Figure 2 · p. 4, §3.2 · p. 5
- Only accepted skill-executed repair segments from completed rollouts are retained as corrective training data, excluding policy actions and rejected trials.The updated policy collects the next round of experience, allowing later iterations to address newly reached states without another demonstration round.Source: §3.2 · p. 5
3.3 REUSING THE SKILL LIBRARY
- Skills pair recognizable local corrections with policies that execute coordinated actions, allowing the foundation model to reason about a repair while the skill handles its physical execution.The library spans primitives such as picking, placing, sweeping, and hanging a broom; existing skill data can be combined with new demonstrations to adapt a primitive for another task.Source: §3.3 · p. 6
4 EXPERIMENTS
The experiments test iterative policy improvement, intervention reliability, and cross-task skill sharing. Skill-space shooting improves autonomous performance across tasks, while shared skill data reduces the new teaching needed to adapt skills and improve a new task.
4 EXPERIMENTS
- The experiments span Stack-3, Drawer, Coffee, and Sweeping, comparing skill-space shooting with sparse-reward latent RL, human-guided primitive composition, and additional demonstrations.Policies are evaluated over multiple improvement rounds; the loop requires human effort only for scene resets beyond initial task and skill demonstrations.Source: §4 · p. 6
4.1 MAIN RESULTS
- Successive updates improve unaided performance on all four tasks without new human demonstrations.Policies are evaluated without the value model, judge, or skill library, so the gains reflect behavior retained by the task policy.Source: §4 · p. 6, Figure 3 · p. 7, §4.1 · p. 7
- Skill-space shooting reaches the highest observed plateau on all four tasks.Table 1 reports success rates or mean progress and summarizes plateau performance using the best of the last three evaluations; COP is evaluated once.Source: Table 1 · p. 7, §4.1 · p. 7
- On Drawer, unaided success rises from 0/20 initially to 7/20 after one update, while DSRL remains at zero after its first update and reaches at most 2/20.The learned skills provide stage-level behaviors for targeted corrections when successful complete-task exploration is difficult.Source: §4.1 · p. 7
- Skill-space shooting exceeds COP on every task after starting below it, and leads DSRL by 55 progress points on Coffee and 63.8 on Sweeping.The comparisons also include HUMAN, which receives matched-duration complete-task demonstrations.Source: §4 · p. 6, §4.1 · p. 8
4.2 RELIABILITY OF INTERVENTIONS AND REPAIRS
- Across all four tasks, trigger accuracy is 71–83%, skill-selection accuracy is 90–100%, repair success is 70–77%, and completion-judgment accuracy is 90–95%.The repair rate is for correctly triggered failure cases; the retry budget allows up to three attempts.Source: §4.2 · p. 8, Table 2 · p. 8
4.3 SHARING AND EXTENDING SKILLS FOR NEW TASKS
- With 10 new sponge demonstrations, sharing cube-picking data raises grasp success from 5% to 70%.At 30 new demonstrations, sharing matches the success obtained with nearly twice as many demonstrations without sharing.Source: §4.3 · p. 9
- For Sponge Wiping, shared pick-and-place primitives support improvement from 45% to full observed task success, reaching that level one iteration earlier despite fewer new demonstrations.Both conditions receive the same wiping data; sharing reduces new teaching for picking and placing.Source: Figure 4 · p. 8, §4.3 · p. 9
5 LIMITATIONS AND FUTURE DIRECTIONS
The authors identify three future directions: reduce the effort of expanding skill libraries, support unattended operation, and generalize failure detection across tasks.
Acquiring and expanding skills
- Expanding a fixed skill library still requires initial human effort to specify and teach skills, though learning them from video could reduce demonstration effort.Persistent failures could also guide which skills to acquire next, evaluated by how many tasks each addition benefits.Source: §5 · p. 9
Sustained autonomous operation
- People still reset scenes between rollouts, so unattended operation over days or weeks remains a future goal.Learned reset behaviors or task loops could support sustained operation and test whether corrective supervision remains useful as tasks and surroundings change.Source: §5 · p. 9
Generalizing failure detection
- The failure detector requires per-task training, although skill selection and repair verification use a common procedure.A foundation model that anticipates failure from proposed motions could eventually decide both when to shoot and which skill to try.Source: §5 · p. 9
6 CONCLUSION
Skill-space shooting turns reusable skills into corrective supervision and incorporates accepted repairs into task policies. Experiments show repeated autonomous policy gains and reduced teaching needs when skill data is shared across tasks.
Conclusion
- Skill-space shooting uses foundation-model guidance to select skills at policy failures, checks their physical rollouts, and trains policies on accepted repair segments.The detector decides when to invoke a skill; accepted repair segments from completed rollouts update the task policy through DAgger-style training.Source: §6 · p. 9
- Repeated policy gains persist without skill assistance, including improvement from a policy with zero observed initial successes.Source: §6 · p. 9
- Sharing skill data reduces the new teaching needed to prepare another task’s improvement loop.The conclusion links within-task improvement to skill reuse across tasks.Source: §6 · p. 9