PaperlayerView exact revision on arXiv

Source-linked AI summary

From Pretraining to Proficiency: Real-World Subtask RL for Long-Horizon Manipulation with Minimal Human Intervention

Sichang Su, Benjamin Yang, Zhiyun Deng, Boyuan Liang, Yip Fun Yeung, Zelin Wang, Lingfeng Sun

arXiv:2609.21788v1cs.ROcs.LG

TL;DR

Long-horizon tasks can fail at a few critical subtasks despite a pretrained policy performing other behaviors well, and full-task demonstrations repeat behaviors that already work. PARTS uses a frozen base policy, agent-generated selectors and verifiers, residual RL, and success-reweighted retraining to target bottleneck subtasks with local rewards. Full-task success increases from 32% to 61% on YAM and from 50% to 95% on Franka; PARTS raises full-task success by more than 25% over existing real-world RL finetuning methods.

  • Problem

    Long-horizon tasks can fail at a few critical subtasks despite a pretrained policy performing other behaviors well, and full-task demonstrations repeat behaviors that already work.

    Source: §I · p. 2

  • Method

    PARTS uses a frozen base policy, agent-generated selectors and verifiers, residual RL, and success-reweighted retraining to target bottleneck subtasks with local rewards.

    Source: §I · p. 2

  • Results

    Full-task success increases from 32% to 61% on YAM and from 50% to 95% on Franka; PARTS raises full-task success by more than 25% over existing real-world RL finetuning methods.

    Source: §I · p. 2

  • Takeaways & Limitations

    Across bimanual and single-arm platforms, PARTS improves full-task success over the base policy and outperforms existing RL fine-tuning methods with fewer recurring forms of human involvement.

    Source: §V · p. 8

Abstract

from arXiv · show

A pretrained robot foundation policy may execute most of a long-horizon task yet repeatedly fail at a few critical subtasks. Collecting additional full-task demonstrations for supervised fine-tuning (SFT) requires operators to repeat behaviors the policy already performs well. Reinforcement learning (RL) fine-tuning offers a promising path to bridge this gap, but existing approaches struggle to solve long-horizon tasks using only sparse rewards. We present PARTS (Policy Adaptation with RL on Targeted Subtasks), a real-world subtask RL framework that concentrates practice at these bottlenecks while allowing training rollouts to proceed with minimal human intervention. The frozen pretrained policy supplies nominal actions throughout execution, while agent-generated selectors and success verifiers activate residual corrections and provide local outcome rewards. These rewards support learning from successful subtasks even when complete-task successes are scarce. Training combines online RL with success-reweighted retraining, and each retrained residual policy is redeployed to collect further experience. Humans identify bottlenecks during setup and perform physical resets when needed. On bimanual YAM and single-arm Franka tasks, PARTS improves complete-task success from 32% to 61% and from 50% to 95%, respectively, using tens of minutes of real-world RL rollouts per task on average. Compared with existing real-world RL fine-tuning methods, PARTS raises full-task success by more than 25% under the same robot-rollout budget while requiring less human involvement.

I. INTRODUCTION

PARTS targets the few subtasks where a pretrained policy fails, using local supervision and residual RL to improve long-horizon execution with minimal human intervention. On YAM and Franka, it improves full-task success under a limited rollout budget.

I. INTRODUCTION

  • The approach focuses practice on consequential subtasks where failures concentrate, including precision operations and preparatory actions that set up later execution.A successful pickup can still leave an object in a pose that makes insertion difficult.Source: §I · pp. 1–2
  • PARTS keeps the pretrained policy frozen for nominal actions and activates bounded residual corrections at human-identified bottlenecks, with executable selectors and verifiers supplying local rewards.Online RL is followed by success-reweighted retraining, and redeployed policies collect further experience; training rollouts need no human corrections, switching decisions, or outcome labels.Source: §I · p. 2
  • Full-task success rises from 32% to 61% on YAM and from 50% to 95% on Franka; PARTS also raises success by more than 25% over existing real-world RL fine-tuning methods with less human involvement.The results use tens of minutes of real-world RL rollouts per task on average.Source: §I · p. 2
  • PARTS is evaluated on bimanual YAM and single-arm Franka tasks, with full-task success assessed across long-horizon execution.The framework is motivated by the human effort required to collect complete demonstrations that repeat behaviors the policy already performs well.Source: §I · pp. 1–2

II. RELATED WORK

Prior real-world RL work improves policies through several forms of supervision and task decomposition, while PARTS localizes practice to base-policy failures using executable contracts. Its architecture combines local rewards, residual policies, and retraining for full-task execution.

II. RELATED WORK

  • Unlike approaches emphasizing global policy improvement, PARTS targets base-policy failures through executable contracts rather than action variance or a fixed contact phase.Related systems also restrict learning to part of a task.Source: A. Real-World RL in Long-Horizon Tasks · p. 2
  • PARTS trains residual policies on bottlenecks using local rewards and per-bottleneck replay, then retrains on successes plus a fraction of failures before resuming online learning.At evaluation, the system can switch between base and residual policies to complete the full task.Source: Figure 2 · p. 3
  • Automatic verifiers provide local outcome rewards instead of rewards derived from human intervention.Source: A. Real-World RL in Long-Horizon Tasks · p. 3

B. Human Supervision for Real-World Policy Adaptation

Real-world policy adaptation often depends on human monitoring or intervention, while long-horizon tasks make sparse terminal rewards and uniformly spent robot time difficult constraints. The problem formulation treats tasks as sequences of subtasks whose bottlenecks can constrain complete-task success.

B. Human Supervision for Real-World Policy Adaptation

  • Prior systems use human corrections, policy takeover or switching, and success/failure labels, whereas PARTS uses executable verifiers and reset programs to localize supervision to bottlenecks.PARTS rollouts require no human corrections or switching decisions, though resets remain part of the setup.Source: B. Human Supervision for Real-World… · p. 3

A. Problem Statement

  • The task is modeled as a sequence of subtasks with entry states and binary outcome criteria, while the available reward is sparse binary feedback for complete-task success.A small offline expert dataset supplies demonstrations for SFT when the pretrained policy has limited zero-shot performance.Source: A. Problem Statement · p. 3
  • A few subtasks with low completion probability can bound full-task success even when the other stages are reliable.A bottleneck may extend to a preceding subtask when its terminal state determines the difficulty of the next stage.Source: A. Problem Statement · p. 3
  • Sparse long-horizon rewards often leave rollouts without feedback when the base policy rarely completes the task, and uniform training budgets leave few attempts at bottlenecks.Source: A. Problem Statement · p. 3

B. Policy Adaptation with RL on Targeted Subtasks

PARTS freezes the pretrained policy and concentrates real-world RL on selected bottlenecks through bounded residual corrections. Local rewards, online TD3+BC, and success-reweighted retraining support repeated practice while preserving nominal behavior elsewhere.

B. Policy Adaptation with RL on Targeted Subtasks

  • The frozen VLA executes every subtask while lightweight residual policies are activated only at identified bottlenecks.Selectors and verifiers govern activation and local rewards; periodic success-reweighted retraining consolidates successes and redeploys the residual for further experience.Source: B. Policy Adaptation with RL on… · p. 4
  • Residual actors condition on base-policy visual features, proprioception, and nominal action chunks, with masks and physical bounds restricting corrections to selected coordinates.Other action dimensions continue to follow the base policy, and untrained actors output zero residuals.Source: B. Policy Adaptation with RL on… · p. 4
  • Each bottleneck uses chunk-level TD3 with behavior-cloning regularization, trained from replay transitions containing states, nominal chunks, residuals, rewards, and terminal flags.Twin critics estimate residual-chunk values using temporal-difference learning over each action chunk.Source: B. Policy Adaptation with RL on… · p. 4
  • The actor imitates successful residuals and can penalize nonzero residuals from failed episodes, while reference dropout reduces copying of nominal chunks.The failed-episode penalty anchors attempts back toward the nominal policy.Source: B. Policy Adaptation with RL on… · p. 4

C. Agentic Scaffolding

PARTS turns human-defined bottleneck contracts into executable programs that supervise policy selection, local success labeling, and retries or resets. Coding agents author the programs, humans review their decisions, and the programs run without generating motor commands.

C. Agentic Scaffolding

  • Humans specify each bottleneck’s entry set, correctable action coordinates and bounds, outcome criterion, motion budget, and a handful of subtask demonstrations.Coding agents author executable selectors, verifiers, and reset policies from these contracts and robot data; humans review labeled decisions and agents revise the code.Source: C. Agentic Scaffolding · p. 4
  • The illustrated tasks include precision-demanding earbud insertion and cable insertion, plus LEGO sorting where small-brick grasping is a recurring bottleneck.Colored borders identify states produced by targeted residual policies.Source: Figure 3 · p. 5
  • The policy selector activates a residual only when observations support its entry conditions, otherwise leaving control with the base policy.It can follow an ordered residual sequence or choose among currently eligible bottlenecks, and can reactivate a residual when its entry recurs.Source: C. Agentic Scaffolding · p. 5
  • The success verifier returns a binary local reward after event gates and contract postconditions are met, requesting human labels for uncertain verdicts.During evaluation, verified success hands control back to the base policy so full-task execution can continue.Source: C. Agentic Scaffolding · p. 5
  • The reset policy retries failures when starting conditions still hold and resets after successful attempts that changed those conditions.The robot resets when feasible; otherwise a human performs the reset.Source: C. Agentic Scaffolding · p. 5

D. Success-Reweighted Retraining and Redeployment

Success-reweighted retraining addresses sparse rewarded experience and the risk that repeated local resets produce overly similar scenes. It retains all successes and some failures, then redeploys a selected checkpoint for further online learning.

D. Success-Reweighted Retraining and Redeployment

  • Online updates can receive few rewarded transitions when a bottleneck’s base success is low, and repeated local restaging can overfit policies to nearly identical scenes.Such policies may fail under the state variation produced by preceding nominal behavior at evaluation.Source: D. Success-Reweighted Retraining and… · p. 5
  • The curated replay keeps every successful episode and a uniformly sampled fraction of failed episodes, increasing rewarded experience while retaining failures as negatives.Fresh actor and critic networks are trained on this fixed dataset.Source: D. Success-Reweighted Retraining and… · p. 5
  • An operator selects a candidate checkpoint, and that checkpoint with its curated replay initializes a new online run for further updates and rollouts.Retraining therefore changes the policy used to collect subsequent data rather than serving only as final policy extraction.Source: D. Success-Reweighted Retraining and… · p. 5

IV. Real-World Experiments

The experiments are framed around whether targeted subtask RL improves complete long-horizon task performance, how it compares with full-task RL under matched rollout budgets, and whether success-reweighted retraining matters.

IV. Real-World Experiments

  • The experiments ask whether targeted subtask RL improves a pretrained VLA on complete long-horizon tasks.Source: §IV · p. 5
  • They compare PARTS with full-task RL fine-tuning methods under a matched robot-rollout budget.Source: §IV · p. 5
  • They test whether success-reweighted retraining matters.Source: §IV · p. 5

A. Tasks and Setup

The evaluation covers three long-horizon manipulation tasks on YAM and Franka, with bottlenecks ranging from precision insertion to repeated grasping. The setup combines human-defined subtask contracts and task-specific outcome and reset procedures with complete-task and per-stage evaluation.

A. Tasks and Setup

  • Earbud insertion on YAM is a dependent chain: a poor case-holding pose affects both insertions, and small clearances and occlusion complicate control and outcome assessment.The three bottlenecks are case preparation (RL0), first insertion (RL1), and second insertion (RL2).Source: A. Tasks and Setup · pp. 5–6
  • LEGO sorting on YAM repeats a grasp ten times within 150 seconds; the bottleneck is grasping with either arm, while the base policy handles approach, transport, and placement.The SFT demonstrations use larger bricks, so the nominal gripper may close too little to retain one even after a well-placed approach.Source: A. Tasks and Setup · p. 6
  • Cable unplug-and-plug on Franka requires aligning and inserting the connector within the port tolerance; insertion is evaluated from states produced by extraction.The task takes 20–120 seconds, while bottleneck subtasks typically last 3–15 seconds.Source: A. Tasks and Setup · p. 6
  • Humans identify bottlenecks, provide a few subtask demonstrations, and specify each contract; human checks or physical resets are used when automated verification or restoration is insufficient.Earbud verifier labels receive human checks; human resets are needed when the robot cannot restore the scene or its hardware cannot perform the reset.Source: A. Tasks and Setup · p. 6
  • Evaluation uses 20 complete episodes per method and reports per-stage success alongside normalized progress to capture partial completion.Earbud progress credits one third per completed stage and cable progress one half per completed stage; LEGO reports mean fraction of bricks sorted before the deadline.Source: A. Tasks and Setup · p. 6

B. Baselines

PARTS is compared with the frozen SFT policy and three real-world RL fine-tuning baselines. The comparison matches robot-rollout time, while methods differ in where they spend it and in their policy updates.

B. Baselines

  • Every RL method starts from the same π0.5-SFT policy and receives the same robot-rollout time, without corrective teleoperation or DAgger-style human interventions during RL rollouts.Baselines use human resets and human reward labeling.Source: B. Baselines · p. 6
  • DSRL explores in the frozen VLA’s latent noise space, while EXPO-FT edits action chunks under Q-guidance and continues updating the π0.5 backbone.Both methods train on the full task; EXPO-FT’s published regime additionally uses human interventions and automatic reward detectors, which this comparison does not use.Source: B. Baselines · p. 6
  • RLT freezes the VLA and trains a small actor-critic after a human-selected handoff, without handing control back to the VLA for the rest of the episode.For LEGO, where the grasp bottleneck recurs and requires repeated switching, RLT is run as full-task RL.Source: B. Baselines · p. 6
  • The matched budget counts elapsed robot time during RL rollouts, excluding resets and update pauses; PARTS spends it inside bottlenecks, unlike the full-task rollout baselines.Wall-clock and compute costs can differ.Source: B. Baselines · pp. 6–7

C. Experimental Results

PARTS improves performance over the base policy and outperforms RL fine-tuning baselines under matched robot-rollout time. Success-reweighted retraining further improves earbud subtask and full-task success.

C. Experimental Results

  • PARTS improves full-task evaluations across all three long-horizon tasks and both robot platforms, including settings with weak base policies.On earbuds, its full-task success is four times the base policy’s rate; on LEGO, progress rises from 54% to 82% with 17 minutes of RL rollouts.Source: C. Experimental Results · p. 7
  • Under the same robot-rollout time, PARTS consistently outperforms all baselines across tasks of varying difficulty.The baselines use sparse terminal rewards over 20–120-second episodes, whereas PARTS uses local rewards over 3–15-second subtask windows with initial success rates above 25%.Source: C. Experimental Results · p. 7
  • Removing success-reweighted retraining reduces both subtask and full-task success on earbuds, where retraining is applied to RL1 and RL2 but not RL0.Restaging RL1 and RL2 produces similar configurations across attempts; the authors report that retraining on curated episodes can help escape local optima.Source: C. Experimental Results · pp. 7–8

V. CONCLUSION

The conclusion presents PARTS as a way to improve pretrained policies by concentrating practice on failing subtasks. It also identifies greater automation of setup, rewards, and recovery as future work.

V. CONCLUSION

  • Across bimanual and single-arm platforms, PARTS improves full-task success over the base policy and outperforms existing RL fine-tuning methods under matched robot-rollout budgets, with fewer recurring forms of human involvement.The framework targets the few subtasks where a pretrained policy fails, with minimal human intervention during training.Source: §V · p. 8
  • Future work targets full autonomy through agents that identify bottlenecks, generate dense subtask rewards, and construct recovery phases.The stated aims include automating setup decisions, accelerating learning when successful outcomes are rare, and reducing reliance on human resets.Source: §V · p. 8

Related papers

Loading 2609.21788v1…