Source-linked AI summary
Hierarchical Advantage Weighting for Online RL Fine-Tuning of VLAs from Sparse Episode Outcomes
Tongyan Fang, Siyuan Huang, Naiyu Fang, Ganlong Zhao, Zhongjin Luo, Jianbo Liu, Xiaogang Wang, Ying Dong, Hongsheng Li
TL;DR
Online RL fine-tuning of VLAs needs per-transition supervision despite receiving only binary episode outcomes, while existing scalar signals conflate viability and efficiency. HABC separates these signals with intervention-aware credit assignment, raising success from SFT baselines of 36%/44%/12% to 92%/88%/38% on three contact-rich bimanual tasks.
Problem
Online RL fine-tuning of VLAs receives only binary episode outcomes, while existing scalar advantages conflate viability and efficiency and provide limited transition-level guidance.
Method
HABC uses dual-head viability and efficiency critics, a state-adaptive advantage gate, and intervention-aware credit assignment to produce per-transition actor weights.
Results
92%/88%/38% success, up from SFT baselines of 36%/44%/12%, on three contact-rich bimanual tasks.
Takeaways & Limitations
Viability weighting supports learning when success is rare, while efficiency weighting distinguishes faster completions as success becomes frequent.
Takeaways & Limitations
HABC assumes reliably detected intervention boundaries and is currently evaluated only on single-task fine-tuning.
Abstract
from arXiv · showhide
When pretrained VLA policies are fine-tuned through online RL, each rollout episode produces only a single binary outcome (success or failure), yet the actor update requires per-transition supervision. Existing approaches commonly reduce this sparse outcome to a single scalar reward or advantage signal, which conflates distinct forms of transition-level feedback and provides limited guidance once basic task success becomes achievable. First, a single scalar signal conflates the two objectives of viability and efficiency; once basic success is achieved, the binary label provides no gradient to distinguish efficient completions from slow ones. Second, real-world rollouts mix autonomous and intervention segments; naively assigning episode outcomes across these boundaries introduces incorrect credit assignment. To address these issues, we propose Hierarchical Advantage-Weighted Behavior Cloning (HABC), which trains separate critic heads for these two objectives on different data subsets and combines their outputs with a state-adaptive balance. A state-adaptive gate $g_t$ merges their one-step advantages, prioritizing viability when success is uncertain and shifting to efficiency only when viability is high, and converts the result into per-transition weights on the actor loss. Intervention-aware credit assignment further restricts outcome labels to segments executed by the current policy, preventing supervision from leaking across intervention boundaries. In real-robot experiments on three contact-rich bimanual tasks, HABC raises success from supervised fine-tuning (SFT) baselines of 36%, 44%, and 12% to 92%, 88%, and 38%.
1 Introduction
HABC addresses sparse episode supervision by separating viability and efficiency signals with a dual-head critic, combining their advantages adaptively into transition weights. It also restricts outcome credit to policy-executed segments and improves success over SFT baselines across three tasks.
- Signal decomposition: Sparse binary outcomes encode separable viability and efficiency signals that require different training data and matter at different training stages.Viability is supervised from outcome-labeled windows, whereas efficiency is estimated only from successful trajectories; viability dominates when failures are common, and efficiency matters when success is high.
- Intervention-aware credit assignment: Intervention-aware credit assignment restricts outcome labels to policy-execution segments, avoiding supervision leakage across mixed-control episodes.Naively assigning episode outcomes across policy and human segments can upweight mistakes that triggered intervention or penalize corrective actions.
- Hierarchical signal decomposition: HABC uses viability and efficiency critic heads whose one-step advantages are combined through a state-adaptive gate into bounded per-transition actor-loss weights.The viability head is trained on all outcome-labeled windows, while the efficiency head is trained on successful trajectories only.
- Experimental result: 92%/88%/38% success rates improve over SFT baselines of 36%/44%/12% across three contact-rich bimanual tasks.The reported improvements are from 36%, 44%, and 12% to 92%, 88%, and 38%, respectively.
2 Related Work
Related work spans online RL for robot manipulation, generative action-policy optimization, intervention-based imitation learning, and advantage-weighted or advantage-conditioned actor updates. These approaches use online interaction, policy-gradient estimation, human demonstrations, or critic-derived weights and returns.
- Online RL fine-tuning: Online RL methods improve robot-manipulation performance through off-policy, PPO-style, iterative, trajectory-level, or curriculum-based training.Examples include HIL-SERL, ConRFT, RIPT-VLA, iRe-VLA, VLA-RL, and SimpleVLA-RL.
- RL for generative action policies: Generative action-policy methods optimize sampling processes with policy gradients or combine implicit Q-learning with diffusion-policy extraction.ReinFlow, FPO, DPPO, and RFS use policy-gradient estimation, while IDQL uses implicit Q-learning; ARFM balances advantage preservation and flow-loss variance offline.
- Intervention-based imitation learning: Intervention-based imitation methods aggregate, reweight, adapt, or model human demonstrations and correction data for robot learning.DAgger, HG-DAgger, IWR, Sirius, AIM, RaC, and MILE represent these strategies.
- Advantage-weighted and advantage-conditioned actor updates: Advantage-weighted actor methods derive weights from critic advantages, while alternative approaches extract policies from value estimates, returns, or reward-derived prompts.AWR and AWAC use exp(Â/β), IQL and CQL use value estimates, Decision Transformer uses returns, and Recap and SOP convert reward-derived advantages into prompt tokens.
3 Method
HABC decomposes sparse episode outcomes into separate viability and efficiency critics, combines their one-step advantages with a state-adaptive gate, and restricts outcome attribution to policy-controlled segments. This yields per-transition actor weights while avoiding credit leakage across intervention boundaries.
- Dual-head critics: HABC trains separate viability and efficiency critic heads because sparse outcomes otherwise conflate whether actions preserve task viability with whether they make efficient progress.Viability is emphasized when failures are frequent, whereas efficiency becomes informative once success is reliable.
- Dual-head critics: The viability critic estimates eventual success probability from all outcome-labeled policy-execution windows, including successful and failed episodes.Binary cross-entropy supervision remains informative even when the success rate is low.
- Dual-head critics: The efficiency critic predicts steps to success using successful trajectories only, with non-terminal cost −1 and terminal success target 0.It is trained with Huber loss and distinguishes fast from slow progress as viability approaches 1.
- Adaptive advantage weighting: HABC converts both critics into one-step advantages and combines them with a per-state gate that prioritizes viability at low success probability and efficiency at high success probability.The gate weights low-viability states by viability improvement and high-viability states by efficiency improvement within the same batch.
- Intervention-aware attribution: Outcome labels apply to entire fully autonomous episodes but only to the post-intervention policy suffix when human intervention occurs.Intervention windows provide actor imitation supervision and efficiency targets without receiving the episode outcome label.
4 Experiments
Experiments on three real-robot dual-arm manipulation tasks show that HABC improves sparse-outcome fine-tuning through viability and efficiency weighting, with further gains from intervention reweighting. The learned value heads also generalize across initial states and support recovery-oriented behavior.
- Step 1: Initial online fine-tuning: HABC achieves the highest Step 1 success rate on all three tasks: 60%, 78%, and 22%.The equal-budget comparison initializes all online methods from the same SFT checkpoint.
- Step 1: Initial online fine-tuning: HABC-V surpasses every non-HABC baseline on every task, indicating that soft viability weighting extracts more useful supervision than hard filtering.The baselines include SFT, Imit-DAgger, Imit-Recap, HABC-V, and full HABC.
- Efficiency: Upgrading from HABC-V to HABC reduces successful-trial trajectory length on every task by −55, −162, and −32 frames.Shorter trajectories measure more efficient completions without conflating efficiency with failure rate.
- Step 2: Intervention reweighting: 92% on Pencil Pouch, 88% on Paper Bag, and 38% on Snack Bag are reached by the best HABC+IR checkpoints after continued training.Intervention reweighting is enabled only after the initial HABC phase and applies gt to intervention windows.
- Value head generalization: The viability head’s pv > 0.6 region expands across observed initial pouch-center positions as online fine-tuning proceeds.This indicates generalization beyond memorized training labels.
- Per-transition weighting and recovery: HABC upweights successful grasps and recovery transitions while downweighting stalled, inefficient regrasp motion through separate viability and efficiency signals.Qualitative rollouts show HABC detecting manipulation failures and executing corrective motions, unlike SFT policies that repeat failures or enter unrecoverable loops.
5 Conclusion and Limitations · A Parent Value Head
HABC converts sparse episode outcomes into per-transition behavior-cloning weights using dual-head critics and intervention-aware credit assignment, while leaving the deployed actor unchanged. Its limitations include dependence on reliable intervention boundaries, weak efficiency supervision when success is rare, and evaluation restricted to single-task fine-tuning; the parent value head is retained for compatibility but not actor weighting.
- 5 Conclusion and Limitations: HABC uses a dual-head critic and intervention-aware credit assignment to transform sparse episode outcomes into per-transition behavior-cloning weights without changing the deployed actor.The viability head supports learning when success is rare, while the efficiency head shortens trajectories after success becomes reliable.
- 5 Conclusion and Limitations: Restricting outcome labels to policy-execution segments prevents credit leakage across intervention boundaries.This mechanism depends on reliably detected intervention boundaries; noisy labels would corrupt Vv supervision.
- 5 Conclusion and Limitations: Ve trains only on successful trajectories, making its signal weakest precisely when success is rare.This limitation affects the efficiency-related supervision described for HABC.
- 5 Conclusion and Limitations: HABC is currently evaluated only on single-task fine-tuning, leaving multi-task and cross-embodiment extensions open.Future directions include adaptive gating, multi-step advantage estimation, and denser outcome signals for contact-rich re…
- A Parent Value Head: The pretrained value head Vψ uses a distributional formulation that predicts logits over value bins.It is trained by cross-entropy against a mixed TD/MC target.
- A Parent Value Head: The switch variable ct selects the MC target at episode boundaries and the TD target elsewhere.During online fine-tuning, Vψ continues updating on BS ∪ BO from the pretrained IQL checkpoint.
- A Parent Value Head: Vψ is retained for compatibility with the Imit-Recap baseline but is not used in HABC’s actor weighting.The base VLA model is π0.5.
B Compared Update Rules
This section summarizes the update rules used by HABC and two imitation-style baselines for completeness.
- The section compares the update rule used by HABC with those of two imitation-style baselines.
- HABC is included as one of the methods whose update rule is summarized.
- Two imitation-style baselines are included alongside HABC in the update-rule summary.
B.1 HABC · B.2 Imit-Recap · B.3 Imit-DAgger
HABC updates separate viability and efficiency critics, uses a state-adaptive gate and intervention-aware weighting for actor training. Imit-Recap applies thresholded TD-residual filtering, whereas Imit-DAgger uses equal parts SFT and intervention data without rollout or critic reweighting.
- B.1 HABC: HABC updates the parent value head on SFT and autonomous batches, the viability critic on labeled data, and the efficiency critic on successful autonomous data.The parent value head is not used in actor weighting.
- B.1 HABC: After warmup, HABC computes the gate g_t from detached critic-head outputs on successful autonomous data and, when enabled, intervention data.Before warmup, g_t is set to 1 for all non-SFT samples.
- B.1 HABC: HABC assigns gated weights to successful autonomous transitions and intervention samples, zero weight to failed autonomous transitions, then normalizes and updates the actor on all three batches.Intervention weights use g_t only when intervention reweighting is enabled; otherwise they equal 1.
- B.2 Imit-Recap: Imit-Recap includes an online transition in the actor loss only when its critic TD residual exceeds a validation-set threshold ϵ.This implements Recap’s hard-threshold filtering but is not a full reproduction of its advantage-conditioned actor.
- B.2 Imit-Recap: Imit-Recap updates its value head using SFT and autonomous data, computes one-step advantages for autonomous rollouts, and preserves SFT and intervention weighting rules.SFT weights are 1, while intervention weights remain unchanged under the intervention mask.
- B.3 Imit-DAgger: Imit-DAgger trains the actor on a 50/50 mixture of SFT and intervention data without rollout transitions or critic-derived reweighting.Its scalar intervention weights are 1, with M int i used as the per-dimension action mask.
- B.3 Imit-DAgger: Each Imit-DAgger update samples the mixed batch, sets SFT weights to 1, applies the intervention action mask, and updates the actor on SFT and intervention batches.The recipe does not use rollout transitions or critic-derived reweighting.
C Intervention-Aware Credit Assignment Illustration · D Windowing Details
The method routes outcome supervision by controller and objective, using policy-execution windows for viability and successful windows for efficiency. Intervention-containing episodes label only the post-intervention policy suffix, while earlier policy segments remain unlabeled.
- C Intervention-Aware Credit Assignment Illustration: Vv uses all outcome-labeled policy-execution windows to predict viability, while Ve uses only successful policy or intervention windows to predict progress and efficiency.The two value heads produce advantages that become transition-level actor weights.
- C Intervention-Aware Credit Assignment Illustration: Naive episode-level supervision can assign credit to the wrong controller because the source of the final success or failure label is uncertain.Intervention-aware routing separates policy execution from intervention segments before assigning supervision.
- D Windowing Details: For episodes containing intervention, the trajectory is split into policy-execution segments and intervention segments.This segmentation defines which portions can receive the episode outcome label.
- D Windowing Details: Only the post-intervention policy-execution suffix receives the episode outcome label in intervention-containing episodes.The suffix is executed by the current policy from a corrected state onward.
- D Windowing Details: Earlier policy-execution segments remain in the replay buffer but do not receive outcome labels.This prevents labels from being assigned to policy behavior that occurred before correction and intervention.
- D Windowing Details: For fully autonomous episodes, the full trajectory receives the episode outcome label.No intervention boundary requires restricting the labeled window.
E Data Collection Protocol
Online fine-tuning begins from 200 SFT demonstrations and proceeds in rounds of autonomous rollouts followed by 6k gradient steps, with failed episodes split between human intervention and unassisted failure. Three initial rounds precede continued HABC+IR training, yielding 92% success on Pencil Pouch and 88% on Paper Bag.
- Online Fine-Tuning Schedule: Each task starts with 200 SFT demonstration episodes, followed by rounds collecting 100 autonomous rollouts and training for 6k gradient steps.Each round adds 100 autonomous rollout episodes to the replay buffer.
- Intervention Handling: Approximately half of failed rollouts receive human intervention, while the remainder are recorded as unassisted failures.In intervention episodes, the operator takes over and completes the task.
- Pencil Pouch: 92% success is reached by the final HABC+IR checkpoint for Pencil Pouch after 3 initial online rounds and continued HABC+IR training.The initial-phase checkpoint follows 300 rollout episodes on top of 200 SFT demonstrations.
- Paper Bag: 88% success is reached by the final best HABC+IR checkpoint for Paper Bag under the same 3-round initial schedule followed by continued training.The schedule consists of 3 initial rounds followed by continued HABC+IR rounds.
F Hyperparameters
HABC uses fixed optimization and rollout constants, including a failure penalty, warmup period, discount factor, batch size, and stale-rollout cutoff. All runs use pure TD supervision for Ve with action-expert training disabled.
- Core constants: C = 100 is the episode failure penalty, applied as r = −C on failed-episode transitions for parent-value TD updates and Imit-Recap advantage computation.HABC’s actor weighting does not use this reward directly.
- Core constants: Nwu = 500 warmup steps, Huber δ = 1.0, γ = 0.99, and batch size B = 256 define the training configuration.These constants are used across all HABC runs.
- Run settings: A stale-rollout cutoff of 10000 model indices is used, while all runs retain pure TD supervision for Ve and disable action-expert training.These settings specify rollout freshness and auxiliary-training behavior.
G Weight Statistics
HABC produces non-uniform, task-dependent transition weights, while intervention windows receive near-uniform average weights with concentrated variation. Successful-trial trajectory length serves as an efficiency readout, with reductions from HABC-V to HABC supporting Ve’s intended role.
- Autonomous rollout weights: 0.76, 0.68, and 0.86 are HABC’s mean pre-normalization weights on successful autonomous transitions for Pencil Pouch, Paper Bag, and Snack Bag, respectively.These task-dependent values indicate that HABC weighting is non-trivial rather than near-uniform.
- Comparison with Imit-Recap: 22, 27, and 19 out of every 100 sampled autonomous transitions are retained by Imit-Recap for Pencil Pouch, Paper Bag, and Snack Bag, respectively.The remaining transitions are discarded by a hard threshold, making this filtering more aggressive than HABC’s soft weighting.
- Intervention weights: 1.0, 1.1, and 0.95 are the mean intervention-window weights for Pencil Pouch, Paper Bag, and Snack Bag, respectively, under intervention reweighting.Individual intervention transitions remain non-uniform, but their distribution is more concentrated than autonomous rollout data.
- Trajectory efficiency: Mean trajectory length over successful evaluation trials directly measures efficiency because fewer frames indicate less redundant motion and recovery before completion.Consistent reductions from HABC-V to HABC in Figure 3 support Ve’s intended role.