PaperlayerView exact revision on arXiv

Source-linked AI summary

Dissecting Advantage-Guided Post-Training for Vision-Language-Action Policies

Jiahang Cao, Hanye Zhao, Hang Lai, Shenyu Zhang, Xiaoshen Han, Xinghang Li, Futeng Liu, Wanli Peng, Heyun Wang, Yunhong Wang, Jason Li, Yong Yu, Weinan Zhang

arXiv:2609.28161v1cs.RO

TL;DR

Heterogeneous robot data create a challenge for advantage-guided VLA post-training: construction, calibration, and utilization choices are coupled, making their separate effects difficult to identify. This study screens those stages with offline diagnostics before evaluating selected choices in real-robot policy training. Across four bimanual tasks, the resulting recipe improves mean progress and success over SFT initialization by 0.42 and 0.63, respectively.

  • Problem

    Because post-training choices jointly shape how advantages influence learning, it is difficult to attribute performance differences to individual stages, while exhaustive real-robot comparisons are costly.

    Source: §I · p. 1

  • Method

    A sequential evaluation uses stage-specific offline diagnostics to select advantage construction and calibration choices before real-robot evaluation of their policy-training use.

    Source: §I · p. 2, B. Advantage Calibration · p. 3

  • Results

    The recipe combining temporal-difference construction, group-wise calibration, and continuous weighting gains 0.42 in task progress and 0.63 in success over SFT Init across four bimanual tasks.

    Source: §VI · p. 8

  • Takeaways & Limitations

    Offline diagnostics show overall alignment with downstream real-world policy performance, supporting their use to interpret outcomes and select designs without exhaustive evaluation of every combination.

    Source: §I · p. 2

Abstract

from arXiv · show

Advantage-guided reinforcement learning provides a practical way to post-train vision-language-action (VLA) policies using limited robot data. However, its performance depends on several coupled choices, including how critic-derived advantages are constructed, calibrated, and used for policy training. Existing recipes often combine these choices into a single end-to-end procedure, making their individual effects difficult to identify. In this work, we dissect advantage-guided VLA post-training through a controlled empirical study that separates these design choices while accounting for their distinct estimands. We develop stage-specific offline evaluation methods to screen alternative choices efficiently, without requiring extensive real-robot policy evaluations for every possible combination. The staged evaluation identifies a modular recipe that combines temporal-difference advantage construction, group-wise calibration, and continuous advantage weighting. Across four real-world bimanual tasks, the resulting recipe improves mean task progress and success over the SFT initialization by 0.42 and 0.63, respectively. Moreover, the proposed evaluation diagnostics show an overall alignment with downstream real-world performance, supporting their use for interpreting empirical outcomes and selecting advantage-guided post-training designs in practice.

I. INTRODUCTION

Advantage-guided VLA post-training must decide how heterogeneous samples influence learning, but coupled choices make it difficult to identify which design choices improve performance. The paper evaluates construction, calibration, and utilization sequentially, using offline diagnostics to screen choices before real-robot policy evaluation.

I. INTRODUCTION

  • Mixed robot data include successful behavior, failures, and corrective or recovery actions, making sample weighting consequential for policy learning.Equal weighting may expose failures as demonstrations, while aggressive filtering may discard corrective behavior.Source: §I · p. 1
  • Because pipeline choices jointly affect sample weights and exposure, existing end-to-end comparisons cannot isolate individual stage effects, and exhaustive real-robot evaluation is costly.The three stages are advantage construction, calibration, and utilization.Source: §I · p. 1
  • A sequential three-stage protocol screens construction and calibration offline, then evaluates selected choices together at the policy level on real robots.Propagating selections between stages avoids evaluating every cross-stage combination on robots.Source: §I · pp. 1–2, Figure 2 · p. 2
  • The selected pattern favors n-step TD construction, value-based calibration, and continuous weighting; real-world ablations show overall alignment between offline diagnostics and policy performance.Continuous weighting performs best on task progress and final success with the selected construction and calibration.Source: §I · p. 2
  • The study contributes a staged screening protocol, a modular post-training recipe, and real-world evidence across tasks with aligned offline diagnostics.The protocol is intended to reduce costly real-robot evaluations of alternative choices.Source: §I · p. 2

II. RELATED WORK

Prior work adapts supervised and reinforcement fine-tuning, intervention data, and critic-guided learning to VLA post-training. The paper distinguishes its focus by isolating construction, calibration, and utilization under shared experimental conditions.

II. RELATED WORK

  • SFT matches policies to demonstrated or corrected actions [2],, while RFT uses reward or value feedback to favor higher-reward behavior [8], [11],, –.Human-takeover data provide corrective trajectories; HG-DAgger applies interactive dataset aggregation but uniformly supervises intervention data.Source: §II · pp. 2–3
  • Offline RL methods differ in value estimation and policy updates: IQL and AWR [32] use exponential advantage weighting, while CRR supports binary or continuous critic-guided regression.Recent VLA methods adapt relative trajectory advantages [8], [9] or chunk-aligned temporal-difference learning [28]; ALOE [10] and RECAP [35] use distinct value and policy-update approaches.Source: §II · p. 3
  • Prior methods couple advantage construction and utilization across differing settings, whereas this study separates construction, calibration, and utilization under a shared actor, dataset, and training budget.Source: §II · p. 3

III. PRELIMINARIES

The paper formulates VLA post-training as a sparse-reward MDP with action chunks and describes TD-based critic learning and advantage estimation. Construction compares value-based and bootstrapped-return estimates at the chunk time scale.

III. PRELIMINARIES

  • VLA post-training is modeled as an MDP whose state includes language, visual observations, and robot state, and whose actions are chunks executed before the next policy query.The transition state is reached after the chunk's primitive actions have been executed.Source: §III · p. 3
  • The adopted sparse reward is 1 only on a successful episode's terminal transition and 0 otherwise.Source: §III · p. 3

A. Advantage Construction

  • Advantage construction comprises critic learning followed by estimation; TD-based critics learn the action-value function using a Bellman target.The target value network's parameters are updated as an exponential moving average of the value network's parameters.Source: A. Advantage Construction · p. 3
  • The value function may be fitted by expectile regression over Q-values or, for the MC variant, directly to empirical returns.An alternative continuation value comes from Q at a policy-sampled action.Source: A. Advantage Construction · p. 3
  • Q−V compares an action's value with its state-value baseline, while n-step TD compares a bootstrapped return with the current state value.Chunk-level indexing means rewards, discounting, termination masks, and bootstrap states share the action-chunk time scale.Source: A. Advantage Construction · p. 3

B. Advantage Calibration

The section defines group-wise advantage calibration and frames utilization as weighting the supervised policy loss according to calibrated advantages and a selected rule.

B. Advantage Calibration

  • Group-wise normalization calibrates advantages within groups, with ordinary normalization recovered when all samples form one group.The grouping rule assigns each example to a group; a scale floor guards against instability when within-group variance is small.Source: B. Advantage Calibration · p. 3

C. Advantage Utilization

  • The actor objective averages supervised policy losses using nonnegative sample weights over the training batch.For the π0.5 actor, the loss is the flow-matching loss.Source: C. Advantage Utilization · p. 4
  • Each sample’s weight is determined by its calibrated advantage and the selected utilization rule.Source: C. Advantage Utilization · p. 4

IV. STUDY DESIGN

The study separates advantage-guided post-training into sequential construction, calibration, and utilization stages, screening choices offline before policy-level evaluation. Stage I compares critic and advantage constructions using intervention and progress ordering diagnostics.

IV. STUDY DESIGN

  • The protocol screens construction and calibration offline before testing their combined effects through actor training and deployment, avoiding exhaustive real-robot evaluation of every cross-stage combination.It asks which constructions recover action-quality orderings, how calibration affects cross-sub-task comparability, and how utilization affects real-world policy performance.Source: §IV · p. 4

A. Stage I: Advantage Construction

  • Stage I evaluates seven advantage-construction configurations across four value-learning schemes, including IQL, SARSA, ALOE, and Monte Carlo.IQL and SARSA use both Q−V and n-step TD scoring; ALOE uses policy-sampled TD targets, while Monte Carlo is evaluated only with n-step TD.Source: A. Stage I: Advantage Construction · p. 4
  • Because the constructions estimate different quantities, Stage I compares the action orderings they induce rather than their numerical advantage values.The study treats agreement between the two diagnostics as stronger screening evidence than either alone.Source: A. Stage I: Advantage Construction · p. 4
  • The intervention diagnostic Rint measures how often takeover windows receive higher estimated advantages than the immediately preceding policy-controlled windows.The paired windows are equal in length; takeovers occur when human operators judge the policy likely to fail.Source: A. Stage I: Advantage Construction · p. 4
  • The progress diagnostic Rprog measures how often advancing segments receive higher advantages than matched stalled segments within the same sub-task.Segments are identified using trajectory descriptions, visual observations, and robot states; 0.5 indicates chance ordering for both diagnostics.Source: A. Stage I: Advantage Construction · p. 4

B. Stage II: Advantage Calibration

Stage II holds the selected advantage construction fixed and compares calibration grouping rules. Its offline diagnostic targets systematic differences in calibrated advantage levels across sub-tasks.

B. Stage II: Advantage Calibration

  • Stage II varies only the calibration grouping rule after fixing the advantage construction selected in Stage I.The motivation is that sparse-reward, long-horizon sub-tasks differ in state distribution and remaining horizon, which may make raw advantage levels vary with progress.Source: B. Stage II: Advantage Calibration · pp. 4–5
  • Calibration is intended to make advantages reflect relative action quality within decision contexts rather than absolute progress, while making comparable-quality actions comparable across sub-tasks.The diagnostic reuses Stage I’s sub-task partition, but calibration operators do not receive sub-task labels; statistics are computed separately by data source.Source: B. Stage II: Advantage Calibration · p. 5
  • The comparison includes unchanged raw advantages, source-level global normalization, temporal-bin normalization, and value-bin normalization.Temporal bins partition each source by episode-relative position; value bins partition each source by clipped estimated state value.Source: B. Stage II: Advantage Calibration · p. 5
  • Temporal and Value use eight bins per source, with sparse bins falling back to source-level Global statistics and normalization statistics held fixed during Stage III.Bins with fewer than 32 samples use the fallback; the scale floor is set to 0.1 times each source’s global standard deviation.Source: B. Stage II: Advantage Calibration · p. 5
  • The diagnostic uses η² to measure cross-sub-task differences in calibrated advantage levels; lower values indicate smaller differences between sub-task means relative to overall variation.Source: B. Stage II: Advantage Calibration · p. 5

C. Stage III: Advantage Utilization

Stage III compares post-training conditions that share an actor, dataset, and training budget but assign different weights to samples. The conditions distinguish continuous or hard advantage-based weighting from weighting by data source.

C. Stage III: Advantage Utilization

  • DAgger gives unit weight to human demonstrations and takeover segments and zero weight to autonomous rollouts, while SFT Init receives no further actor update.Source: C. Stage III: Advantage Utilization · p. 5
  • Weight assigns each sample a continuous exponential weight based on its calibrated advantage, with β = 1 and c = 2 in the experiments.Source: C. Stage III: Advantage Utilization · p. 5
  • Filter gives unit weight to samples above the training-batch q-quantile and zero weight to the rest, using q = 0.5.It serves as a hard approximation that discards low-advantage samples.Source: C. Stage III: Advantage Utilization · pp. 5–6
  • Weight versus Filter isolates continuous versus binary weighting, while comparing either with DAgger contrasts advantage-based weighting with data-source weighting.All three post-training conditions are also compared with SFT Init to test whether post-training improves over the initialization.Source: C. Stage III: Advantage Utilization · p. 6

V. EXPERIMENT RESULTS

The experiments screen advantage construction and calibration offline before evaluating policy utilization on four real-world tasks, then revisit construction and calibration alternatives in real-world ablations.

V. EXPERIMENT RESULTS

  • The study screens advantage construction and calibration offline before evaluating policy utilization on four real-world tasks.The protocol follows Sec. IV.Source: §V · p. 6
  • Policy utilization is evaluated after the offline screening of construction and calibration choices.Source: §V · p. 6
  • Real-world ablations revisit alternative advantage-construction and calibration choices.Source: §V · p. 6

A. Experimental Setup

The experiments use four bimanual manipulation tasks and a fixed offline dataset for each task, with policies evaluated on a shared robot platform and interface.

A. Experimental Setup

  • The four bimanual tasks span object insertion, long-horizon sequencing, multi-object interaction, and deformable-object handling.Source: A. Experimental Setup · p. 6
  • Each task’s fixed offline dataset combines about 40 hours of expert demonstrations, 100 autonomous rollouts, and 100 human-takeover rollouts.Source: A. Experimental Setup · p. 6
  • The bimanual platform uses two 6-DoF arms, three RGB cameras, and one-second chunks of 30 end-effector actions executed at 30 Hz.All compared policies share the same embodiment, observation and action spaces, deployment interface, and evaluation initial-condition schedule.Source: A. Experimental Setup · p. 6
  • The n-step TD backup spans 10 chunks, or 10 seconds of execution, using γ = 0.96 and n = 10.Source: A. Experimental Setup · p. 6

B. RQ1: Advantage Construction

The construction results favor n-step TD advantages, with IQL retained for later stages; value calibration most reduces sub-task-dependent advantage variation among the tested methods.

B. RQ1: Advantage Construction

  • Across IQL, SARSA, and ALOE critics, n-step TD exceeds chance on both diagnostics, with Rint ranging from 0.639 to 0.773 and Rprog from 0.805 to 0.953.Q−V and Monte Carlo variants are generally closer to chance.Source: B. RQ1: Advantage Construction · p. 6, Figure 4 · p. 7
  • Limited action diversity may weaken Q−V because subtracting similar-action values cancels shared value signal, leaving a small residual vulnerable to estimation noise.The authors contrast this with n-step TD’s use of rewards and value changes along logged continuations.Source: B. RQ1: Advantage Construction · p. 6
  • IQL with n-step TD achieves the highest four-task mean Rint, 0.718, while its mean Rprog, 0.897, nearly matches SARSA’s 0.898.The authors retain this construction for subsequent stages.Source: B. RQ1: Advantage Construction · p. 6

C. RQ2: Advantage Calibration

  • Mean η2 falls from 0.092 for Raw to 0.080 for Global, 0.044 for Temporal, and 0.026 for Value calibration.Value achieves a lower mean η2 than Temporal; the authors carry Value calibration forward and revisit Global and Temporal in real-world ablations.Source: C. RQ2: Advantage Calibration · p. 7

D. RQ3: Advantage Utilization

Continuous advantage weighting performs best among the tested policy-training conditions, and the selected construction and calibration recipe aligns with offline diagnostics across four real-world tasks.

D. RQ3: Advantage Utilization

  • Among the three post-training conditions, Weight has the highest four-task point estimates for progress and success over SFT Init.Its finer-grained sample-level supervision emphasizes higher-scoring actions while retaining nonzero weights for remaining samples, reducing sensitivity to occasional critic misranking.Source: D. RQ3: Advantage Utilization · p. 7
  • With Weight fixed, replacing n-step TD with Q−V generally lowers both metrics, while Global or Temporal calibration degrades performance in most tested cases.Higher Stage-I ordering concordance and lower Stage-II η2 generally correspond to higher real-world progress and success.Source: D. RQ3: Advantage Utilization · p. 7, Figure 5 · p. 8
  • The selected recipe achieves four-task means of 0.86 for task progress and 0.74 for task success, gains of 0.42 and 0.63 over SFT Init.It has the highest point estimates for both metrics on every task.Source: §VI · p. 8

VI. CONCLUSION

  • The conclusion identifies IQL with chunk-level n-step TD and value-based calibration as the selected recipe, paired with continuous advantage weighting.Construction and calibration ablations generally align with offline diagnostics, supporting their use to narrow the design space before costly real-robot evaluation.Source: §VI · p. 8
Loading 2609.28161v1…