PaperlayerView exact revision on arXiv

Source-linked AI summary

FlowPRO: Reward-Free Reinforced Fine-Tuning of Flow-Matching VLAs via Proximalized Preference Optimization

Yihao Wu, He Zhang, Junbo Tan, Xueqian Wang, Zhengyou Zhang

arXiv:2606.05468v1cs.RO

TL;DR

Reliable real-robot post-training for VLAs is difficult because failure signals and rewards are costly to obtain. FlowPRO addresses this with reward-free offline preference optimization tailored to flow-matching action heads, paired intervention data, and dense interpolation-based supervision. Across four long-horizon bimanual tasks, it achieves the highest success rate while outperforming four representative baselines.

  • Problem

    Reliable deployment of post-trained VLAs remains difficult because real-robot demonstrations and task-specific feedback are expensive, while existing gains saturate on hard failures.

  • Method

    FlowPRO combines RPRO’s proximalized flow-matching preference objective with intervention-and-rollback data collection, Smooth Interpolation, and batch mixing.

  • Results

    Across four long-horizon bimanual tasks and two π0-family base policies, FlowPRO consistently attains the highest success rate and shortest completion time, outperforming four representative baselines and all loss-component ablations.

  • Takeaways & Limitations

    The results indicate that FlowPRO is an effective and reliable post-training recipe for flow-matching VLA policies.

  • Takeaways & Limitations

    Evaluation is restricted to a single bimanual platform, and rollback timing and horizon still depend on a human operator.

Abstract

from arXiv · show

Post-training Vision-Language-Action (VLA) models into policies that can be reliably deployed on real robots remains a major bottleneck. SFT and DAgger exploit failure signals only indirectly, and reward-based RL is bottlenecked by the difficulty of real-world reward design and of training reliable critics. We present FlowPRO, a reward-free offline reinforced fine-tuning framework for flow-matching VLAs. Algorithmically, we propose RPRO (Robotic Flow-matching Proximalized Preference Optimization), a preference-optimization objective tailored to the flow-matching action head of VLA models. RPRO pairs a contrastive optimizer with an explicit proximal regularizer that anchors the absolute magnitude of the implicit reward, thereby eliminating the reward-hacking failure mode of plain Flow-DPO. On the data side, a teleoperated intervention-and-rollback paradigm produces naturally paired positive and negative trajectories $(τ^w, τ^l)$ on a real robot from a single operator action; a Smooth Interpolation procedure, combined with batch mixing, then converts these sparse corrections into dense per-state supervision while preserving the base policy's capabilities. On four long-horizon bimanual tasks, FlowPRO attains the highest success rate, outperforming four representative baselines, and ablations confirm the contribution of each loss component.

1 Introduction

FlowPRO addresses unreliable real-robot VLA post-training by combining reward-free preference optimization with intervention-based data collection and dense per-state supervision.

  • 1 Introduction: Existing pipelines either weakly exploit rollout failures, require difficult reward or value modeling, or weaken per-state preference learning and inherit reward hacking in flow-matching VLAs.These limitations motivate FlowPRO’s reward-free offline formulation.
  • 1 Introduction: FlowPRO combines a flow-matching preference objective, paired intervention data, Smooth Interpolation, and batch mixing for offline VLA post-training.Stage 1 trains a base policy, while Stage 2 iteratively optimizes paired corrections and preserves prior capabilities.
  • 1 Introduction: RPRO adds an explicit proximal regularizer to Flow-DPO, anchoring implicit-reward magnitude and eliminating reward hacking while unifying preference pairs with SFT demonstrations.The method is designed for flow-matching VLA action heads.
  • 1 Introduction: A single operator-triggered rollback and teleoperation action yields naturally paired positive and negative trajectories from the same recovery state.Varying the rollback horizon also diversifies pair initial states without separate positive and negative rollouts.
  • 1 Introduction: Smooth Interpolation converts sparse trajectory corrections into dense per-state preference supervision, while fixed-ratio batch mixing combines new corrections, historical preferences, and SFT data.The recipe is intended to improve sample efficiency under scarce real-robot corrections while preserving base-policy capabilities.

2 Related Work

Prior robot-policy post-training methods rely on human corrections, explicit rewards, value models, or trajectory-level preferences, each imposing limitations for real-robot flow-matching VLAs.

  • 2 Related Work: DAgger-style methods address distribution shift through on-policy expert corrections but require continuous supervision and do not exploit suboptimal data through contrastive learning.Vanilla SFT discards rollout failures, while DAgger uses them to trigger correction rather than as a direct optimization signal.
  • 2 Related Work: Reward- and value-based robot fine-tuning depends on explicit rewards, critics, or on-policy interaction, which are difficult to scale on real hardware.Dense rewards are especially difficult to obtain for contact-rich manipulation.
  • 2 Related Work: Preference optimization avoids explicit reward models, but DPO variants inherit likelihood underdetermination; trajectory-level VLA methods additionally dilute per-state signals.FlowPRO extends PRO to continuous action generation in flow-matching VLAs.

3 Method

FlowPRO extends proximalized preference optimization to flow-matching VLAs and builds its training data from teleoperated corrections, interpolation, and mixed replay.

  • 3 Method: FlowPRO consists of preference-optimization foundations, RPRO for flow matching, human-in-the-loop data collection, and a training-data composition pipeline.The method begins with an SFT base policy and then performs iterative offline fine-tuning.
  • 3.1 Preliminaries: DPO constrains only relative preferred-versus-dispreferred likelihood, allowing both likelihoods to decrease and producing reward hacking; PRO adds a proximal regularizer anchored on unobserved actions.This proximal construction is extended to flow matching as RPRO.
  • 3.2 RPRO: Robotic Flow-matching Proximalized Preference Optimization: Flow-matching preference optimization uses velocity-field regression as a tractable proxy for negative log-likelihood, defining an implicit reward from current and reference flow-matching losses.The action head transports Gaussian noise to action chunks along a linear interpolant.
  • 3.2 RPRO: Robotic Flow-matching Proximalized Preference Optimization: RPRO combines a contrastive optimizer with a symmetric regularizer minimized at zero implicit reward, thereby anchoring absolute reward magnitude and preventing likelihood underdetermination.The full objective also includes an SFT regression term on positive actions.
  • 3.2 RPRO: Robotic Flow-matching Proximalized Preference Optimization: On identical pairs, contrastive gradients cancel, leaving the proximal and SFT terms to anchor the policy near the reference while moving toward the SFT target.This permits identical-pair samples to be routed through the same RPRO loss.
  • 3.3 Human-in-the-Loop Data Collection: Intervention-and-rollback records the policy’s erroneous continuation and the operator’s correction from the same restored state, producing directly paired preference trajectories.The rollback horizon is operator chosen and can vary across interventions.
  • 3.4 Training Data Composition: Smooth Interpolation synthesizes missing positive actions on negative states, while positive-trajectory and SFT samples use identical preferred and dispreferred actions so only regularized SFT-like updates remain.Negative-state bridges use Bézier position interpolation, Slerp orientation interpolation, and linear gripper interpolation.
  • 3.4 Training Data Composition: Fixed-ratio batches mix current preferences, historical preferences, and SFT data to target new failures, prevent regression, and mitigate catastrophic forgetting.The current preference set receives the largest share, while replay and SFT preserve prior capabilities.

4 Experimental Evaluation

FlowPRO is evaluated on four long-horizon bimanual tasks against representative baselines and loss-component variants. It strictly dominates all baselines across eight task–base strata, while ablations show that the full RPRO objective improves both ID and OOD performance.

  • 4.1 Experimental Setup: The evaluation uses four long-horizon bimanual tasks, three training seeds, and 100 randomized rollouts per seed for each success-rate estimate.The tasks are PACK, CAP, USB, and CASE; results report cross-seed mean ± standard deviation.
  • 4.2 Experimental Results: FlowPRO strictly dominates DAgger, DAgger-Buffered, PI0.6*, and TPO across all 8 task–base strata, with p < 10^-3 against each baseline.Final success rates are reported after K=3 rounds on four real-robot tasks; Table 1 reports success rate and completion time.
  • 4.2 Experimental Results: RPRO outperforms trajectory-wise TPO by 3–7 pp on every stratum, consistent with dense per-state preference signals from Smooth Interpolation.Trajectory-level contrast dilutes per-state learning signals and requires memory-intensive trajectory storage that can force state subsampling.
  • 4.2 Experimental Results: RPRO exceeds PI0.6* by 2–7 pp on identical preference data by injecting preference guidance directly into per-state, per-chunk action generation.PI0.6* instead relies on a single conditioning token and a pure regression objective to discover improved and unimproved partitions.
  • 4.2 Experimental Results: RPRO gains 8–15 pp over vanilla DAgger, while DAgger-Buffered gains 2–7 pp over vanilla DAgger, highlighting the value of negative trajectories and correction-focused batch weighting.DAgger uses positive samples only, whereas RPRO contrasts positive and negative trajectories; the batch schedule up-weights informative failure states.
  • 4.2 Experimental Results: Removing the SFT term lowers SR from 94%/91% to 88%/81% (ID/OOD), while removing the proximal regularizer lowers it to 87%/70%.The SFT term reinforces base skills and accelerates fitting to positive actions; the proximal regularizer prevents unchecked reward growth and distribution contamination.
  • 4.2 Experimental Results: Removing the contrastive term lowers SR to 90%/75%, with the largest degradation under OOD initial conditions where negative-action supervision helps avoid nearby failure modes.The ablation compares SFT alone with the full RPRO objective on PACK.
  • 4.2 Experimental Results: Removing both the SFT anchor and proximal regularizer collapses SR to 13%/5%, whereas full RPRO strictly dominates all loss-component variants on PACK under ID and OOD conditions.Plain DPO drives the positive-action reward below zero toward the −102 scale despite a positive relative margin, exposing reward hacking.

5 Conclusion

FlowPRO is presented as a reward-free offline fine-tuning framework for flow-matching VLAs, combining RPRO with teleoperated intervention-and-rollback preference data. Across four long-horizon bimanual tasks, it achieves the strongest reported success and completion-time results while removing Flow-DPO’s reward-hacking behavior.

  • Across four long-horizon bimanual real-robot tasks and two π0-family policies, FlowPRO achieved the highest success rate and shortest completion time, outperforming four baselines and all loss-component ablations.The conclusion identifies these as consistent results across the evaluated tasks and base policies.
  • FlowPRO combines the RPRO loss with teleoperated intervention-and-rollback preference construction in a reward-free offline fine-tuning framework.
  • Diagnostic implicit-reward curves showed that RPRO’s proximal regularizer removes the reward-hacking failure mode exhibited by Flow-DPO.
  • With only a small number of human interventions, FlowPRO converts scarce on-robot corrections into deployment-grade policy improvements.

6 Limitations and Future Work

The evaluation is limited to a single bimanual platform, leaving generalization to mobile and dexterous manipulation untested. Rollback timing and horizon selection also currently depend on a human operator.

  • Evaluation is restricted to a single bimanual platform, so generalization to mobile and dexterous manipulation remains future work.
  • The human operator currently decides when to roll back and how far the rollback horizon Δ should extend.The paper proposes a learned failure detector as a future way to trigger rollback autonomously.

A Asymptotic Vanishing of the Hyper-Response Reward in Continuous Action Spaces

In continuous action spaces with absolutely continuous policy and reference distributions, the hyper-response reward vanishes exactly rather than approximately. This result justifies removing that term from the flow-matching PRO objective, subject to the stated distributional assumption.

  • Under absolute continuity with respect to Lebesgue measure, the hyper-response set excludes only the two observed actions and has full probability under both policies.The proposition states πθ(H | s) = πref(H | s) = 1 for any finite preference pair.
  • Flow-matching VLA heads satisfy the absolute-continuity assumption when their noise prior is full-rank, as in the π0 architecture described here.
  • Finite-precision sampling introduces at most a quantitative slack |rθ(s, H)| ≤ O(δ^dim), while the training loss uses a continuous flow-matching surrogate rather than directly evaluating singleton probabilities.For the stated 1000-dimensional example, the discretization effect is described as many orders of magnitude below typical implicit-reward scales.
  • The hyper-response reward therefore satisfies rθ(s, H) = 0 exactly in the population limit, unlike the approximate vanishing argument used for discrete LLM actions.In the discrete setting, the observed actions generally retain nonzero probability mass; continuous actions instead have zero singleton mass.
  • The exact vanishing result justifies dropping the hyper-response term from the contrastive-plus-regularizer objective in Eq. (5).

B Derivation of the Gradient Property for Identical Pairs

For identical positive and negative actions, the contrastive gradient cancels and the proximal term acts as a bounded, state-adaptive trust-region anchor. The section also describes the trajectory-matching metric, Smooth Interpolation safeguards, iterative training loop, and hardware platform.

  • B Derivation of the Gradient Property for Identical Pairs: When aw = al, the contrastive term’s gradient difference, ∇θrθ(s, aw) − ∇θrθ(s, al), equals 0, so the term vanishes automatically regardless of prefactor −σ(0) = −1.
  • B Derivation of the Gradient Property for Identical Pairs: The regularizer opposes SFT after πθ overshoots πref, assists it when it trails πref, and switches off at r = 0.Its magnitude is bounded and grows with deviation from the reference policy.
  • C Trajectory Point Distance Metric: Trajectory matching weights end-effector position, orientation, and gripper width using d(M, M′) = ||pM − pM′||2 + 0.5·dgeo(RM, RM′) + 0.2·|gM − gM′|.The weights 0.5 and 0.2 were chosen empirically.
  • D Smooth Interpolation Algorithm: Smooth Interpolation transitions from M to J with a cubic Bézier position bridge, Slerp orientation, and linear gripper interpolation before tracking the positive chunk.The bridge uses ρ = 0.7, while the source tangent is omitted to avoid bending toward the erroneous direction.
  • E Physical Plausibility of Smooth Interpolation: Physical-plausibility safeguards combine small 50 Hz displacements, sampling M from [0, L − H], and RPRO steering that makes residual-risk states unreachable at deployment.Across K = 3 rounds and four tasks, no rollout failure was attributed to interpolation artifacts.
  • F Iterative Training Procedure: FlowPRO first trains on DSFT, then repeats data collection and RPRO updates for K rounds using mixed preference and SFT batches on the Dobot XTrainer platform.The platform has two 6-DoF arms, parallel-jaw grippers, one global RGB camera, and two wrist cameras.

G.2 Training Setup

Training uses a shared flow-matching setup with fixed action horizons, three RPRO rounds, task-independent loss coefficients, and repeated seeded evaluation.

  • G.2 Training Setup: The policy outputs H = 50 action steps, corresponding to 1.0 s of motion at the 50 Hz control rate.Inference uses 10 denoising steps per action chunk.
  • G.2 Training Setup: Stage 1 trains the SFT base policy for 60,000 optimizer steps with per-GPU batch size 16, corresponding to global batch size 64 across four GPUs.
  • G.2 Training Setup: RPRO uses λPRO = 1/3, λSFT = 1, and β = 3.5 across all four tasks.These are the only RPRO-specific hyperparameters added to the base flow-matching pipeline.
  • G.2 Training Setup: Each main-table cell averages three random seeds, with 100 ID and OOD rollouts evaluated per seed.Other hyperparameters are inherited unchanged from the base flow-matching policy.

H Task Details

The evaluation covers four long-horizon bimanual tasks whose failures span precision insertion, coordinated assembly, and deformable multi-stage manipulation. Their pipelines are detailed stage by stage in Fig. 7.

  • H Task Details: PACK requires two sequential container insertions with sub-centimeter alignment, while rim collisions and tilted wedging are difficult for SFT to recover from.Its failures occur near the end of the trajectory, where SFT receives no negative-outcome signal.
  • H Task Details: The complete per-stage pipelines for PACK, CAP, USB, and CASE are illustrated in Fig. 7.The figure extends the main-text pipeline with every sub-stage.
  • H Task Details: CAP stresses three localized failure modes: grasping an upright pen, grasping a flat cap, and aligning both objects during free-floating assembly.The difficulties occur across early and late rollout segments and involve both arms.
  • H Task Details: USB tests sub-millimeter two-arm insertion, where bezel collisions and tilted plugs can prevent full seating because the socket has almost no clearance.The left arm holds the strip while the right arm inserts the USB lamp.
  • H Task Details: CASE is the most long-horizon task, chaining six coordinated stages whose early errors propagate through precision zipping and deformable-case interactions.The task includes picking, unzipping, placing, holding open, inserting a pen, and rezipping.

I Baseline Methods

The baselines span positive-only imitation, controlled batch mixing, advantage conditioning, and trajectory-wise preference optimization. Their formulations isolate whether gains arise from data composition, preference losses, reward granularity, or proximal and SFT terms.

  • I Baseline Methods: DAgger aggregates only positive expert corrections and retrains by behavior cloning, discarding negative trajectories and leaving the SFT-to-correction ratio uncontrolled.DAgger-Buffered retains positive corrections separately and applies FlowPRO’s controlled mixing schedule to isolate batch composition.
  • I Baseline Methods: PI0.6* reconditions the flow-matching expert on binary positive/negative labels and uses classifier-free guidance to steer generation toward improvement.This isolates conditioning-based policy improvement from contrastive-loss optimization using the same preference data.
  • I Baseline Methods: TPO replaces RPRO’s per-state reward gap with a trajectory-summed reward proxy while retaining the same preference-loss family and SFT term.The comparison isolates state-wise versus trajectory-wise preference construction.
  • I Baseline Methods: The four comparators start from the same SFT checkpoint and use the same iterative data-collection protocol, enabling controlled comparisons across loss and data-design choices.The evaluation also includes a user study of the teleoperated data-collection system.
  • J Loss Formulations for Baselines and Ablations: The ablation path compares SFT, DPO, DPO + SFT, PRO without SFT, and full RPRO to separate contrastive, proximal, and SFT contributions.DPO + SFT removes the proximal regularizer, whereas PRO removes the SFT term.

K User Study on the Data-Collection System

The user study evaluates the teleoperated data-collection system, while the statistical appendix tests RPRO’s cross-stratum advantage and directional consistency. User responses were favorable across most dimensions, and RPRO significantly outperformed all baselines across task–base strata.

  • K User Study on the Data-Collection System: The user study asked operators who completed sessions across PACK, CAP, USB, and CASE to rate five system properties on a 5-point Likert scale.The dimensions were Effortless, Learnable, Responsive, Lightweight, and Reusable.
  • K User Study on the Data-Collection System: Operators rated the system favorably on all five dimensions, with agreement reaching roughly 80% or more for Effortless, Learnable, Responsive, and Reusable.Lightweight remained majority-positive at around 56%, with about 14% disagreeing or strongly disagreeing.
  • L.1 Test setup, assumptions, and multiplicity control: The statistical tests model each cell as Binomial(n = 100, p), use Wilson 95% intervals, and pool task–base evidence with CMH tests.Bonferroni correction uses α/4 = 0.0125 across the four baseline-comparison families.
  • L.2 Cochran–Mantel–Haenszel test, stratified by task and base policy: The CMH design treats each of the 8 task–base combinations as a stratified 2×2 table comparing RPRO with a baseline on success versus failure.The analysis uses one-sided tests because RPRO’s observed direction was favorable in every stratum.
  • L.2 Cochran–Mantel–Haenszel test, stratified by task and base policy: RPRO significantly outperformed each baseline across all 8 task–base strata, with Bonferroni-corrected CMH p < 10^-3 for every comparison.The analysis covers four tasks and two base policies, with 100 rollouts per method in each stratum.
  • L.3 Stratified sign test on directional consistency: RPRO was strictly best in all 8 strata, with no ties or losses, and the one-sided sign test gave p = 3.9×10^-3.The winning success rates ranged from 92% to 99% for RPRO across the task–base combinations.

L.4 Loss-component ablations on PACK (Q3 & Q4)

PACK ablations show that each RPRO component contributes statistically detectable improvement, while removing the proximal and SFT terms causes severe degradation. The odds-ratio interval is omitted for the DPO comparison because very small baseline counts make its Wald approximation unreliable.

  • L.4 Loss-component ablations on PACK (Q3 & Q4): The RPRO-versus-DPO pooled odds-ratio confidence interval is omitted because baseline success counts as low as 5/100 make the log-odds Wald interval unreliable.The score-based CMH p-value remains reported because it does not require that normal approximation.
  • L.4 Loss-component ablations on PACK (Q3 & Q4): Adding the SFT term to PRO improves PACK success at p ≈ 6×10^-3, while adding the contrastive PRO term to SFT improves it at p ≈ 1.2×10^-3.The contrastive-term gain is driven primarily by the OOD stratum, with z = 3.01 and p ≈ 10^-3 at the cell level.
  • L.4 Loss-component ablations on PACK (Q3 & Q4): Removing both the proximal regularizer and SFT term collapses PACK success to 13%/5%, with p < 10^-50.The comparison is RPRO versus plain DPO without the SFT term.
  • L.4 Loss-component ablations on PACK (Q3 & Q4): Replacing the proximal regularizer with plain DPO degrades PACK success at p ≈ 3.1×10^-5, confirming a measurable contribution from proximalization.All four ablation-family CMH p-values remain significant after Bonferroni correction.
Loading 2606.05468v1…