PaperlayerView exact revision on arXiv

Source-linked AI summary

$π$-StepNFT: Wider Space Needs Finer Steps in Online RL for Flow-based VLAs

Siting Wang, Xiaofeng Wang, Zheng Zhu, Minnan Pei, Xinyu Cui, Cheng Deng, Jian Zhao, Guan Huang, Haifeng Zhang, Jun Wang

arXiv:2603.02083v2cs.ROcs.CV

TL;DR

Flow-based VLAs make online RL difficult because multi-step sampling renders exact likelihoods intractable, while stochastic exploration can become misaligned under coarse supervision. π-StepNFT addresses this with a critic- and likelihood-free, step-wise contrastive framework, and reports improved LIBERO performance and stronger ManiSkill OOD generalization. The paper presents this design as a scalable approach for fine-tuning generalist robot policies.

  • Problem

    Multi-step ODE sampling makes flow-based VLA action likelihoods computationally intractable, limiting direct application of standard online RL.

  • Method

    π-StepNFT uses SDE-based exploration, immediate-next-step supervision, and logistic contrastive ranking without auxiliary value networks or likelihood computation.

  • Results

    π-StepNFT improves LIBERO performance over SFT by 32.9% and outperforms critic-based baselines by 11.1% in unseen ManiSkill scenarios.

  • Takeaways & Limitations

    Wider exploration paired with finer-grained guidance supports few-shot LIBERO gains and superior OOD generalization on visually diverse ManiSkill tasks.

  • Takeaways & Limitations

    The oracle outcome-conditioned improvement direction is a reference quantity that is not directly observable from online rollouts.

Abstract

from arXiv · show

Flow-based vision-language-action (VLA) models excel in embodied control but suffer from intractable likelihoods during multi-step sampling, hindering online reinforcement learning. We propose \textbf{\textit{$\boldsymbolπ$-StepNFT}} (Step-wise Negative-aware Fine-Tuning), a critic-and-likelihood-free framework that requires only a single forward pass per optimization step and eliminates auxiliary value networks. We identify that wider exploration spaces necessitate finer-grained, step-wise guidance for alignment. Empirically, $π$-StepNFT unlocks latent potential on LIBERO with competitive few-shot robustness. Moreover, it achieves superior generalization on ManiSkill, outperforming value-based baselines in OOD scenarios by preventing overfitting to multimodal features. This property offers a scalable solution promising for complex real-world applications.

1. Introduction

Flow-based VLAs face a tension between narrow deterministic exploration and wider but misaligned stochastic exploration, while multi-step likelihoods hinder direct online RL. π-StepNFT addresses this with SDE exploration, step-wise supervision, and likelihood-free contrastive ranking, achieving strong LIBERO and ManiSkill results.

  • Motivation: Flow-based VLAs offer broad capabilities, but multi-step ODE sampling makes exact action likelihoods computationally intractable for online RL.Likelihood gradients require expensive Jacobian trace estimation or solver backpropagation.
  • Motivation: Deterministic ODE rollouts constrain exploration to a narrow expert manifold, limiting recovery after deviations during testing.The narrow manifold provides a strong behavioral prior but lacks exploratory capacity for self-improvement.
  • Motivation: SDE sampling widens exploration, but coarse terminal supervision can misalign policies because injected noise accumulates across sequential denoising steps.Terminal matching produces unstable, high-variance gradients and corrections that lag behind on-policy drift.
  • π-StepNFT: π-StepNFT is critic- and likelihood-free, eliminates auxiliary value networks, and requires only one forward pass per optimization step.The design targets multimodal overfitting associated with auxiliary critics.
  • π-StepNFT: π-StepNFT combines SDE-based exploration with step-wise ranking guidance that targets the immediate next denoising state for finer alignment.Its objective uses noise-aware local guidance and logistic contrastive ranking to reinforce successful exploration.
  • Empirical validation: On LIBERO, π-StepNFT improves over SFT by 32.9%, while on ManiSkill it outperforms critic-based baselines by 11.1% in unseen scenarios.The reported results cover few-shot robustness on LIBERO and visually diverse OOD generalization on ManiSkill.

2. Related Works

Prior work addresses intractable likelihoods in generative policy optimization through explicit gradients, reward weighting, preference-based ranking, or likelihood approximations. These approaches differ in computational cost, gradient variance, and their ability to balance exploration width with supervision granularity.

  • Connection to embodied control: SDE-based likelihood approximations facilitate exploration through noise injection, yet existing methods often struggle to balance exploration width with supervision granularity.π-StepNFT extends this efficient paradigm to embodied control and its sequential supervision gap.
  • Optimization paradigms: Explicit gradient and advantage methods treat denoising sequentially but often require expensive backpropagation through solvers.They directly use policy-gradient-style likelihood or advantage information.
  • Optimization paradigms: Reward-weighted methods avoid exact likelihoods by reweighting regression targets, but can suffer from high-variance gradient estimation.Their main trade-off is lower likelihood complexity versus noisier optimization signals.
  • Optimization paradigms: Preference and contrastive methods provide a more stable likelihood-free alternative by aligning distributions through ranking.Diffusion-NFT is identified as a likelihood-free framework using implicit forward-process updates.

3. Preliminaries

Flow-based VLA policies transform noise into continuous actions through a time-dependent vector field and discretized denoising dynamics. Their deterministic and stochastic samplers support efficient generation and exploration, but likelihood intractability motivates likelihood-free RL fine-tuning.

  • Notation: The sampler uses continuous denoising time t, while environment steps index the episode trajectory; embodied control typically uses few solver steps for real-time operation.Keeping the two indices distinct avoids ambiguity between RL dynamics and flow dynamics.
  • Flow matching: A VLA flow policy generates continuous actions x0 from Gaussian noise x1 using a time-dependent vector field vθ(x,t,c).Flow matching trains the vector field to approximate the target field along interpolated states.
  • ODE sampling: Deterministic ODE sampling integrates dx = vθ(x,t,c)dt from t = 1 to t = 0, but its trajectory lacks the exploratory capability required for reinforcement learning.Euler discretization produces the next solver state using a step size δt.
  • SDE sampling: SDE sampling injects Gaussian noise through an Euler-Maruyama update, widening exploration while preserving the marginal distribution.The resulting one-step transition has a Gaussian density whose mean is an affine function of the network output.
  • SDE sampling: The affine transition mean enables gradients to propagate from transition targets to policy parameters without backpropagating through the ODE solver.This relationship supports efficient likelihood-free optimization.
  • RL fine-tuning: π-StepNFT collects SDE rollout transitions with terminal rewards, samples denoising steps, and optimizes velocity predictions with drift regularization.The algorithm maintains an experience buffer and updates the policy across collected solver transitions.
  • Likelihood gap: Standard policy gradients require action log-likelihoods, but flow-based policies make these expensive and unstable because likelihood computation integrates Jacobian traces along the generation trajectory.This likelihood gap motivates the paper’s likelihood-free approach.

4. Method

π-StepNFT expands exploration with SDE rollouts and supplies finer, step-wise supervision through mirrored transition comparisons. Its likelihood-free objective uses episode outcomes to align updates with successful transitions while avoiding the implicit separation penalty of weighted-MSE.

  • Framework: π-StepNFT is an online RL framework for flow-based VLAs that uses step-wise supervision and does not require auxiliary critics.The method is designed around solver transitions rather than terminal-only outputs.
  • Wider Space: SDE-based rollouts inject controlled noise, expanding the behavioral manifold beyond deterministic ODE trajectories.The wider exploration space lets the policy traverse adjacent states around expert trajectories.
  • Finer Steps: Immediate transitions xt → xt− with variance normalization replace unstable terminal-x0 supervision under accumulated rollout noise.This provides finer-grained, lower-variance local gradients for alignment.
  • Step-wise Transitions: Mirrored velocity candidates are constructed symmetrically around the rollout policy along the update direction, with β controlling the deviation.The candidates induce Gaussian transition means with shared covariance, enabling variance-normalized step errors.
  • Contrastive Objective: Episode labels rank mirrored branches by their likelihood on the observed transition, increasing the positive preference for successes and reversing it for failures.The computable ranking surrogate replaces an inaccessible outcome-conditioned oracle ratio.
  • Objective Comparison: The logistic ranking loss preserves directional alignment by removing weighted-MSE’s quadratic separation penalty that suppresses policy updates.Its push-pull behavior uses both branches and is associated with sharper gradients, faster convergence, and higher asymptotic performance.

5. Experiments

Experiments evaluate π-StepNFT on LIBERO and ManiSkill, then isolate the effects of stochastic exploration, step-wise supervision, contrastive ranking, critic removal, and hyperparameters. Results show improved few-shot performance, stronger OOD generalization, and more stable training under the proposed design.

  • Experimental Setup: π-StepNFT is evaluated on LIBERO’s four suites and ManiSkill’s compositional PutOnPlateInScene OOD setting.LIBERO reports average success over 500 episodes per suite; ManiSkill evaluates 4,352 compositional tasks.
  • LIBERO: Unlocking potential from few-shot SFT: 90.5% and 94.0% average success rates are achieved by π-StepNFT from π0 and π0.5 few-shot SFT baselines, respectively.The corresponding SFT baselines start at 57.6% and 77.1%.
  • LIBERO: Unlocking potential from few-shot SFT: 86.7% versus 81.4% on π0 Long shows π-StepNFT outperforming critic-free GRPO, while PPO retains an advantage on long-horizon tasks.The comparison indicates competitive guidance without estimating advantages, although critic-based temporal credit assignment remains beneficial for long horizons.
  • ManiSkill: Critic-free generalization: 50.4% OOD average for π0 is 11.1% above PPO, while π0.5 reaches 59.5% versus 49.3%, with the method nearly doubling π0 Semantic-shift success to 49.1%.The results are reported on ManiSkill’s visually diverse OOD settings.
  • Impact of Stochastic Exploration: SDE exploration improves performance only with noise-aware mean correction, whereas deterministic ODE rollouts plateau early.The ablation links effective exploration to a learning signal aligned with the noisy transition and policy velocity field.
  • Regression Target Granularity: Step-wise xt− supervision remains stable under aggressive updates, while terminal x0 regression causes instability and requires conservative synchronization.The local target accelerates convergence under stochastic rollouts by addressing distribution shift from active exploration.
  • Objective Formulation: Ranking vs. wMSE: Combining positive and negative branches outperforms single-branch objectives and weighted-MSE by enforcing a contrastive push-pull preference.Weighted-MSE degenerates to fitting one branch in the binary-reward setting, leaving one-sided supervision.
  • Necessity of Value Estimation: Sparse binary trajectory outcomes remain competitive with dense value estimates and produce smoother training for general manipulation tasks.The probability-based formulation can also accept critic-learned step-wise success probabilities for finer credit assignment.

6. Conclusion

The paper presents π-StepNFT as a critic-and-likelihood-free framework for flow-based VLAs. It uses finer-grained guidance for wider exploration, improves LIBERO few-shot performance and ManiSkill OOD generalization, and avoids auxiliary value networks.

  • 6. Conclusion: π-StepNFT structurally eliminates auxiliary value networks and requires only a single forward pass per optimization step.It is presented as a critic-and-likelihood-free framework for flow-based VLAs.
  • 6. Conclusion: Wider exploration spaces require finer-grained, step-wise guidance for effective alignment.This is the paper’s central design conclusion.
  • 6. Conclusion: π-StepNFT unlocks latent potential in LIBERO few-shot SFT settings and achieves superior OOD generalization on ManiSkill by preventing multimodal overfitting.The conclusion frames the approach as a scalable paradigm for fine-tuning generalist robot policies in complex real-world scenarios.

Impact Statement

π-StepNFT is presented as a likelihood-free, critic-free framework intended to improve embodied-agent training efficiency, robustness, and accessibility. The paper connects these properties to lower computational barriers and improved OOD behavior.

  • Broader impact: The framework is positioned as supporting more accessible and sustainable robotic learning beyond its algorithmic contribution.The impact statement links the method to broader participation and embodied-AI research accessibility.
  • Accessibility: π-StepNFT uses a likelihood-free, critic-free approach with a single forward pass per optimization step, reducing hardware demands for training large-scale VLAs.The paper frames this reduced overhead as lowering barriers for smaller labs and academic groups.
  • Safety and robustness: Improved OOD generalization may help agents behave more reliably in unstructured real-world settings.The statement is presented as a potential safety and robustness benefit rather than a demonstrated deployment guarantee.

A. Theoretical Analysis and Proofs

The theoretical section derives the solver’s affine mean form and establishes the variance-normalized error relationship used by the method. These results support a likelihood-ratio interpretation of branch comparisons.

  • Solver derivation: The flow-SDE solver’s next mean is expressed in affine form as µt(v) = Ut(xt, t) + Bt(t)v.The derivation obtains the coefficients of xt and v from the Euler–Maruyama discretization.
  • Solver derivation: The derivation shows that the solver mean matches the affine representation used in Equation 3.This connects the explicit coefficient calculation to the solver formulation used later in the analysis.
  • Likelihood relationship: For two Gaussian branches sharing covariance, subtracting log densities cancels the normalization constant and yields a likelihood-ratio relation.The proof begins from the Gaussian log-density and compares the two branch distributions.

A.3. Proof of Proposition 4.3 (Bayes Monotonicity)

The analysis formalizes outcome-conditioned transition comparisons, constructs mirrored local branches, and relates ranking-based updates to likelihood and alignment signals. It also states that general-valued rewards need not recover a single oracle direction without additional assumptions.

  • Bayes monotonicity: Bayes’ rule expresses posterior success probability as a monotone function of the oracle likelihood ratio.With prior success and failure probabilities positive, the derivative of the posterior with respect to the likelihood ratio is strictly positive.
  • Branch construction: The method replaces the unobservable outcome-conditioned oracle comparison with symmetric perturbations around the rollout velocity and ranks branches by observed-transition likelihood.The mirrored branches are formed along the update direction using a trust-region scale β.
  • Alignment analysis: Under the binary success setting, posterior expectation and oracle velocity decompositions support an alignment interpretation for sufficiently small policy updates.The analysis uses the affine one-step mean and shared covariance to connect local gradients with an oracle mean gap.
  • Alignment analysis: For general r ∈[0, 1], the conditional expected update is governed by correlation between the terminal signal and local rollout residuals rather than a single oracle mean-gap direction.Recovering one oracle direction requires additional assumptions relating the reward signal to latent optimality.
  • Ranking calibration: π-StepNFT minimizes softplus(y(E+ −E−)), a convex upper bound on ranking errors, whereas wMSE penalizes branch separation through a squared-error term.In the binary case, the ranking objective pulls the positive branch toward the observation while pushing the negative branch away.

C.1. Detailed Introduction of Benchmarks

The evaluation uses LIBERO and ManiSkill as complementary multitask benchmarks with different observation, action, reward, and generalization settings. ManiSkill specifically tests compositional OOD generalization across visual and task factors.

  • Evaluation setup: The evaluation covers 2 multitask benchmarks: LIBERO and ManiSkill.The benchmarks provide complementary settings for embodied VLA evaluation.
  • LIBERO: LIBERO reports average success rates across four suites—Spatial, Object, Goal, and Long—using 500 episodes per suite.Each suite contains 50 states and 10 sub-tasks, with RGB inputs, language instructions, proprioception, continuous end-effector actions, and sparse binary rewards.
  • ManiSkill: ManiSkill uses PutOnPlateInScene with 4,352 compositional tasks derived from 16 objects, 17 receptacles, and 16 tabletop scenes.The setting uses a third-person view, language instructions, joint poses, continuous joint-space actions, and a composite reward.
  • ManiSkill: Table 3 organizes ManiSkill OOD tasks across Vision, Semantics, and Execution categories.These categories define the benchmark’s task-mapping dimensions for OOD evaluation.

C.2. Hyperparameters for Training

Table 4 lists the hyperparameter settings used for Libero and ManiSkill training.

  • Table 4 provides hyperparameter settings for Libero and ManiSkill.

D. Additional Ablation: Step Selection Strategy

The ablation compares uniformly random solver-step sampling with fixed-step supervision. Random selection improves optimization stability and final success rate by covering multiple denoising stages.

  • The default Random Step strategy samples solver transitions uniformly across denoising stages, exposing training to different noise levels.Each iteration samples j ∼ U{0, . . . , K − 1} from a K-step Flow-SDE rollout.
  • The ablation holds the solver, objective, training budget, and environment settings constant across Random Step and Fixed Step variants.
  • Uniformly random step selection achieves more stable optimization and higher final success than fixed-step choices.The authors hypothesize that fixed-step supervision narrows the noise regime, whereas random selection covers multiple denoising stages.
  • Figure 5 compares performance between uniform random solver-step sampling and fixed-step selection strategies.
Loading 2603.02083v2…