PaperlayerView exact revision on arXiv

Source-linked AI summary

Online Evolution Strategy for Flow-Matching VLA Policies via Self-Supervised Trajectory Distribution Optimization

Gongxin Yao, Yongsheng Zhao, Jiayin Deng, Deng Liang, Han Gao, Lei Zhao, Baoping Cheng

arXiv:2609.38855v1cs.ROcs.AIcs.SYeess.SY

TL;DR

Flow Matching VLAs can generate different trajectories for the same task, but successful and failed behaviors may coexist in their learned distributions. Online-ES refines those distributions through trajectory-space exploration and execution feedback, achieving improvement comparable to reinforcement fine-tuning in simulation and real-world settings.

  • Problem

    Flow Matching VLAs can sample diverse trajectories, but successful and failed behaviors coexist, so a failure may reflect an unfavorable trajectory sample rather than an inability to solve the task.

    Source: §I · p. 1

  • Method

    Online-ES explores perturbed action trajectories using execution feedback, transfers the evolution direction into model parameters with a self-supervised objective, and incorporates failures as negative feedback.

    Source: §I · p. 2

  • Results

    Online-ES achieves policy improvement comparable to reinforcement fine-tuning in simulation and real-world environments without learning a value model or computing complex advantages.

    Source: §I · p. 2

  • Takeaways & Limitations

    The framework improves Flow Matching VLA performance through execution feedback without additional human demonstrations.

    Source: §VI · p. 8

Abstract

from arXiv · show

Vision-Language-Action (VLA) models based on generative frameworks, such as Flow Matching, have recently achieved impressive performance in robotic manipulation. Unlike deterministic policies, Flow Matching enables VLA models to learn conditional action trajectory distributions, where latent noise vectors induce different actions under the same task scenario. However, we observe that these distributions are often ill-formed, with successful and failed behaviors coexisting while considerable probability mass remains in unfavorable regions. To this end, we propose Online-ES, an online adaptation framework for Flow Matching VLAs based on Evolution Strategy (ES), which refines the learned action trajectory distribution through interaction feedback. Instead of pruning the latent noise space, our method performs evolutionary exploration directly in the action trajectory space, where diverse trajectories generated by Flow Matching provide candidate solutions for adaptation. By perturbing sampled trajectories and evaluating their execution outcomes, we derive a self-supervised MSE objective that transfers the evolution direction from trajectory space into model parameter space. Mathematically, we prove that the proposed objective provides an unbiased estimator of the optimal evolution direction. Moreover, we also incorporate failure experiences as negative feedback to regularize the evolution direction, steering the policy away from previously explored failure regions. Experiments in both simulation and real-world environments demonstrate that Online-ES achieves policy improvement comparable to reinforcement fine-tuning, without learning a value model or computing advantages.

I. INTRODUCTION

Deterministic action experts can collapse multimodal manipulation solutions into averaged behaviors, while Flow Matching policies can generate diverse trajectories in the same scenario. Online-ES uses execution feedback to refine those trajectory distributions through action-space exploration and self-supervised parameter updates.

I. INTRODUCTION

  • Deterministic action experts can collapse multiple feasible manipulation trajectories into averaged behaviors.Manipulation tasks may admit several trajectories that achieve the same objective.Source: §I · p. 1
  • Flow Matching can produce both successful and failed trajectories under the same task, with failure sometimes reflecting an unfavorable noise sample rather than inability to solve it.Different latent noise vectors generate diverse trajectories for the same scenario.Source: §I · p. 1
  • Noise-space pruning is difficult to generalize because chunk effects are hard to disentangle and unfavorable regions vary across tasks.Long-horizon action chunks depend on observations produced by preceding chunks.Source: §I · pp. 1–2
  • Online-ES explores perturbed candidate trajectories and uses a self-supervised MSE objective to transfer the evolution direction into model parameters.The method estimates the trajectory distribution with sampled trajectories and evaluates perturbations through environment interaction.Source: §I · p. 2
  • Failed experiences provide negative feedback that steers evolution away from previously explored failure regions.The paper proves the proposed objective is an unbiased estimator of the optimal evolution direction.Source: §I · p. 2

A. Flow Matching for Generative Modeling

Flow Matching transports noise samples through a learned vector field, while VLA policies combine vision-language representations with action generation and may use stochasticity to represent alternative trajectories. The section connects this formulation to Evolution Strategies and explains how noise-conditioned action chunks form long-horizon trajectories whose outcomes motivate online refinement.

A. Flow Matching for Generative Modeling

  • A time-dependent vector field transports noise along an ODE, with different initializations producing distinct paths and samples.The paper contrasts this continuous transport with diffusion models' multi-step reverse denoising.Source: A. Flow Matching for Generative Modeling · p. 2

B. Vision-Language-Action Models

  • VLA policies integrate pretrained vision-language backbones with action experts, and recent work applies reinforcement learning to improve them beyond supervised fine-tuning.The passage contrasts those approaches with exploiting Flow Matching trajectory diversity for evolutionary exploration.Source: B. Vision-Language-Action Models · p. 2

C. Evolution Strategy

  • Evolution Strategy uses stochastic perturbations to estimate optimization directions for expected fitness without explicit objective derivatives.The paper extends this approach to action trajectory space for online adaptation.Source: C. Evolution Strategy · p. 2

A. Problem Formulation

  • The policy integrates an observation- and language-conditioned velocity field from a latent-noise initial state to generate action chunks that compose long-horizon trajectories.Different noise vectors in the same scenario can yield both successful and failed behaviors, motivating online refinement of the trajectory distribution.Source: A. Problem Formulation · pp. 2–3

B. Evolution Strategy Fundamentals

Evolution Strategy searches around a current solution with perturbed candidates, scores them by fitness, and aggregates perturbations to estimate an update direction.

B. Evolution Strategy Fundamentals

  • ES generates perturbed candidates around the current solution to explore its local neighborhood.The method uses random perturbations and fitness-based selection.Source: B. Evolution Strategy Fundamentals · p. 3
  • Candidate fitness weights the perturbations used to estimate the optimization direction for a maximization objective.Higher objective values indicate better solutions.Source: B. Evolution Strategy Fundamentals · p. 3
  • The solution is updated with step size α, and ES avoids explicit objective derivatives.The approach has also been extended to deep neural networks as an alternative to reinforcement learning.Source: B. Evolution Strategy Fundamentals · p. 3

IV. METHODOLOGY

Online adaptation refines the trajectory distribution using execution feedback, exploring locally in trajectory space rather than perturbing the VLA’s parameters.

IV. METHODOLOGY

  • The adaptation goal is to refine the action trajectory distribution using execution feedback, typically a binary success indicator.Maximizing the stated objective is equivalent to maximizing average success rate.Source: A. Extending Evolution Strategy to… · p. 3
  • Parameter-space ES is inefficient for large VLA models and small parameter perturbations may produce infeasible or unsafe trajectories.The method therefore explores directly in action trajectory space.Source: A. Extending Evolution Strategy to… · p. 3
  • Online-ES perturbs trajectories with Gaussian noise, with σ controlling the perturbation magnitude.This local exploration is intended to preserve pretrained behavior while transferring beneficial explorations to parameter space.Source: A. Extending Evolution Strategy to… · p. 3

B. Evolution Direction of Action Trajectory Distribution

The method estimates a distribution-level evolution direction by sampling trajectories, exploring their local neighborhoods, and weighting perturbations by execution feedback.

B. Evolution Direction of Action Trajectory Distribution

  • Multiple latent noise vectors generate trajectories whose set provides a Monte Carlo approximation of the action trajectory distribution.Each sampled trajectory is explored locally with Gaussian perturbations of magnitude σ.Source: B. Evolution Direction of Action… · p. 4
  • The evolution objective maximizes expected execution feedback under stochastic trajectory perturbations.The trajectory gradient gives the steepest ascent direction.Source: B. Evolution Direction of Action… · p. 4
  • With Gaussian perturbations, the derivative can be expressed using feedback-weighted perturbations, with the expectation taken over perturbation noise.The resulting feedback-weighted perturbations provide an unbiased estimate of each trajectory’s evolution direction.Source: B. Evolution Direction of Action… · p. 4

C. Self-Supervision for Parameter Adaptation

A self-supervised MSE objective transfers the trajectory-space evolution direction into parameter space through the model’s trajectory Jacobian.

C. Self-Supervision for Parameter Adaptation

  • The MSE loss uses sampled trajectories and perturbed variants, weighted by their execution feedback, to transfer the evolution direction into parameter space.The perturbed trajectories are detached so gradients back-propagate only through the original trajectories.Source: C. Self-Supervision for Parameter… · p. 4
  • The parameter gradient projects the evolution direction through the model Jacobian, and gradient descent cancels the loss-gradient sign.The same projection applies to the overall direction estimated across trajectory samples.Source: C. Self-Supervision for Parameter… · p. 4

D. Failure-Aware Regularization

Online-ES uses failure experiences as negative feedback to regularize the evolution direction. Successful and failed trajectories jointly attract the distribution toward high-feedback regions and repel it from explored failure regions.

D. Failure-Aware Regularization

V. EXPERIMENT

The experiments use a frozen-feature GR00T N1.5 baseline and update its action expert online. The implementation supports single-task and parallel multi-task adaptation, with trajectory smoothing for real-world execution.

A. Implementation Details

  • For multi-task adaptation, GPUs collect trajectories for separate tasks and synchronize gradients before policy updates.Real-world trajectories are smoothed with quintic polynomial fitting or spline interpolation before execution.Source: A. Implementation Details · p. 5
  • Online adaptation updates only the GR00T N1.5 action expert while freezing the vision, depth, and language modules.The model includes a DiT-based action expert conditioned on features from the other modules.Source: A. Implementation Details · p. 5
  • Simulation adaptation uses 8 H100 GPUs, while the single-task real-world setting uses one H100 GPU on an AgiBot G1 robot.The robot has dual arms with 14-DoF joints and parallel-jaw grippers.Source: A. Implementation Details · p. 5

B. Simulation Results

Online-ES improves performance in both single-task and multi-task LIBERO adaptation settings. In multi-task adaptation, it achieves competitive improvement relative to reinforcement fine-tuning despite not attaining the highest success rate.

B. Simulation Results

  • The simulation study fine-tunes the baseline on the official human-demonstration dataset for 30,000 steps before online adaptation.LIBERO contains four suites with 10 tasks per suite.Source: B. Simulation Results · p. 5
  • On the seventh LIBERO Spatial task with fixed scene initialization, Online-ES raises average success to nearly 100% after 300 adaptation steps.The perturbation magnitude σ is 0.03.Source: B. Simulation Results · p. 5
  • With random scene initialization, success improves across tasks but more slowly than in the fixed-scene setting and remains below 100%.The setting uses 50 object placements; Fig. 5 shows curves for one representative task from each of the four suites.Source: B. Simulation Results · pp. 5–6, Figure 5 · p. 6
  • Online-ES achieves competitive policy improvement in multi-task LIBERO adaptation, though it does not attain the highest success rate.The comparison includes VLA-RL based on OpenVLA and πRL based on π0.5.Source: B. Simulation Results · p. 6

C. Real-World Results

In the real-world pink-bowl grasping task, online adaptation reduces both failure and invalid-contact outcomes. The experiments smooth perturbed trajectories before execution and evaluate outcomes over repeated rollouts.

C. Real-World Results

  • The evaluation distinguishes a gripper miss, an improper rim contact, and a proper grasp with the bowl rim between the fingers.Invalid Contact is considered closer to successful grasping than Failure.Source: C. Real-World Results · p. 6
  • The real-world benchmark uses the pink-bowl grasping subtask, with 300 teleoperation demonstrations for offline training.The baseline is trained for 30,000 steps before adaptation.Source: C. Real-World Results · p. 6
  • High-order polynomial refitting converts independent action-level noise into a smoother, longer-horizon trajectory perturbation for execution.The method is used because high-frequency inputs can induce severe joint oscillations.Source: C. Real-World Results · p. 6
  • After 150 online adaptation steps, both Failure and Invalid Contact cases show clear decreases across evaluations of 100 rollouts per row.The rollout buffer maintains a successful-to-failed trajectory ratio of 8:2.Source: C. Real-World Results · pp. 6–7

D. Ablation Studies

The ablations report faster success-rate convergence with failure-based regularization, a benefit from broad early exploration as perturbations anneal, and smoother trends with larger trajectory buffers. The conclusion reports improvements in simulation and real-world manipulation without additional demonstrations, while identifying more efficient use of interaction experience as future work.

D. Ablation Studies

  • Table III reports real-world online-adaptation quantitative results evaluated over 100 independent trials.Source: Table III · p. 7
  • Increasing the action trajectory buffer size from |Ω| = 1 to 5 or 10 yields a smoother, more stable upward success-rate trend.This ablation was conducted on the 7th task of LIBERO Spatial.Source: Figure 9 · p. 7, D. Ablation Studies · p. 7
  • With σ annealed from 0.3 to 0.03, success rates initially lag the fixed σ = 0.03 setting, then catch up and surpass it.The annealed schedule decays over 2,500 adaptation steps; the authors suggest broad exploration can capture information missed by a narrow range.Source: Figure 10 · p. 7, D. Ablation Studies · p. 7
  • Failure-based regularization accelerates the convergence of success rate.The comparison is between adaptation using successful experiences only and adaptation using mixed successful and failed experiences.Source: D. Ablation Studies · pp. 7–8, Figure 11 · p. 8

VI. CONCLUSION

  • Online-ES improves Flow Matching VLA performance in simulation and real-world manipulation without additional human demonstrations.Future work will focus on improving the efficiency of utilizing interaction experiences for online adaptation.Source: §VI · p. 8

Related papers

Loading 2609.38855v1…