PaperlayerView exact revision on arXiv

Source-linked AI summary

Flow Policy Gradients for Robot Control

Brent Yi, Hongsuk Choi, Himanshu Gaurav Singh, Xiaoyu Huang, Takara E. Truong, Carmelo Sferrazza, Yi Ma, Rocky Duan, Pieter Abbeel, Guanya Shi, Karen Liu, Angjoo Kanazawa

arXiv:2602.02481v1cs.ROcs.AI

TL;DR

Robot policy gradients typically rely on differentiable action likelihoods, limiting policy representations and making flow-policy likelihoods expensive to compute. The paper introduces FPO++, a likelihood-free flow policy-gradient objective with per-sample ratios and an asymmetric trust region, and evaluates it across locomotion, humanoid, and manipulation settings. FPO++ enables stable training, sim-to-real transfer, and fine-tuning while revealing exploration and robustness benefits of flow policies.

  • Problem

    Likelihood-based policy gradients constrain robot policies to simple distributions, while computing likelihoods for flow policies can be expensive.

  • Method

    FPO++ improves flow policy gradients with per-sample ratio clipping and an asymmetric trust region, while using zero-initialized sampling at test time.

  • Results

    FPO++ enables stable flow-policy learning across locomotion, humanoid motion tracking, manipulation fine-tuning, and sim-to-real tasks.

  • Takeaways & Limitations

    The results show that policy-gradient training and sim-to-real transfer can work for flow policies without explicit action likelihoods.

Abstract

from arXiv · show

Likelihood-based policy gradient methods are the dominant approach for training robot control policies from rewards. These methods rely on differentiable action likelihoods, which constrain policy outputs to simple distributions like Gaussians. In this work, we show how flow matching policy gradients -- a recent framework that bypasses likelihood computation -- can be made effective for training and fine-tuning more expressive policies in challenging robot control settings. We introduce an improved objective that enables success in legged locomotion, humanoid motion tracking, and manipulation tasks, as well as robust sim-to-real transfer on two humanoid robots. We then present ablations and analysis on training dynamics. Results show how policies can exploit the flow representation for exploration when training from scratch, as well as improved fine-tuning robustness over baselines.

I. INTRODUCTION

Likelihood-based policy gradients have driven robot-control progress but constrain policies through action-likelihood computation, which is costly for expressive flow representations. This work studies likelihood-free flow policy gradients and introduces improvements for challenging robotics settings.

  • I. INTRODUCTION: Flow-based policy gradients target more expressive action distributions than the diagonal Gaussians commonly used in online robotic reinforcement learning.Likelihood computation for flow policies can require expensive sampling or integration to account for volume changes.
  • I. INTRODUCTION: FPO++ introduces per-sample ratio clipping and an asymmetric trust region to make flow policy gradients more robust in challenging robotics settings.The framework bypasses explicit likelihood computation while supporting expressive flow-based policies.
  • I. INTRODUCTION: The method is evaluated across legged locomotion, humanoid motion tracking, and single-arm and bimanual manipulation, including training from scratch, sim-to-real transfer, and demonstration-pretrained fine-tuning.The experiments also examine algorithmic choices and test-time sampling strategies.
  • I. INTRODUCTION: FPO++ improves quadruped locomotion gaits over Gaussian PPO baselines with the same rewards and is more robust to base-policy choice during fine-tuning.These findings come from the paper’s analysis of training dynamics.
  • I. INTRODUCTION: PPO updates stochastic policies using sampled actions, likelihood ratios, and advantage estimates, requiring differentiability from action likelihoods rather than rewards or environment dynamics.This likelihood-based formulation is popular for its simplicity and empirical performance.

III. IMPROVED FLOW POLICY OPTIMIZATION

FPO enables PPO-style policy-gradient training for flow policies without explicit likelihoods by using conditional flow-matching losses. Its loss differences approximate action log-likelihood differences and shift probability flow toward higher-reward actions.

  • III. IMPROVED FLOW POLICY OPTIMIZATION: FPO trains flow-parameterized policies without explicit likelihoods, avoiding computationally prohibitive divergence integration in reinforcement learning.The approach uses conditional flow matching to support policy-gradient-style training.
  • III. IMPROVED FLOW POLICY OPTIMIZATION: FPO supports more complex action distributions than the diagonal Gaussians commonly used in online robotic reinforcement learning.This formulation preserves a PPO-style clipped objective while changing how policy ratios are estimated.
  • III. IMPROVED FLOW POLICY OPTIMIZATION: FPO approximates action log-likelihood differences with conditional flow-matching loss differences, then uses advantages to shift probability flow toward higher-reward actions.The losses are estimated from sampled noise and flow-step pairs, noised actions, and squared velocity-prediction errors.
  • III. IMPROVED FLOW POLICY OPTIMIZATION: Conditional flow matching estimates policy losses by sampling Gaussian noise and flow steps, interpolating actions, and averaging squared errors in predicted velocities.These losses feed the FPO ratio used for policy updates.

B. FPO++

FPO++ refines FPO with per-sample ratios and an asymmetric trust region. These changes provide finer-grained clipping and constrain updates that would aggressively increase conditional flow-matching losses.

  • B. FPO++: FPO++ introduces per-sample ratios and an asymmetric trust region because standard FPO required refinements for reliable performance on difficult tasks.The two changes target the objective’s trust-region behavior during policy updates.
  • B. FPO++: Per-sample ratios clip each noise-and-flow-step sample independently, providing a finer-grained trust region during multiple gradient steps.Standard FPO clips only after averaging samples, so all or none of an action’s samples are clipped.
  • B. FPO++: ASPO uses PPO clipping for positive-advantage actions and the more constrained SPO objective for negative-advantage actions.This asymmetry matches the directions in which the update decreases or increases conditional flow-matching loss.
  • B. FPO++: SPO supplies a gradient that pulls excessive ratios back instead of zeroing gradients beyond the trust region.For negative advantages, this discourages large conditional flow-matching-loss increases.
  • B. FPO++: Applying SPO to negative advantages helps preserve entropy and stabilize the variational gap by penalizing aggressive likelihood decreases and posterior-divergence increases.These effects are stated through the interpretation of conditional flow-matching loss as a variational bound.

C. FPO++ Objective

FPO++ combines per-sample ratio clipping with an asymmetric trust region and uses stochastic flow exploration during training but zero-sampling for evaluation. Across IsaacLab locomotion tasks, it remains stable and maintains high returns.

  • FPO++ Objective: FPO++ combines an ASPO trust region with per-sample ratio clipping in its objective.The objective is defined for Monte Carlo pairs and uses the advantage sign to specify the ASPO trust region.
  • Zero-Sampling: FPO++ explores with Gaussian initial noise during training and uses zero-sampling at test time and for evaluation.Training integrates from ϵ ∼N(0, I), whereas evaluation initializes integration from ϵ =⃗0.
  • Locomotion Benchmarks: The locomotion experiments used four robots, with velocity-conditioned policies trained in parallel IsaacLab environments.The robots were Unitree Go2, Boston Dynamics Spot, Unitree H1, and Unitree G1.
  • Locomotion Benchmarks: FPO++ was stable across all locomotion tasks and achieved and maintained high episode returns.Standard FPO was more prone to local minima and catastrophic failures in these environments despite hyperparameter tuning.

B. Humanoid Sim-to-real

The paper evaluates flow policies across humanoid sim-to-real transfer and manipulation fine-tuning, including direct deployment with reduced sampling steps. FPO++ supports robust physical-robot behavior and consistently strong fine-tuning performance, while its components improve locomotion returns.

  • Humanoid Sim-to-real: FPO++ learned robust gaits on a Booster T1 and dynamic motion sequences on a Unitree G1 physical robot.The authors describe this as the first humanoid sim-to-real demonstration using a flow policy without expert distillation or explicit likelihoods.
  • Manipulation Fine-tuning: FPO++ consistently achieved high manipulation success rates and converged faster than the fine-tuning baselines.Vanilla FPO also performed well on simpler tasks, whereas both DPPO variants underperformed FPO-based methods.
  • Manipulation Fine-tuning: FPO++ fine-tuning was more robust to the initial base policy performance than the DPPO runs.DPPO often failed when the base policy had low stochastic-sampling success, including a Can task starting near 10% success.
  • Objective Ablations: Per-sample ratios produced higher and more consistent locomotion returns across training runs and environments.The comparison used final training and evaluation returns across clipping parameters and random seeds.
  • Objective Ablations: ASPO produced higher and more consistent locomotion returns than standard PPO clipping and SPO across runs.The ablation varied clipping parameters and random seeds for each trust-region implementation.

D. Training Ablations

The ablations show that FPO++’s per-sample ratio and ASPO trust region stabilize locomotion training, preserve entropy, and reduce gradient variance, while zero-sampling improves evaluation behavior and latency.

  • Component ablations: FPO++ achieves consistent locomotion performance across hyperparameters and seeds, whereas disabling either component significantly lowers returns and increases variance across embodiments.The ablation evaluates the per-sample ratio and ASPO trust region separately across locomotion robots.
  • Entropy: ASPO preserves entropy during H1 locomotion training and avoids the entropy collapse observed with PPO trust-region clipping.The authors connect this entropy preservation to improved robustness in the ASPO ablation.
  • Gradient variance: FPO++ reduces empirical gradient variance, with finer-grained clipping and ASPO producing more similar per-sample gradients within policy updates.The gradient analysis uses cosine similarity between individual gradients and the update-average gradient.
  • Fine-tuning limitation: ASPO sometimes degrades manipulation fine-tuning, where entropy preservation is less important and well-initialized flow policies reduce the need to constrain the variational gap.The authors identify this as a task-dependent limitation of the trust-region change.
  • Zero-sampling: Zero-sampling creates large train–evaluation gaps in locomotion and can raise manipulation success rates from around 10% to over 70% before fine-tuning.The cited examples compare stochastic sampling with zero-sampling at evaluation.
  • Zero-sampling: Zero-sampling also reduces Euler integration steps with negligible performance loss, enabling lower-latency action sampling on the robot’s onboard computer.This result is reported alongside the manipulation and locomotion evaluation comparisons.

F. Comparison with Gaussian PPO

Compared with Gaussian PPO, FPO++ learns higher-return locomotion policies with lower seed variance and more expressive, correlated action distributions that produce different gaits. Its broader practical scope includes flow-policy training, sim-to-real transfer, and fine-tuning, but it requires more wall-clock time in some settings.

  • Locomotion comparison: FPO++ locomotion policies almost always converge to higher returns than the compared Gaussian PPO configurations, with reduced variance across random seeds.The comparison spans multiple environment counts and identifies robustness to small batches and increased parallelism in selected robots.
  • Gait behavior: FPO++ produces more consistent trot gaits than Gaussian PPO under the same Spot rewards, which often yields more symmetric pronk gaits.The gait differences are reported as reproducible across seeds, clipping parameters, and learning rates.
  • Action distributions: FPO++ develops action-dimension coupling, including negative correlations between left and right hip joints, unlike diagonal-covariance Gaussian PPO implementations.The correlations are visualized during training and are consistent with alternating gaits such as trotting.
  • Limitations: FPO++ generally takes longer to run than Gaussian PPO, including 23 versus 19 minutes for G1 locomotion to reach evaluation return 25.Motion-tracking experiments can take as much as 3x longer than a tuned Gaussian PPO baseline.
  • Scope and significance: FPO++ provides a unified flow-policy approach that succeeds in online policy-gradient training, real-world continuous control, sim-to-real transfer, and fine-tuning.The paper frames these results as evidence that explicit likelihoods are not required for policy-gradient robot control.

Appendix of “Flow Policy Gradients for Robot Control”

The appendix documents related flow-policy learning work, experimental environments and rewards, robot coverage, and analyses showing that FPO++ improves gradient alignment while supporting diverse locomotion and manipulation settings.

  • Related work: The appendix distinguishes online policy-gradient training from offline flow and diffusion RL approaches based on AWR, Q-learning, or related objectives.It identifies numerical instability from backpropagating through multi-step denoising as a challenge for Q-learning with generative policies.
  • Training analysis: FPO++’s per-sample ratio and ASPO trust region are motivated as mechanisms that improve gradient alignment and stability during policy updates.The analysis uses cosine similarity between individual gradients and the average gradient within each update.
  • Experimental coverage: The experiments cover quadruped and humanoid locomotion, motion tracking, and manipulation across diverse robot embodiments and control regimes.The appendix figures identify simulated Go2, Spot, H1, and G1 robots, with physical deployment on G1 and Booster T1.
  • Locomotion rewards: Locomotion rewards combine velocity tracking, air time, stability and smoothness penalties, with additional gait-related shaping for Spot.Default quadruped environments use less shaping to allow gaits to emerge, while Spot includes synchronization and foot-clearance terms.

C.2 Motion Tracking

The motion-tracking experiments use a 29-DoF G1 robot with 50 Hz control and BeyondMimic-based task design. Domain randomization perturbs physics, initialization, external forces, and actuator latency to support sim-to-real transfer.

  • Control setup: Motion tracking uses the 29-DoF Unitree G1 at 50 Hz, with a 0.005 s simulation timestep and four-step control decimation.Reward design, observations, and termination conditions follow BeyondMimic.
  • Domain randomization: Domain randomization varies friction, restitution, joint defaults, center of mass, external forces, and initial motion states.These perturbations are applied to improve transfer robustness rather than to define the nominal tracking task.
  • Hyperparameters: The appendix reports separate FPO++ hyperparameter tables for locomotion and motion tracking experiments.These tables distinguish the final experimental values from the swept values.
  • Domain randomization: Actuator command delays of 0–10 ms are simulated at episode reset to improve robustness to control latency during sim-to-real transfer.The delay spans zero to two simulation steps at the stated timestep.

2) Training Hyperparameters:

The training setup uses shared actor–critic architectures and carefully tuned flow-loss stabilization procedures. For motion tracking, FPO++ remains stable while vanilla FPO becomes numerically unstable and collapses.

  • Training setup: Training uses 4096 parallel environments, 96-step rollouts, and 3-layer MLP actor and critic networks with hidden sizes (1024, 512, 256).
  • Flow-loss stabilization: FPO++ stabilizes ratio computation by clamping CFM losses before differencing, addressing numerical risks from exponentiated squared-loss differences.
  • Manipulation hyperparameters: Manipulation experiments use actor and critic learning rates of 1 × 10−5 and 1 × 10−4, with GAE λ = 0.99 and task-dependent discount factors.
  • Baseline comparison setup: Baseline comparisons adapt DPPO implementations and tune hyperparameters so methods share comparable initialization and evaluation conditions.
  • Motion-tracking training stability: FPO++ maintains stable value and policy losses, improves monotonically, and converges to the high-return policy used for real-world deployment.Vanilla FPO initially learns but then collapses, with deteriorating reward and episode length alongside large value- and surrogate-loss spikes.

D.2 Comparison with Gaussian PPO for motion tracking

FPO++ enables stable sim-to-real motion tracking but remains slightly behind highly tuned Gaussian PPO in simulated total return. Comparisons indicate that regularization choices materially affect this gap.

  • Gaussian PPO comparison: FPO++ achieves stable sim-to-real transfer for complex high-DoF motion tracking, but Gaussian PPO converges to higher simulated total returns.FPO++ retains a slight episode-length advantage while overall return remains lower after regularization features are included.
  • Regularization analysis: FPO++ outperforms a Gaussian PPO baseline that omits both entropy regularization and KL-adaptive learning rates in return and stability.
  • Comparison with vanilla FPO: Motion-tracking training curves show vanilla FPO failing, whereas FPO++ trains stably and supports deployment on the real G1 robot.
  • Regularization analysis: Adding entropy regularization and adaptive learning rates gives Gaussian PPO higher peak returns, while FPO++ maintains a slight episode-length advantage.
  • Manipulation comparison: FPO++ attains the highest final success rates on both RoboMimic Can and Square in the adapted flow-based fine-tuning comparison.

D.4 Detailed analysis on Robomimic can experiment

Robomimic fine-tuning performance depends strongly on the base policy’s exploration-time success rate. FPO-based methods remain robust in high-variance settings, while ASPO’s entropy preservation helps locomotion more than manipulation fine-tuning.

  • Base-policy quality: FPO++ and vanilla FPO are relatively robust to high-variance fine-tuning, whereas DPPO variants degrade substantially under the same base-policy conditions.
  • Base-policy quality: DPPO variants degrade under low exploration-time success but become competitive when the base policy reaches 64.06% random-sampling success.Low random-sampling success yields few successful trajectories, sparse terminal rewards, and high gradient variance.
  • Manipulation ablations: The per-sample ratio benefits manipulation fine-tuning, but the ASPO trust region is detrimental on the RoboMimic Square and DexMimicGen Threading tasks.ASPO’s entropy preservation is useful for discovering new locomotion behaviors but can introduce undesirable behaviors when behavior cloning already provides a good initialization.
  • Base-policy quality: Base-policy quality differs sharply between zero-sampling and random-sampling evaluation, with higher-quality policies reaching 96.11% and 64.36%, respectively.
  • Flow-field analysis: PPO-clipped FPO++ narrows its action distribution as performance collapses, whereas ASPO preserves a wider, more exploratory distribution at comparable or converged training stages.
Loading 2602.02481v1…