PaperlayerView exact revision on arXiv

Source-linked AI summary

RFPO: Rectified Flow Policy Optimization for Embodied Control

Ting Huang, Lisiyu Pan, Haoyu Wang, Zeyu Zhang, Siyuan Qian, Yanjun Li, Yandong Guo, Boxin Shi, Hao Tang

arXiv:2610.10453v1cs.RO

TL;DR

Reward-optimized flow policies can lose control reliability when their numerical integration budget is reduced. RFPO combines reward-aware online Reflow with multi-budget action supervision, and one-step execution remains close to 64-step performance across four embodiments.

  • Problem

    Reward-optimized flow policies are not explicitly constrained to remain reliable under coarse integration, so reducing Euler steps can substantially change actions and degrade control.

    Source: §I · pp. 1–2

  • Method

    RFPO rectifies student-induced transport paths through reward-aware online Reflow and uses a frozen Gaussian PPO controller for complementary action-space supervision at full and intermediate budgets.

    Source: §I · pp. 1–2

  • Results

    Across four embodiments, one-step returns remain within 2.4% of corresponding 64-step values; on Go2, one-step retains 98.5% of 64-step reward.

    Source: §I · p. 2

  • Takeaways & Limitations

    Real-robot experiments demonstrate stable one-step locomotion across forward, lateral, and yaw commands.

    Source: §I · p. 2

    Real-robot validation is limited to Go2 locomotion and a small command set; training also adds full-budget rollouts and a separately trained PPO teacher.

    Source: §V · pp. 7–8

Abstract

from arXiv · show

Flow-based policies provide an expressive framework for continuous robot control, but their iterative ODE integration incurs substantial inference cost. Naively reducing the integration budget can severely degrade control, since policies optimized under full-step execution are not explicitly constrained to remain reliable under coarse numerical integration. We refer to this mismatch as the few-step discretization gap. To address this problem, we introduce RFPO, a flow-policy optimization framework for reliable few-step execution. Reward-aware online Reflow rectifies student-induced transport paths during on-policy learning, making the resulting policy more robust to coarse integration. A frozen Gaussian PPO controller supplies complementary action-space supervision at full and intermediate integration budgets, while the deployed policy remains a single flow student executed with one Euler step. Across Unitree Go2, Boston Dynamics Spot, Unitree H1, and Unitree G1, RFPO consistently preserves full-step control performance under one-step execution, with one-step returns remaining within 2.4% of their corresponding 64-step values across both zero and random initialization. On Unitree Go2, one-step execution retains 98.5% of the 64-step reward while reducing onboard mean inference latency from 4.39 ms to 0.08 ms, yielding a 54.9x speedup. Real-robot experiments further validate stable one-step locomotion. Code: https://github.com/AIGeeksGroup/RFPO. Website: https://aigeeksgroup.github.io/RFPO.

I. INTRODUCTION

RFPO addresses the few-step discretization gap that can make reward-optimized flow policies unreliable under coarse integration. It combines reward-aware online Reflow with multi-budget action supervision and achieves robust one-step control across four embodiments.

I. INTRODUCTION

  • Reward-driven generative-policy optimization includes flow-policy methods such as FPO and FPO++ [32].Prior flow-policy RL work makes reward optimization increasingly practical, while RFPO focuses on preserving behavior under reduced integration budgets.Source: §I · p. 1
  • Across four embodiments and both zero and random initialization, one-step returns remain within 2.4% of corresponding 64-step values; on Go2, one-step retains 98.5% of 64-step reward.Go2 onboard mean inference latency falls from 4.39 ms to 0.08 ms, a 54.9× speedup; real-robot experiments demonstrate stable one-step locomotion.Source: §I · p. 2
  • The few-step discretization gap arises because reward-optimized flow policies can degrade under aggressive step reduction despite strong full-step performance.Existing objectives do not explicitly constrain the vector field to remain reliable under coarse numerical integration.Source: §I · pp. 1–2
  • RFPO combines student-induced online Reflow during on-policy learning with multi-budget action-space supervision from a frozen Gaussian PPO controller.Training-only components are removed at deployment, leaving one flow student executed with one Euler step.Source: §I · pp. 1–2

II. RELATED WORK

Related work establishes generative flow policies and reward optimization as relevant to robot control, while distinguishing RFPO's focus on coarse integration during reward optimization. The formulation defines the discretization gap and presents training signals that support one-step deployment.

II. RELATED WORK

  • Prior flow-policy RL methods primarily optimize return, while many few-step acceleration approaches target generative modeling, imitation learning, or post-hoc acceleration.RFPO instead performs student-induced Reflow during on-policy reward optimization and adds action-space supervision at full and intermediate budgets.Source: §II · p. 2

A. Overview and Problem Formulation

  • A conditional flow policy maps robot observations to continuous actions, and inference numerically solves its learned velocity field from t = 1 to t = 0.The underlying convention treats t = 1 as the noise endpoint and t = 0 as the action endpoint.Source: A. Overview and Problem Formulation · p. 2
  • The few-step discretization gap is the mismatch in which a small Euler-step budget produces an action that can deviate substantially from the same policy's full-budget action.The original FPO++ [32] on-policy actor–critic objective optimizes performance at the training integration budget without explicitly shaping the vector field for coarse execution.Source: A. Overview and Problem Formulation · p. 3
  • RFPO uses reward-aware online Reflow, multi-budget action distillation, zero-start sampling, and adaptive compute regularization during training.At deployment, auxiliary components are removed and one flow student runs with one Euler step.Source: A. Overview and Problem Formulation · p. 3, Figure 2 · p. 3

B. Reward-Aware Online Reflow

Reward-aware online Reflow rectifies transport paths induced by the current student during on-policy training. Advantage estimates focus that rectification on behavior useful for control.

B. Reward-Aware Online Reflow

  • Reflow uses trajectories induced by the current student to address the effect of flow geometry on coarse-solver error.The coupling is induced by the evolving student rather than the external PPO teacher.Source: B. Reward-Aware Online Reflow · p. 3
  • The auxiliary Reflow and distillation losses use a mixture of initializations, while on-policy rollouts retain Gaussian initialization.The mixture includes zero-start samples and is restricted to those auxiliary losses.Source: B. Reward-Aware Online Reflow · p. 3
  • On-policy advantages focus rectification on behavior useful for control.The resulting objective aligns the evolving student with straighter transport couplings while retaining the reward-driven update.Source: B. Reward-Aware Online Reflow · p. 4
  • When all minibatch advantages are non-positive, the Reflow weighting uses uniform weights.For positive-advantage samples, normalized weights are used.Source: B. Reward-Aware Online Reflow · p. 4

C. Multi-Budget Action Distillation

Multi-budget action distillation uses a frozen Gaussian PPO controller as an independent action-space reference. The student is trained against it at full and intermediate budgets, but not at one step.

C. Multi-Budget Action Distillation

  • A frozen Gaussian PPO controller supplies action-space targets independently of the Reflow couplings.The controller defines action targets but does not construct the Reflow couplings.Source: C. Multi-Budget Action Distillation · p. 4
  • The same flow student is trained toward the PPO reference at full and intermediate integration budgets.One-step execution is not included in the distillation budgets, so it receives no direct one-step teacher supervision.Source: C. Multi-Budget Action Distillation · p. 4
  • Online Reflow shapes transport geometry, while action distillation preserves an action-space reference across budgets.The objectives constrain different aspects of the policy.Source: C. Multi-Budget Action Distillation · p. 4

D. Auxiliary Adaptive Compute Regularization

Auxiliary adaptive compute regularization exposes the student to heterogeneous solver budgets during training. Its budget predictor is not required at deployment.

D. Auxiliary Adaptive Compute Regularization

  • A lightweight predictor defines a categorical distribution over solver budgets for heterogeneous-budget training.The fidelity branch uses the highest-probability budget.Source: D. Auxiliary Adaptive Compute… · p. 4
  • The fidelity reference is the detached full-budget action under zero initialization.The passage introduces this reference before defining the corresponding regularizer.Source: D. Auxiliary Adaptive Compute… · p. 4
  • The auxiliary regularizer broadens training budgets and encourages low-cost execution while maintaining consistency with the full-budget student.The budget predictor is used only during optimization, not deployment.Source: D. Auxiliary Adaptive Compute… · p. 4

E. Joint Optimization and Deployment

Joint optimization combines the original on-policy objective with auxiliary flow and action regularizers. Deployment removes training-only components and uses the learned student with one Euler step.

E. Joint Optimization and Deployment

  • The final training objective combines the original on-policy objective with auxiliary flow and action regularizers.Training follows FPO++ rollout and advantage estimation; student-generated Reflow endpoints and PPO action targets are detached from their target branches.Source: E. Joint Optimization and Deployment · p. 4
  • At deployment, the learned student starts from zero and executes a single Euler step, without the PPO teacher or budget predictor.The deployed controller consists only of the learned student policy.Source: E. Joint Optimization and Deployment · p. 4
  • The cross-embodiment evaluation reports average episodic reward under 64-step and one-step Euler execution across two quadruped and two humanoid embodiments.Reward scales differ across tasks, so comparisons should be made within each embodiment.Source: Table I · p. 5

A. Experimental Setup

The experiments evaluate velocity control on two quadrupeds and two humanoids, with embodiment-specific policy inputs, rewards, and control interfaces. The Go2 hardware study uses the official velocity task and replays recorded commands in simulation for comparison.

Environment and task

  • Velocity-control tasks cover Unitree Go2, Boston Dynamics Spot, Unitree H1, and Unitree G1 (29 DoF), with embodiment-specific observations, rewards, and low-level interfaces.The PPO teacher and flow-policy variants share each embodiment’s task definition and domain-randomization protocol.Source: A. Experimental Setup · p. 5
  • The real-robot study uses the official Go2 velocity task, with 45-D proprioception and continuous joint-control targets through the low-level PD interface.The observation excludes base linear velocity.Source: A. Experimental Setup · p. 5

Integration and evaluation protocol

  • Evaluation compares 64-step and one-step Euler execution across embodiments, with additional Go2 sweeps at 32, 16, 8, 4, and 1 steps under zero and random initialization.Each setting uses 256 parallel environments and one episode per environment; flow-policy results use EMA checkpoints.Source: A. Experimental Setup · p. 5

Metrics and real-robot evaluation

  • The study reports episodic reward for all embodiments, plus Go2 tracking error, fall fraction, onboard latency, and hardware-to-simulation joint-position agreement.A fall is defined by base height below 0.2 m; onboard latency is measured with single-thread ONNX Runtime on the Go2 CPU.Source: A. Experimental Setup · p. 5

B. Cross-Embodiment Few-Step Control

RFPO keeps one-step returns close to 64-step returns across four robot embodiments and both zero and random initialization. Comparisons show that FPO++ and action distillation alone remain fragile at one step, while qualitative examples show fixed one-step RFPO deployment in simulation and on hardware.

B. Cross-Embodiment Few-Step Control

  • FPO++ shows a substantial few-step discretization gap on every task, and KD-only still degrades severely at one step despite improving full-step control on some embodiments.Fig. 3’s Go2 example shows FPO++ and KD-only failing to track a 0.8 m/s command, while RFPO maintains stable command tracking.Source: B. Cross-Embodiment Few-Step Control · p. 5, Figure 3 · p. 6
  • RFPO’s 64-to-1-step return changes under zero initialization are −1.5%, +0.4%, −0.9%, and +1.7% on Go2, Spot, H1, and G1, respectively; random-initialization changes stay within 2.4%.The results indicate that one-step behavior is not specific to one morphology or deterministic zero initialization.Source: B. Cross-Embodiment Few-Step Control · pp. 5–6
  • Fig. 4 presents representative simulation and real-world executions of fixed one-step RFPO for forward locomotion, lateral locomotion, and yaw rotation.Frames are sampled at comparable normalized phases of each motion sequence.Source: Figure 4 · p. 6

C. Ablation Study

On Go2, online Reflow preserves episodic reward as Euler steps are reduced, while action-space supervision complements it by maintaining command tracking. The onboard evaluation also reports substantially lower inference latency at one step.

C. Ablation Study

  • Uniform Reflow keeps Go2 zero-initialized return nearly flat from 64 to one step (33.57 → 32.84), whereas FPO++ falls from 21.29 to −187.70 and KD-only collapses to −4919.70.Reward-aware Reflow shows similar numerical robustness; the paper identifies transport rectification as the primary mechanism for robust coarse integration.Source: C. Ablation Study · p. 6, Table II · p. 6
  • One-step tracking errors are 0.068 for Reflow+KD and 0.075 for RFPO, with no observed falls; reward-aware Reflow alone has 0.805 tracking error despite high episodic reward.The passage concludes that Reflow provides numerical robustness and action-space supervision complements it by preserving control fidelity.Source: C. Ablation Study · p. 6, Table III · p. 7
  • Fig. 5 shows concentrated transport bands that evolve smoothly over flow time for representative hip and thigh joints across all four Go2 legs.The action-flow density visualization provides a qualitative view of the learned transport structure.Source: C. Ablation Study · p. 6, Figure 5 · p. 7

D. Onboard Inference Efficiency

E. Real-Robot Deployment

Real-robot evaluation reports consistent matched-command joint responses between simulation and hardware for three Go2 motions, while the conclusion reports strong one-step control performance. The paper limits its hardware evidence to Go2 locomotion and notes additional training and deployment constraints.

E. Real-Robot Deployment

  • Joint-position trajectories show correlations of 0.992–0.997 and MAE of 0.055–0.085 across forward, lateral, and yaw motions.The matched-command analysis measures joint-position consistency, not full state-level sim-to-real equivalence.Source: E. Real-Robot Deployment · p. 7, Table V · p. 7

V. LIMITATIONS

  • Real-robot validation is limited to Unitree Go2 locomotion and a limited set of commands, with matched-command analysis assessing joint-position consistency rather than full state-level equivalence.Broader humanoid and contact-rich real-world evaluation would further establish generality.Source: §V · pp. 7–8
  • Deployment uses a fixed one-step Euler solver, while online Reflow requires full-budget student rollouts and action distillation relies on a separately trained frozen PPO controller.Adaptive compute regularization is used only during training; the authors identify reducing training overhead and teacher dependence, and online budget selection, as future directions.Source: §V · p. 8

VI. CONCLUSION

  • One-step execution retains 98.5% of 64-step reward on Unitree Go2, and one-step returns remain within 2.4% of corresponding 64-step returns across two quadruped and two humanoid embodiments.On Go2, onboard mean inference latency falls from 4.39 ms to 0.08 ms, a 54.9× speedup; real-robot experiments validate stable one-step locomotion across multiple commands.Source: §VI · p. 8
Loading 2610.10453v1…