Source-linked AI summary
Mean Flow Policy with Instantaneous Velocity Constraint for One-step Action Generation
Guojian Zhan, Letian Tao, Pengcheng Wang, Yixiao Wang, Yiheng Li, Yuxin Chen, Hongyang Li, Masayoshi Tomizuka, Shengbo Eben Li
TL;DR
Generative policies face a trade-off between expressive action distributions and the computational cost of iterative sampling. The paper proposes MVP, which models mean velocity for one-step generation and adds IVC as a boundary condition; it reports state-of-the-art robotic manipulation success rates with substantial training and inference speed improvements.
Problem
Existing generative policies require iterative multi-step refinement, creating training and inference overhead despite their ability to model complex action distributions.
Method
MVP models the mean velocity field for direct one-step generation and uses IVC during training as an explicit boundary condition.
Results
MVP achieves state-of-the-art success rates on Robomimic and OGBench while substantially improving training and inference speed over flow-policy baselines.
Takeaways & Limitations
One-step mean-velocity generation combines the reported expressiveness of generative policies with improved efficiency for robotic control.
Takeaways & Limitations
MVP has higher learning difficulty than a standard flow policy because its mean-velocity ODE lacks explicit boundary conditions, which can affect learning accuracy and policy expressiveness.
Abstract
from arXiv · showhide
Learning expressive and efficient policy functions is a promising direction in reinforcement learning (RL). While flow-based policies have recently proven effective in modeling complex action distributions with a fast deterministic sampling process, they still face a trade-off between expressiveness and computational burden, which is typically controlled by the number of flow steps. In this work, we propose mean velocity policy (MVP), a new generative policy function that models the mean velocity field to achieve the fastest one-step action generation. To ensure its high expressiveness, an instantaneous velocity constraint (IVC) is introduced on the mean velocity field during training. We theoretically prove that this design explicitly serves as a crucial boundary condition, thereby improving learning accuracy and enhancing policy expressiveness. Empirically, our MVP achieves state-of-the-art success rates across several challenging robotic manipulation tasks from Robomimic and OGBench. It also delivers substantial improvements in training and inference speed over existing flow-based policy baselines.
1 INTRODUCTION
Existing generative policies can represent complex action distributions but incur iterative sampling costs, motivating a policy that combines expressiveness with one-step generation. MVP models mean velocity, while IVC supplies a boundary condition intended to improve learning accuracy and expressiveness.
- Motivation: Generative policies model flexible, potentially multi-modal action distributions but depend on iterative refinement that slows online training and real-time inference.The computational overhead is especially relevant when actions must be sampled at every environment step.
- MVP: MVP learns the mean velocity field to map Gaussian noise directly to a multi-modal action distribution in one step.This design preserves the expressive role of flow-based policies while removing multi-step sampling overhead.
- Learning challenge: The mean-velocity ODE can have multiple solutions because its boundary value is not explicitly enforced, making accurate and expressive learning more difficult.MVP must model mean velocity over arbitrary time intervals, while the underlying first-order ODE lacks an explicit boundary condition.
- IVC: IVC pairs interval-average velocity loss with instantaneous velocity loss at the interval start, acting as an auxiliary boundary-condition loss.The method adds negligible computational overhead while improving accuracy.
- Results: MVP achieves state-of-the-art success rates on Robomimic and OGBench while substantially speeding training and inference relative to flow-policy baselines.The reported evaluation covers challenging robotic manipulation benchmarks and emphasizes practical real-time use.
2 PRELIMINARIES
The preliminaries frame reinforcement learning as policy optimization in an MDP and introduce flow matching as deterministic ODE-based generative modeling. Flow matching learns a continuous velocity field that transports a source distribution toward a target distribution, with practical sampling often requiring numerical multi-step integration.
- Reinforcement Learning: An RL problem is modeled as an MDP with state and action spaces, transitions, rewards, and a discount factor.The policy aims to maximize expected cumulative discounted reward.
- Reinforcement Learning: Off-policy learning uses a Q-function to guide policy improvement by evaluating expected cumulative return for state-action choices.The Q-function supports selecting actions that improve expected return.
- Flow Matching: Flow matching constructs continuous-time generative models by learning a deterministic instantaneous vector field governed by an ODE.This contrasts with diffusion models, which use stochastic differential equations.
- Flow Matching: The training path linearly interpolates between source and target samples, with a constant target velocity equal to their difference.The learned field is trained to represent the velocity along this source-to-target path.
- Flow Matching: Although flow matching is formulated with straight paths, fitted paths can curve, so Euler integration and multi-step discretization are often needed for high-quality samples.The numerical solver is applied to the learned probability-flow ODE.
3 METHOD
MVP learns a mean velocity field to map Gaussian noise directly to actions in one step, while generate-and-select uses the critic to choose among diverse candidates. IVC supplies the missing boundary condition that makes mean-flow learning well-posed and improves policy fitting accuracy.
- 3.1 MEAN VELOCITY POLICY: MVP models mean velocity over an interval rather than instantaneous velocity, enabling direct one-step generation from Gaussian noise to actions.The policy combines mean-flow generation with a generate-and-select mechanism for action selection.
- 3.1 MEAN VELOCITY POLICY: The mean-flow training objective minimizes the residual of a mean-flow identity derived by differentiating the interval-based velocity relation.The total time derivative is expanded with the chain rule and can be computed efficiently using a Jacobian-vector product.
- 3.1 MEAN VELOCITY POLICY: Generate-and-select samples N diverse candidate actions, evaluates them with the critic, and uses the highest-Q candidate as the policy output and training target.The resulting action also supports environment interaction and target-value calculation.
- 3.1 MEAN VELOCITY POLICY: The policy-improvement theorem separates a non-negative best-of-N advantage gain from fitting error caused by critic inaccuracy and mean-flow matching error.Reducing mean-flow error is therefore important for improving policy performance under the theorem’s assumptions.
- 3.2 THE INSTANTANEOUS VELOCITY CONSTRAINT AS A BOUNDARY CONDITION: Without a boundary condition, the mean-flow ODE admits a family of solutions with an unknown integration constant, allowing persistent bias in the learned field.The loss supplies dynamics for t < r but is blind to the boundary; near-equal time pairs are too rare to provide a robust implicit condition.
- 3.2 THE INSTANTANEOUS VELOCITY CONSTRAINT AS A BOUNDARY CONDITION: IVC enforces the known instantaneous velocity at the boundary, forcing the integration constant and cumulative error to zero and making the learning problem well-posed.The resulting smaller mean-flow matching error supports more effective policy improvement.
4 EXPERIMENTS
MVP is evaluated on nine sparse-reward robotic manipulation tasks against strong offline-to-online RL baselines, with ablations examining IVC, one-step variants, and efficiency. It matches or exceeds multi-step baselines on eight tasks and combines strong success rates with faster training and inference than most alternatives.
- Benchmark and baselines: The evaluation covers three Robomimic tasks and six OGBench tasks using multi-human and play-style datasets, respectively.
- Main results: MVP matches or exceeds multi-step flow-matching baselines on eight of nine tasks and ranks second on the remaining task with 0.92 versus 0.94.
- Main results: MVP achieves the highest average success rate, 0.88 ± 0.05, and reaches 0.52 ± 0.11 on Cube-triple-task4 versus QC’s 0.46 ± 0.13.
- Ablation study: Increasing the IVC weight improves performance on Cube-triple-task4 from 0.30 ± 0.21 without IVC to 0.45 ± 0.15 with partial IVC.
- Ablation study: Naive one-step baselines achieve success rates near zero on Cube-triple-task3 and Cube-triple-task4, whereas MVP reaches 0.71 ± 0.06 and 0.52 ± 0.11.
- Efficiency analysis: MVP has the highest success rate and fastest online training, while its inference time is similar to FQL and faster than BFN and QC.
- Efficiency analysis: BFN and QC are slower because they rely on 10-step flow policies, while FQL’s fast inference requires training multiple policies and remains slower to train overall.
5 RELATED WORK
Related work positions generative policies as expressive tools for complex action distributions, while emphasizing their iterative-sampling latency in online RL. Policy expressiveness is also relevant to offline pretraining and online adaptation.
- Offline-to-online RL: Offline-to-online RL commonly combines static-dataset pretraining with algorithmic designs such as behavioral regularization, conservatism, and out-of-distribution detection.
- Offline-to-online RL: Expressive policy networks can better capture behavioral-policy distributions offline and support adaptation during online fine-tuning.
- Generative models as RL policies: Generative models can represent complex, multi-modal policies in offline and online RL through expressive transformations.
- Generative models as RL policies: Their iterative sampling requires many function evaluations, creating prohibitive latency for high-throughput online RL.
6 CONCLUSION
The paper concludes that MVP combines one-step action generation with expressive policy modeling through IVC. Experiments support strong success rates and improved training and inference speed, while additional GPU memory and broader validation remain limitations.
- Conclusion: MVP combines high time-efficiency from fastest one-step action generation with expressiveness supported by the instantaneous velocity constraint.
- Conclusion: Robomimic and OGBench results show state-of-the-art success rates and substantial training and inference speed improvements.
- Limitations and future work: Training requires additional GPU memory because of the Jacobian-Vector Product operation.
- Limitations and future work: Future work will validate MVP on more tasks and real robotic platforms.
A THEORETICAL ANALYSIS ON THE MEAN VELOCITY POLICY IMPROVEMENT
This section develops the theoretical analysis underlying policy improvement for the mean velocity policy training paradigm. It organizes the analysis around update procedures, assumptions, a formal theorem, and properties of the improvement gain.
- Theoretical analysis: The analysis provides a theoretical policy improvement guarantee for the mean velocity policy training paradigm.
- Theoretical analysis: It covers policy-update implementation procedures, core assumptions, a useful lemma, the formal improvement theorem, and three properties of the improvement gain.
A.1 IMPLEMENTATION PROCEDURES OF POLICY UPDATE
The policy update samples candidate actions from a base mean velocity policy, selects the highest-critic candidate, and trains the updated policy to match it. The analysis models matching error and critic error to derive a performance-improvement bound, while characterizing the best-of-N advantage gain.
- Policy update procedure: The update samples N candidate actions from πold(·|s) and uses Qϕ to select the best candidate as the target action a∗(s).The selected target is then used in the policy-matching step.
- Policy update procedure: πnew is trained to match a∗(s), but its sampled action need not equal the target; Assumption 3 bounds the expected matching distance by ϵA.The conditional distribution M(·|a∗) describes matching given one target, whereas πnew averages over targets generated by sampling and selection.
- Assumptions: The analysis assumes uniformly bounded critic-fitting error ϵQ and LQ-Lipschitz continuity of Qπold with respect to actions.These assumptions control critic inaccuracies and the Q-value impact of imperfect action matching.
- Performance analysis: The proof applies the Performance Difference Lemma to connect value improvement with the expected advantage of πnew relative to πold.It first bounds the single-step expected advantage and then aggregates the bound over time using the discounted state-visitation distribution.
- Performance analysis: Imperfect matching reduces the target action’s Q-value by at most LQϵA in expectation, using Q-value smoothness and the bounded matching error.The bound is established for a fixed target and then extended to the marginal updated policy.
B.1 NUMERICAL RESULTS OF ABLATION STUDY
The ablation section reports an IVC ablation and compares one-step variants of the baselines, but the supplied passages contain only the table references and no numerical entries.
- IVC ablation: Table 4 is identified as the ablation study measuring the impact of IVC, but its numerical results are not included in the supplied passages.No metric values or task-level comparisons are available here.
- One-step baseline comparison: Table 5 is identified as a comparison with one-step variants of the listed baselines, without numerical results in the supplied passages.The available text does not specify the compared values or outcomes.
C ENVIRONMENTS DESCRIPTION
The evaluation uses Robomimic and OGBench manipulation environments spanning simple pickup and placement tasks to long-horizon multi-cube rearrangements. Supplementary visualizations show representative successful episodes for these challenging, sparse-reward settings.
- Robomimic: Robomimic contains three tasks—lift, can, and square—using 300 successful trajectories per task collected from six operators with mixed proficiency.The tasks require cube pickup, can placement into a smaller bin, and precise placement of a square nut on a rod.
- Robomimic: Robomimic assigns −1 reward when a task is incomplete and 0 when it is completed.All three tasks use binary task-completion rewards.
- OGBench: OGBench cube-double and cube-triple domains require moving two or three cubes to target locations, with reward −nwrong until all cubes are correctly positioned.Episodes terminate when every cube reaches its correct position and the reward becomes 0.
- OGBench: Representative OGBench tasks include moving, swapping, stacking, and cyclically rearranging multiple cubes.The listed tasks span Cube-double-task2, Cube-double-task3, Cube-double-task4, and Cube-triple-task2 through Cube-triple-task4.
- Task complexity: The OGBench tasks increase spatial-reasoning demands through coordinated relocation, stacked-structure disassembly, and cyclic rearrangement.These task structures are described as increasingly complex spatial requirements.
- Supplementary visualizations: Supplementary visualizations depict successful episodes and are intended to demonstrate precise, robust trajectories under long-horizon reasoning and sparse rewards.Figures 7 and 8 cover representative Robomimic and OGBench episodes, while Figure 6 shows the nine challenging tasks.
E REPRODUCIBILITY STATEMENT
The reproducibility statement directs readers to the detailed hyperparameters for all algorithms. These settings are provided in Table 6.
- Hyperparameters: The hyperparameters of all algorithms are reported in Table 6.The supplied passages provide the table reference but not the individual settings.