PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey of Flow Matching in Reinforcement Learning

Nabuat Zaman Nahim, Fairoz Nower Khan, Peizhong Ju

arXiv:2610.16275v1cs.LGcs.AI

TL;DR

Flow-based RL has grown across offline learning, online fine-tuning, and alignment, but its literature lacks a systematic overview. This survey organizes the field by modeled distribution and RL-signal integration, synthesizing a unified taxonomy and comparative account of flow-based methods. It concludes that flow matching offers a favorable trade-off against diffusion in RL contexts, while single-step generation has a quality boundary during early exploration.

  • Problem

    Flow-based RL spans diverse applications but lacks a systematic overview connecting its algorithmic principles and contributions.

    Source: §1 · p. 2

  • Method

    The survey organizes flow-matching methods by the distribution modeled and the mechanism through which the RL signal enters.

    Source: §12 · p. 42

  • Results

    Flow matching offers a more favorable trade-off than diffusion in RL contexts, combining multimodal expressivity with superior inference latency, deterministic safety bounds, and stable gradient pathways.

    Source: §10.5 · p. 38

  • Takeaways & Limitations

    The taxonomy connects offline energy-weighted regression, online policy optimization, and flows for distributions beyond actions within a unified view of flow-based RL.

    Source: §12 · p. 42

    Single-step collapse's quality guarantee depends on a low-variance target distribution, a condition that holds at convergence but not during early exploration.

    Source: §7.2 · p. 32

Abstract

from arXiv · show

Flow Matching (FM) has recently emerged as a principled and efficient generative modeling framework for reinforcement learning (RL), enabling expressive, multimodal policy parameterizations via deterministic probability transport. Compared to diffusion-based policies that rely on stochastic denoising chains, FM uses sampling based on ordinary differential equations (ODEs), with learned velocity fields, which can substantially reduce inference latency and simplify the incorporation of RL objectives. As research in flow-based RL rapidly accelerates across offline continuous control, online fine-tuning, and foundation model alignment, the literature has become highly fragmented. In this survey, we provide a comprehensive taxonomy of flow-matching approaches in reinforcement learning. We organize the literature along two axes: the target distribution being modeled (e.g., action policies, value critics, transition dynamics) and the mechanism of RL signal integration (e.g., energy-weighted regression, flow-based policy gradients, and group relative policy optimization). Furthermore, we survey emerging frontiers such as discrete and non-Euclidean action spaces, provide a systematic comparative analysis against Gaussian and diffusion baselines, and outline critical open problems. Ultimately, this survey serves as a foundational roadmap for the next generation of generative reinforcement learning and alignment.

1 Introduction

Flow matching offers a deterministic, ODE-based alternative for expressive generative policies, while its rapid spread across RL settings has left the literature without a unified overview. This survey positions flow matching as a distinct framework and organizes its methods, applications, comparisons, and open challenges.

1 Introduction

  • Flow matching learns deterministic velocity fields that transport samples from a simple base distribution to a target, turning generative modeling into supervised regression and enabling efficient sampling.Unlike diffusion models, which reverse stochastic noise processes, flow matching uses ODEs and can often sample with relatively few solver steps.Source: §1 · p. 1
  • Flow-based RL has expanded across offline policy learning, online policy optimization, alignment, and structured decision-making, but the literature lacks a systematic overview connecting these contributions.The surveyed uses extend beyond action policies to critics, dynamics, preferences, and multi-agent behavior.Source: §1 · pp. 1–2

1.1 Motivation

  • Diagonal Gaussian policies factorize across action dimensions and represent a single mode, so they can average distinct valid behaviors into unsafe or suboptimal intermediate actions.This mode-averaging issue is especially relevant in offline RL and imitation learning, where datasets may contain multiple valid strategies.Source: §1.1 · p. 2

1.2 Positioning Relative to Existing Surveys

  • The survey distinguishes flow matching from diffusion-focused reviews by examining deterministic vector fields and their RL objectives, then organizes work across offline learning, online adaptation, and broader flow applications.Its scope also includes emerging directions, benchmarks, comparisons with Gaussian and diffusion baselines, and open challenges.Source: §1.2 · pp. 3–4

2 Background

The background introduces RL objectives and generative-model trade-offs, then explains how flow matching trains and samples deterministic probability transports. It frames the central integration choice as whether RL signals shape offline training or online policy updates.

2.1 Reinforcement Learning Preliminaries

  • Offline RL learns from a fixed dataset and often targets value improvement while constraining deviation from behavior; online RL improves policies through environment interaction and policy gradients.The ideal offline target reweights behavior actions by exponentiated Q-values, assuming access to the true Q-function.Source: §2.1 · pp. 4–5

2.2 Diffusion Models in Reinforcement Learning

  • Diffusion models represent multimodal actions and trajectories, but their many denoising steps can create substantial inference latency, while stochastic trajectories can complicate temporal consistency, controllability, and value-based optimization.High-quality generation may require tens or hundreds of denoising steps per action.Source: §2.2 · pp. 5–6

2.3 Flow Matching

  • Flow matching learns a deterministic velocity field transporting a base distribution to a target by regressing on sampled interpolation points, without solving the ODE during training.Under ideal optimization, the induced marginal flow matches the target distribution.Source: §2.3 · pp. 6–7
  • For the common linear optimal-transport path, the target velocity is constant and trajectories are approximately straight, supporting accurate sampling with as few as 1–5 function evaluations.The cited comparison is against diffusion models that often require tens or hundreds of denoising steps.Source: §2.3 · pp. 7–8

2.4 High-level summary of methods for FM on RL

  • Offline flow-based RL commonly incorporates reward or value information through training-time weighted regression, which biases the learned flow toward high-value actions.The target policy is an energy-reweighted behavior distribution, and the regression weight typically increases with Q-values or advantage estimates.Source: §2.4 · pp. 8–9
  • Online flow-policy optimization uses environment interaction and can rely on stochasticized flows, gradients through the ODE, surrogate objectives, or GRPO-style updates.Offline and online methods share a flow-policy backbone and may use KL regularization, Wasserstein anchoring, clipping, or support constraints.Source: §2.4 · pp. 8, 10, Figure 2 · p. 9

3 A Taxonomy of Flow Matching in Reinforcement Learning

The taxonomy organizes flow-based RL along two axes: what distribution a flow models and where the RL signal enters. It also distinguishes offline and online learning from applications beyond policy parameterization.

3 A Taxonomy of Flow Matching in Reinforcement Learning

  • The taxonomy first classifies methods by the modeled distribution—actions, critics, dynamics, or preferences—and by whether the RL signal enters through weighted regression, ODE or pathwise gradients, or group-relative optimization.The survey separately organizes its narrative by offline, online, and beyond-policy learning settings.Source: §3 · pp. 10–11

3.1 Axis A: What Distribution is Modeled?

  • Action flows model conditional action distributions and can represent multimodal behavior; other applications model return distributions, environment transitions, preferred outputs, or joint multi-agent actions.The survey describes action modeling as the most common application and includes examples spanning critics, dynamics transfer, alignment, and decentralized execution.Source: §3.1 · pp. 11–12

3.2 Axis B: Where Does the RL Signal Enter?

  • The RL signal can enter through training-time weighted regression, online policy gradients, or group-relative optimization that ranks sampled candidates by reward.The two-axis view makes under-explored combinations visible, including GRPO for critic or dynamics flows and weighted-regression methods for those flow types.Source: §3 · p. 11, §3.2 · p. 12

4 Offline RL with Q/Energy-Guided Flow Policies

Offline flow RL addresses distribution shift by improving policies learned from fixed datasets while managing out-of-distribution actions. The surveyed methods differ in where value guidance enters and how they handle support constraints and inference cost.

4.1 Regularized Policy Optimization as an Energy-Weighted Target

  • KL-regularized offline RL targets an energy-reweighted behavior distribution, but unknown behavior density and an intractable partition function prevent direct sampling in continuous action spaces.Flow policies can approximate this target while avoiding partition-function evaluation.Source: §4.1 · p. 13

4.2 Energy-Weighted Flow Matching (EFM / QIPO)

  • QIPO/EFM uses energy-weighted flow regression to favor high-value actions, with normalized support-set weights avoiding partition-function computation and ODE-solver backpropagation.Its regression objective recovers the guided flow in expectation without explicitly modeling intermediate energy.Source: §4.2 · pp. 14–15

4.3 FlowQ: Approximating Energy-Guided Paths

  • FlowQ approximates guidance-induced changes to intermediate conditionals with a first-order energy expansion, yielding a shifted-mean Gaussian target velocity for regression.Training remains simulation-free, while inference uses plain ODE sampling without additional test-time guidance.Source: §4.3 · pp. 15–16

4.4 Flow Q-Learning: One-Step Improvement on Top of a Flow Prior

  • FQL keeps flow training behavioral and moves value maximization to a separate one-step actor regularized toward the flow prior, avoiding ODE-time inference costs.The actor is trained with a reparameterized policy-gradient objective and a distillation term.Source: §4.4 · p. 16

4.5 ReFORM: On-Support Improvement via Reflected Noise Manipulation

  • ReFORM combines a behavior-cloning flow with bounded latent noise and reflected manipulation to construct on-support improvement rather than rely solely on statistical penalties.The practical projected Euler step preserves latent-support containment, though it is not mathematically equivalent to the continuous reflected ODE.Source: §4.5 · p. 17

4.6 Efficiency and Stability Add-ons

  • Efficiency add-ons use one-step completion to reduce inference cost and adaptive weight scaling to address variance from peaked Q-weights; Table 2 organizes core methods by guidance, efficiency, and safety.The core designs place guidance in normalized regression weights, intermediate path conditionals, a distilled actor, or structural support constraints.Source: §4.6 · p. 18, Table 2 · p. 18

5 Online Reinforcement Learning with Flow-Matching Policies

This section surveys online reinforcement learning with flow-matching policies, focusing on exploration and policy-gradient needs alongside deterministic ODE-generated actions. It covers noise-injected, direct or surrogate, and group-based optimization; regularization against policy collapse, including adaptive regularization; and a forward-process bridge from diffusion RL to a supervised flow-matching objective.

5 Online Reinforcement Learning with Flow-Matching Policies

  • Online flow policies generate actions through deterministic ODEs, complicating stochastic exploration and policy-gradient estimation; existing methods add noise, optimize through or around the ODE, or use group-relative optimization.These three families differ in how they introduce stochasticity, compute policy gradients, and ensure stability.Source: §5 · p. 19

5.1 Method 1: Noise Injection and Fine-Tuning

  • ReinFlow adds learnable Gaussian transitions to discretized flow sampling for tractable likelihoods and PPO-style updates, then removes the noise module for deterministic deployment.The added noise supports training-time exploration and likelihood computation, while deployment returns to fast deterministic ODE inference.Source: §5.1 · pp. 19–20

5.2 Method 2: Direct and Surrogate Policy Optimization for Flow Policies

  • FPO (McAllister et al., 2025) uses conditional flow-matching losses to form a PPO-style surrogate ratio, avoiding exact likelihoods and remaining agnostic to the sampling method.More generally, direct approaches can treat the flow policy as a reparameterized transport map and differentiate through the ODE solution.Source: §5.2 · pp. 20–21
  • For variable-horizon action chunks, Pfrommer et al. (2025) use reward-weighted flow matching or GRPO to inject RL signals through weighted regression rather than ODE-likelihood gradients.RWFM weights trajectories by exp(αR), while GRPO uses normalized group-relative advantages in the flow-matching regression objective.Source: §5.2 · p. 21
  • PolicyFlow (Yang et al., 2026) estimates the ODE-divergence term with a Hutchinson trace estimator to approximate PPO likelihood ratios while retaining flow-model multimodality and PPO stability.Exact likelihood computation requires integrating velocity-field divergence along the ODE trajectory, which is expensive for on-policy RL.Source: §5.2 · p. 21

5.3 Method 3: The GRPO Family (Group Relative Policy Optimization)

  • GRPO-family flow methods modify exploration, temporal credit assignment, or variance reduction; Flow-GRPO also uses group ranking to align text-to-image flows with human preferences.Flow-GRPO converts the ODE to an SDE; TempFlow-GRPO (He et al., 2025) weights updates by timestep; Smart-GRPO optimizes noise perturbations; SuperFlow (Chen et al., 2025c) adapts group size to optimization difficulty; F5R-TTS (Sun et al., 2025) applies GRPO to speech using WER and speaker-similarity rewards.Source: §5.3 · p. 23

5.4 Adaptive Regularization

  • ORW-CFM-W2 regularizes online reward-weighted flow updates toward a reference distribution, using a tractable Wasserstein-2 penalty to help prevent reward-driven collapse and diversity loss.Its velocity-field-based upper bound avoids the expensive likelihood computations required by common divergence measures such as KL.Source: §5.4 · p. 23
  • ADRPO adjusts divergence regularization by advantage, allowing reliable high-advantage samples greater deviation while constraining uncertain or low-advantage samples; it improves alignment quality while maintaining diversity.The passages report better reward-diversity trade-offs and performance over fixed-regularization methods such as ORW-CFM-W2 for text-to-image alignment and multimodal reasoning.Source: §5.4 · p. 24
  • Adaptive regularization computes a coefficient, uses g(·) to reduce penalties for high-advantage samples in the weighted flow-matching loss, and adds adaptive Wasserstein regularization.Source: §5.4 · p. 25

5.5 Bridging Diffusion and Flows: Forward Process Optimization

  • DiffusionNFT (Zheng et al., 2025b) contrasts positive and negative generations to define an implicit policy-improvement direction, optimizing diffusion models directly on the forward process with a supervised flow-matching objective.The approach addresses the intractable likelihoods of online diffusion RL.Source: §5.5 · p. 25

6 Beyond Policies: Critics, Dynamics, and Alignment

Flow matching extends beyond policy generation to value estimation, dynamics transfer, preference alignment, and general guidance. Across these uses, learned transport processes provide a common way to reshape values, transitions, or output distributions toward desired targets.

6.1 floq: Training Critics via Flow Matching

  • floq models the critic itself as a flow and improves value estimation on challenging offline RL benchmarks as test-time integration steps increase.The number of flow steps provides a compute-scaling axis for value estimation, distinct from increasing network width or ensemble size.Source: §6.1 · pp. 25–26

6.2 Dynamics and Transfer

  • CompFlow estimates source–target dynamics mismatch through flow-based transport and uses the discrepancy to guide data selection and exploration.Its composite model initializes online adaptation from the offline source-transition flow; large estimated gaps guide exploration toward poorly represented regions.Source: §6.2 · pp. 27–28

6.3 Preference Alignment and Reasoning

  • Preference Flow Matching transports rejected outputs toward preferred ones, while Flow Balancing targets the distribution of high-reward reasoning trajectories to preserve diversity while improving correctness.The survey describes these distribution-level objectives as alternatives to scalar reward optimization and explicit reward modeling.Source: §6.3 · pp. 28–29

6.4 General Guidance Frameworks

  • A unified guidance framework interprets methods such as classifier, energy, and diffusion guidance as special cases, and connects RL guidance methods to modifications of the flow’s transport dynamics.The framework derives multiple guidance families and treats existing RL approaches as approximations to an optimal guidance field.Source: §6.4 · pp. 29–30

7 Emerging Frontiers: Action Spaces, Architectures, Value Estimation, and Alignment

Emerging work extends flow matching to structured action spaces, addresses stability and inference costs, applies flows to return distributions, and uses generation trajectories for alignment and planning. The approaches preserve the transport perspective while adapting its geometry, computation, or learning signal to each setting.

7.1 Structured and Non-Euclidean Action Spaces

  • Discrete, combinatorial, and manifold actions require geometry-aware transport; current methods enforce feasibility through discrete jumps, solver projection, or geodesic flows.Discrete and combinatorial approaches incorporate Q-weighted policy improvement, while the surveyed Riemannian policies have so far been evaluated only in imitation learning.Source: §7.1 · pp. 30–31

7.2 Architectural Innovations for Stability and Acceleration

  • Three strategies address online flow-policy stability and cost: redesign the ODE backbone, learn a correction on a frozen flow, or collapse generation to one step.Backbone redesign retains multi-step expressiveness but adds training overhead; residual correction is local to the base distribution; single-step methods can match Gaussian-policy inference speed but may lose multimodality when target variance is high.Source: §7.2 · pp. 31–32

7.3 Generative Value Estimation (Distributional RL)

  • Distributional flow critics model return distributions rather than only expected Q-values, with methods using test-time integration, one-step distillation, or flow-based target construction.The surveyed approaches include floq, DFC, FlowCritic, and EVOR; whether distributional flow critics combine with online RL objectives remains largely untested.Source: §7.3 · p. 32

7.4 Fine-Grained Alignment and Trajectory Planning

  • Alignment methods exploit the flow’s temporal structure for intermediate credit assignment, while trajectory-generation methods model extended state-action sequences or future occupancy.Trajectory-level generation can capture long-range dependencies, but its distributions are higher-dimensional and harder to model accurately; most surveyed methods have been evaluated in relatively simple environments.Source: §7.4 · pp. 32–33

8 Applications of Flow Matching in RL

Flow-matching RL methods are applied across domains whose action geometry, inference budgets, and data modalities shape their design. The surveyed applications range from robotic control and generative-model alignment to speech, reasoning, transfer, and multi-agent tasks.

Robotics and visuomotor control

  • Robotics is the largest application area, spanning single-step visuomotor policies, flow-prior actors, SE(3) pose policies, VLA fine-tuning, and residual refinement for manipulation.The surveyed approaches target diffusion inference latency and support both high-frequency control and local policy refinement.Source: §8 · p. 34

Image and video generation alignment

  • Text-to-image alignment is a major application for flow-based fine-tuning, with GRPO-family methods and stability or credit-assignment variants evaluated using human-preference reward models.The survey also describes flow matching for online RL on diffusion image-generation backbones.Source: §8 · p. 34

Speech, reasoning, and multi-agent applications

  • Other applications include GRPO-based speech and singing generation, reasoning-trajectory distribution matching, and centralized-to-decentralized multi-agent coordination.The survey also covers goal-conditioned trajectory generation and intention-conditioned flow occupancy models for navigation.Source: §8 · p. 34

Transfer learning under dynamics shift

  • CompFlow uses flow matching as a source–target dynamics discrepancy estimator to guide data selection and exploration during transfer learning.The method applies the optimal-transport interpretation of flow matching to environment dynamics.Source: §8 · p. 34

9 Benchmarks and Evaluation Protocols

The survey covers benchmark protocols across offline and online continuous control, generative alignment, and action spaces. It emphasizes that task performance, inference cost, and benchmark family must be considered together, since results from different settings are not directly interchangeable.

Offline RL benchmarks

  • Offline studies commonly report normalized return on D4RL, while newer flow-based work also uses OGBench’s state- and pixel-based tasks.FQL uses OGBench as its main benchmark and also reports D4RL results.Source: §9 · p. 35

Online RL and continuous control benchmarks

  • Online evaluations use several continuous-control suites, but flow-matching methods have not converged on a shared benchmark, making head-to-head comparison difficult.The reviewed methods span MuJoCo Playground, DeepMind Control, Gymnasium MuJoCo, Franka Kitchen, and Robomimic.Source: §9 · p. 35

Alignment and action-space scope

  • Alignment methods are evaluated on text-to-image benchmarks such as GenEval, PickScore, and OCR, while discrete-action benchmarks remain largely absent from flow-matching RL.The survey identifies extending flow matching to discrete or hybrid action spaces as an open direction.Source: §9 · pp. 35–36

Evaluation metrics

  • Inference evaluation includes NFE and wall-clock time, since methods with similar returns can differ in deployment cost and real-time latency.NFE counts the denoising or ODE-solver steps used to generate one action.Source: §9 · p. 36

Evaluation protocol

  • Fair comparisons should consider task performance, inference cost, and benchmark family together because results across benchmark families are not directly interchangeable.Inference cost includes NFE or milliseconds per action; the survey also notes that improvements under one benchmark or inference budget may not transfer to another setting.Source: §9 · pp. 35–36

10 Comparative Analysis: Flows, Diffusion, and Gaussians

The comparison finds that flow matching retains generative policies’ multimodal expressivity while offering advantages over diffusion in inference latency, deterministic support constraints, and gradient estimation. Its practical comparisons remain conditioned on benchmark family and evaluation protocol.

10.1 Representational Expressivity: The Multimodal Imperative

  • Unlike unimodal Gaussian policies, generative flow and diffusion policies can represent multiple action modes, addressing mode averaging in heterogeneous datasets.The comparison describes flow matching and continuous-time diffusion as theoretically equivalent in density expressivity, with their difference lying in probability-transport mechanisms.Source: §10.1 · pp. 36–37

10.2 Trajectory Geometry: Stochastic SDEs vs. Deterministic ODEs

  • Flow matching uses deterministic ODE transport, and reflected boundary conditions in ReFORM keep generated actions within the training data’s safe support.The survey also connects deterministic paths with action-sequence consistency under slight latent-noise perturbations in planning.Source: §10.2 · p. 37

10.3 Inference Latency: The NFE Bottleneck

  • Diffusion policies often require 5 to 100 NFE per action, whereas flow policies can be distilled or evaluated in 1 to 4 NFE with minimal degradation in expected return.The survey identifies NFE as a key inference-latency measure; in online RL it can also affect training cost when gradients pass through sampler steps.Source: §10.3 · p. 37

10.4 Optimization Stability and Gradient Estimation

  • PolicyFlow approximates the old-to-updated CNF importance ratio using a Gaussian likelihood-ratio identity and velocity-field interpolation, enabling PPO-style optimization without fully simulating the ODE during training.The comparison contrasts this pathway with high-variance gradients from differentiating through an unrolled diffusion sampling chain.Source: §10.4 · p. 38

10.5 Summary of Trade-offs and Empirical Comparisons

  • Flow matching offers a more favorable RL trade-off than diffusion through multimodal expressivity, lower inference latency, deterministic safety bounds, and stable gradient pathways, while Gaussian policies remain more computationally efficient.Table 4 summarizes the architectural trade-offs among the policy classes.Source: §10.5 · p. 38, Table 4 · p. 38
  • On OGBench’s 50 state-based tasks, FQL scores 43.8 versus 31.0 for the strongest Gaussian baseline, using one function evaluation per action.The consolidated table reports results under the individual papers’ protocols, not a head-to-head reproduction.Source: §10.5 · p. 39, Table 5 · p. 39
  • Online results show flow policies competitive with or stronger than diffusion and Gaussian baselines, but the table’s scratch-training and pretrained fine-tuning blocks are not directly rankable.The scratch-trained methods use MuJoCo Playground and DeepMind Control, whereas the fine-tuning block uses Gymnasium MuJoCo.Source: §10.5 · p. 39, Table 6 · p. 40
  • On text-to-image alignment benchmarks, TempFlow-GRPO reports GenEval 0.97 in 3,800 steps versus Flow-GRPO’s 0.95 in 5,600 steps, while AWM reports 8–24× wall-clock speedups.The reported results are collected from individual papers rather than reproduced under identical conditions.Source: §10.5 · p. 40, Table 7 · p. 40

11 Open Problems and Future Directions

The survey identifies unresolved challenges in exploration, off-policy learning, high-dimensional visuomotor control, inference speed, and preference alignment. It outlines directions for addressing each while noting that current approaches leave important constraints in place.

11.1 Principled Exploration in Deterministic Flows

  • Online flow methods inject Gaussian noise or convert the ODE to an SDE for exploration, but these heuristics compromise the straight-line optimal-transport properties sought for efficiency.The survey proposes exploration-aware vector fields, including distributions over vector fields or optimistic initialization of the base distribution.Source: §11.1 · p. 41

11.2 Off-Policy and Continuous-Time RL Integration

  • Exact likelihood evaluation makes off-policy methods prohibitively slow for many flow policies, and sample-efficient off-policy continuous control remains unsolved.The survey also identifies a potential connection between continuous-time flow generation and Bellman equations in continuous-time MDPs.Source: §11.2 · p. 41

11.3 Scaling to High-Dimensional Visuomotor Control

  • State-based and simple image-based benchmarks leave a gap to generalist robotics, while high-dimensional visual context can burden flow models relative to action dimensionality.Proposed directions include latent-space RL objectives and Transformer architectures optimized for high-frequency control.Source: §11.3 · p. 41

11.4 Hardware-Aware Solvers and Single-Step Distillation

  • Even 3 to 5 ODE-solver steps per interaction can be too slow for high-frequency robots, while single-step distillation may lose multimodal expressivity through capacity bottlenecks.The survey suggests hardware-aware solvers and progressive distillation that can trade inference latency against action precision.Source: §11.4 · pp. 41–42

11.5 Alignment Beyond Scalar Rewards

  • Scalar reward models can encourage mode collapse toward a single safest response, motivating direct optimization of multidimensional preferences or human feedback.Preference Flow Matching is cited as an approach that treats alignment as transport from rejected to preferred distributions; scaling this to reasoning and coding remains open.Source: §11.5 · p. 42

12 Conclusion

The survey presents a unified taxonomy of flow matching in RL and argues that deterministic probability transport supports decision-making across modeled distributions and RL-signal integration mechanisms. It also identifies unresolved challenges and discusses the dual-use implications of deploying the algorithms it surveys.

12 Conclusion

  • The taxonomy organizes methods by the distribution modeled—from actions to dynamics and critics—and by how they integrate RL signals, from offline energy-weighted regression to online group relative policy optimization.The survey frames deterministic probability transport as a general-purpose tool for decision-making.Source: §12 · p. 42

Broader Impact Statement

  • The survey discusses algorithms used to train autonomous systems, robotics, and large language models, noting that improved sample efficiency and alignment carry dual-use implications.It identifies safe and ethical alignment of reward-guided models as a research-community responsibility, especially for physical or user-facing deployments.Source: Broader Impact Statement · p. 42

A.1 Overview of Diffusion in RL methods

Diffusion-RL methods can be classified by the generated object, spanning planning, policies, constraints, imitation and skills, world models, and preference or reward alignment.

A.1 Overview of Diffusion in RL methods

  • Diffusion planners generate future state-action trajectories together, treating planning as data imputation; Diffuser uses a temporal U-Net and separate reward classifiers.Later work explores hierarchical and latent-space planning, trajectory stitching, priors, and guidance.Source: §A.1 · p. 51
  • Diffusion policies generate one action conditioned on the current state; offline methods pair diffusion actors with Q-learning critics for multimodal datasets (Hansen-Estruch et al., 2023).Robotics applications include visuomotor control and multi-task manipulation.Source: §A.1 · p. 51
  • For constrained or safe RL, diffusion generation incorporates feasibility constraints through guidance signals or constrained policy optimization (He et al., 2023b).Source: §A.1 · p. 51
  • Beyond offline RL benchmarks, diffusion supports goal-conditioned imitation, skill learning, shared autonomy, human-robot collaboration, data augmentation, and behavior retargeting (Reuss et al., 2023; Jain & Ravanbakhsh, 2024; Yoneda et al., 2023; Ng et al., 2023; Chen et al., 2023a; Xu et al., 2023b; Chen et al., 2023c; Lu et al., 2023b; Yu et al., 2023).Source: §A.1 · p. 51
  • As world models, diffusion models generate plausible futures, including in partially observed or visually rich settings such as Atari (Lee et al., 2025b).Source: §A.1 · p. 52
  • A separate line uses RL-style training to align diffusion generation with preferences or task rewards, including customization and reward-driven pipelines.These works appear in diffusion-RL bibliographies because they explicitly optimize diffusion generation using RL signals.Source: §A.1 · p. 52

A.2 How RL objectives enter diffusion models: training-time vs sampling-time

Diffusion-RL methods differ in when the reward or value signal enters: during model training or at sampling time.

A.2 How RL objectives enter diffusion models: training-time vs sampling-time

  • Training-time integration incorporates RL signals directly through value-weighted objectives, constrained updates, or actor-critic and policy-gradient training with diffusion policies (Wang et al., 2024; Li et al., 2024; Fang et al., 2025).Source: §A.2 · p. 52
  • Sampling-time guidance keeps the base diffusion model fixed and biases denoising with an external score, energy, or value signal; energy-guided diffusion is an offline RL example.Related guidance appears in planning, including prior-guided planning and flexible guidance in driving (Ki et al., 2025).Source: §A.2 · p. 52

A.3 Efficiency and fast sampling

Diffusion control faces inference-speed and control-stability challenges; the survey presents flow matching as a response to the speed–quality conflict.

A.3 Efficiency and fast sampling

  • Diffusion-RL accelerates sampling with learning-free solvers such as DDIM and DPM-Solver, or learning-based shortcut training and distillation; offline variants also reduce long-chain backpropagation or approximate training-time sampling (Song et al., 2020; Lu et al., 2022; Song et al., 2023; Ding & Jin, 2024; Chen et al., 2023b; Fan & Lee, 2023; Kang et al., 2023; Venkatraman et al., 2024).Source: §A.3 · p. 52

A.4 From Diffusion to Flow Matching

  • Standard diffusion policies require 10–100 denoising steps per action, while fast solvers can degrade sample quality below 5 steps (Song et al., 2020).Source: §A.4 · p. 52
  • The stochastic reverse SDE complicates precise solver use and makes smooth control updates difficult.Source: §A.4 · p. 52
  • The survey motivates flow matching by the conflict between high-quality generation and real-time inference, citing deterministic straight-line trajectories as enabling faster, more stable policy learning.Source: §A.4 · p. 52

Related papers

Loading 2610.16275v1…