PaperlayerView exact revision on arXiv

Source-linked AI summary

Reinforcement Learning via Fixed-point Flow Matching

Denis Blessing, Onur Celik, Gerhard Neumann

arXiv:2610.12848v1cs.LGcs.RO

TL;DR

Online RL lacks samples from potentially complex, multimodal Boltzmann targets, limiting the direct use of standard flow-matching training objectives. The paper introduces fixed-point flow matching, which uses critic action gradients and current-policy samples in a flow-matching-style update. It proves the optimal policy is a fixed point and reports competitive or better results than strong baselines across offline-to-online and online tasks.

  • Problem

    Standard flow-matching objectives require samples from the optimal policy, which are unavailable in online RL, even though flow policies can represent complex, multimodal Boltzmann targets.

    Source: §1 · p. 1, §3 · p. 4

  • Method

    FFM uses the target flow identity to regress a flow policy toward critic-gradient targets using samples from the current policy, forming a fixed-point iteration.

    Source: §1 · p. 2

  • Results

    FFM is competitive with or better than the strongest available baselines across 50 offline-to-online tasks and 24 online tasks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The optimal policy is a fixed point of FFM, which trains flow policies without target-policy samples or gradients through the sampling chain.

    Source: §1 · p. 2, §7 · p. 9

    FFM requires a differentiable critic in the action and is restricted to continuous action spaces; the update is only as accurate as the critic's gradient.

    Source: §7 · p. 9

Abstract

from arXiv · show

In maximum entropy and behavior-constrained reinforcement learning (RL), the optimal policy is given by a Boltzmann distribution. Generating samples from this distribution can be challenging, as it is often complex and multimodal. While flow policies trained with flow matching provide a powerful framework for representing such distributions, they traditionally rely on access to training data, which is generally unavailable in online RL. To address this limitation, we propose fixed-point flow matching (FFM), a data-free procedure for training and fine-tuning flow policies in RL. FFM leverages the target flow identity to formulate the actor update as a fixed-point iteration, utilizing a flow-matching-style regression objective where the critic's action gradients serve as regression targets. We show that the optimal policy is a fixed point of this iteration. Through extensive experiments, we demonstrate that FFM effectively fine-tunes pretrained flow policies in the offline-to-online setting and enables efficient and accurate training of flow policies from scratch in online RL. These results establish fixed-point flow matching as a principled approach for learning expressive flow-based policies without requiring training data.

1 INTRODUCTION

Online RL lacks samples from the complex Boltzmann targets that flow policies are suited to represent, while common training objectives and existing update methods have practical constraints. FFM uses critic action gradients in a flow-matching-style fixed-point update, with maximum-entropy and offline-to-online instantiations.

1 INTRODUCTION

  • Flow and diffusion policies can represent complex multimodal targets, but stable training objectives typically require target samples unavailable in online RL.Existing online approaches may backpropagate through the integrator, use high-variance importance-weighted regression, or convert flows into diffusion models for stochastic optimal control.Source: §1 · p. 1
  • The target flow identity replaces data-derived flow-matching targets with critic-gradient regression targets while retaining the same population minimizer.The identity expresses the Boltzmann target velocity as a conditional expectation of the target's log-density gradient.Source: §1 · p. 2
  • FFM evaluates target flow matching on current-policy samples and proves that the optimal policy is a fixed point of the resulting iteration.Each iteration generates actions with the current flow, labels them with critic gradients, noises them, and regresses the velocity field onto those labels.Source: §1 · p. 2, Figure 1 · p. 2
  • The paper introduces a velocity-space trust-region loss, reference-score-based behavior-constrained fine-tuning, and temporally correlated prior draws for exploration.Source: §1 · p. 2
  • FFM is competitive with or better than the strongest available baselines across 50 offline-to-online tasks and 24 online tasks.The paper instantiates FFM for maximum-entropy RL and offline-to-online fine-tuning.Source: §1 · p. 2

2 RELATED WORK

Prior RL work uses several ways to make critics guide flow and diffusion policies, including gradients through sampling, critic-weighted regression, and policy gradients. FFM instead uses critic action gradients in a flow-matching objective and directly fine-tunes pretrained flows in velocity space.

2 RELATED WORK

  • Online methods include critic backpropagation through the sampling chain, critic-weighted regression, and policy-gradient estimates through denoising, while FFM uses critic action gradients in flow matching.The passage describes the first approach as memory-intensive and unstable for many integration steps.Source: §2 · p. 3
  • Closest to FFM, DIPO clones actions improved by Q-gradient ascent, QSM uses noisy-action critic gradients as a plug-in target-score approximation, and RFM estimates the posterior-mean target by importance sampling.The passage says QSM's approximation is exact only at the clean end.Source: §2 · p. 3
  • Offline-to-online methods include flow distillation, latent-noise policies, noise-injected flows, and value propagation; FFM fine-tunes pretrained flows directly in velocity space without SDE conversion or adjoint simulation.FFM's behavior-tilted target has the KL-regularized optimum as a fixed point.Source: §2 · p. 3
  • Beyond RL, FFM shares a fixed-point structure with matching objectives for unnormalized densities and with adjoint matching.Source: §2 · p. 3

3 PRELIMINARIES

The preliminaries define policy improvement targets for maximum-entropy and behavior-constrained RL, then describe how flow matching represents a target policy. The standard flow-matching regression requires optimal-policy samples, which online RL does not provide.

3 PRELIMINARIES

  • Maximizing return alone can drive the policy toward a deterministic optimum that often suffers from premature convergence.Source: §3 · p. 3
  • Regularized policy improvement yields a Boltzmann target in maximum-entropy RL and a reference-policy-tilted target in behavior-constrained RL.The temperature λ trades off return against policy stochasticity or deviation from the reference, respectively.Source: §3 · p. 4
  • The target's normalization constant is intractable, and the target can be complex and multimodal, motivating fitting a tractable flow-based policy.Source: §3 · p. 4
  • Flow matching constructs a noisy interpolant between Gaussian noise and a clean target action, then learns the velocity field that transports the prior along its marginals.Integrating the learned velocity field produces terminal samples from the target policy.Source: §3 · p. 4
  • The standard flow-matching objective requires samples from the optimal policy, which are unavailable in online RL; the critic is available to form an alternative regression objective.Source: §3 · p. 4

4 FIXED-POINT TARGET FLOW MATCHING

Target flow matching uses critic action gradients to represent the Boltzmann target velocity, sharing flow matching's population minimizer. FFM replaces unavailable target-policy samples with current-policy samples and proves that the optimal policy is a fixed point of the iteration.

4 FIXED-POINT TARGET FLOW MATCHING

  • The target flow identity expresses the Boltzmann target velocity as a rescaled noisy action plus a conditional average of critic action gradients at clean actions.The setup assumes a differentiable critic and a finite target normalization constant.Source: §4 · pp. 4–5
  • The identity's 1/α(t) factor diverges near the pure-noise end, where critic gradients at clean actions carry little information about the noisy action.Mixing the target with the flow-matching target using a variance-optimal control variate removes this singularity.Source: §4 · p. 5
  • Target flow matching regresses onto critic-derived targets and shares its population minimizer with ordinary flow matching.Unlike the flow-matching target, which encodes where data moves, this target encodes where the critic points.Source: §4 · p. 5
  • The target-flow regression still requires optimal-policy samples, so FFM builds its interpolant from clean actions sampled from the current policy instead.The update iteratively uses this current-policy interpolant in the regression.Source: §4 · p. 5
  • Under regularity conditions on the critic, the exact population FFM iteration leaves an optimal policy unchanged, making it a fixed point.The theorem assumes the regression is solved exactly over all velocity fields.Source: §4 · p. 6
  • In practice, FFM samples actions with forward-only flow integration, computes critic action gradients, then noises actions and minimizes the regression loss.The action gradients can be stored in a replay buffer for sample reuse.Source: §4 · p. 6

5 APPLICATIONS IN REINFORCEMENT LEARNING

FFM is applied to offline-to-online fine-tuning and maximum-entropy online RL, with extensions for trust-region updates and correlated exploration.

Offline-to-online RL

  • Offline-to-online fine-tuning targets a behavior-tilted policy that keeps the pretrained flow as a reference while improving the critic value.FFM can obtain the reference policy’s score from its pretrained velocity field; the setting motivates staying near the data where offline critics are reliable.Source: §5 · p. 6

Maximum entropy reinforcement learning

  • For maximum-entropy RL, FFM estimates policy entropy from generated actions using instantaneous change of variables and Hutchinson’s trace estimator.The estimator uses one vector–Jacobian product per integration step and requires no auxiliary density network.Source: §5 · p. 6

Trust-region constraints

  • A trust-region loss constrains each fixed-point update to stay close to the previous flow policy, with the multiplier adjusted according to constraint violations.The method alternates gradient descent on policy parameters with gradient ascent on the multiplier.Source: §5 · p. 7

Correlated exploration

  • Because a flow policy maps prior draws deterministically to actions, correlated prior draws can produce sustained, directed exploration across environment steps.The stationary Ornstein–Uhlenbeck chain preserves each draw’s Gaussian marginal, so the policy distribution and fixed point remain unchanged; only visited states change.Source: §5 · p. 7

6 EXPERIMENTS

Across offline-to-online and online RL evaluations, FFM is competitive with baselines and shows particular advantages on hard long-horizon and high-dimensional tasks.

Evaluation scope

  • The experiments evaluate offline-to-online fine-tuning on 50 tasks and online on-policy training from scratch on 24 tasks.The offline-to-online setting uses OGBench domains; online RL uses MJX Playground and humanoid tasks.Source: §6 · p. 7

Offline-to-online RL

  • 0.95 mean vs. 0.93 for the next-best method at 1.5M steps; FFM trails TRQAM offline, 0.63 vs. 0.68, then achieves the best aggregate success.The offline-to-online comparison aggregates success over five tasks per domain; FFM has the widest margins on hard, long-horizon tasks, including HumanoidMaze-Large, 0.99 vs. 0.76.Source: Figure 2 · p. 8, §6 · p. 8

Online RL

  • On lower-dimensional Playground tasks FFM performs comparably to baselines, while on high-dimensional humanoid tasks it converges faster to the highest performance alongside TRUDI.The online evaluation covers 20 Playground tasks and four humanoid tasks, using IQM with 95% stratified bootstrap confidence intervals across 10 seeds per environment.Source: §6 · p. 8, Figure 3 · p. 9
  • FFM + OU converges faster to a return similar to the best-performing baseline DIME on Playground tasks.The aggregate curve combines FFM + OU on the subset of tasks with correlated-exploration runs and FFM on the remaining tasks.Source: §6 · pp. 8–9, Figure 3 · p. 9

7 CONCLUSION

The paper presents fixed-point flow matching as a data-free way to train flow policies by replacing data-derived flow-matching targets with critic gradients. It reports competitive or better performance across offline-to-online and online RL tasks, while requiring a differentiable target.

7 CONCLUSION

  • FFM replaces data-derived flow-matching targets with critic-gradient regression and uses a fixed-point iteration whose fixed point is the optimal policy.The procedure requires no gradients through the sampling chain, conversion to a stochastic differential equation, or adjoint simulation.Source: §7 · p. 9
  • Across 50 offline-to-online tasks and 24 online tasks, both FFM instantiations are competitive with or better than the strongest available baselines.The two instantiations are online maximum-entropy actor–critic and offline-to-online fine-tuning against a behavior-tilted target.Source: §7 · p. 9

Limitations

  • FFM requires a differentiable target, restricting its current scope to continuous action spaces and making policy updates depend on critic-gradient accuracy.Extensions to nondifferentiable targets and discrete action spaces are left for future work.Source: §7 · p. 9

A PROOFS

The proofs connect the target flow identity to regression objectives and establish the optimal policy as a fixed point under stated conditions. The argument also extends to behavior-tilted targets.

Proof of the target flow identity

  • The target flow identity follows by combining the velocity–score bridge with a target-score identity, which averages the target score at clean samples.For a Boltzmann target, its log-density gradient is the critic action gradient divided by λ; inserting the identity into the bridge yields the target velocity expression.Source: A PROOFS · p. 18

Proof of target flow matching

  • Target flow matching and ordinary flow matching share the same population minimizer because their least-squares targets have the same conditional expectation given the noisy action and time.The positive time-only weighting does not change this pointwise minimizer.Source: A PROOFS · p. 18

Proof of the fixed point

  • If the current policy is optimal and regression is solved exactly over all velocity fields, the next policy remains optimal and subsequent updates return the same velocity field.The theorem assumes a continuously differentiable critic and the stated integrability and positive-weight conditions.Source: A PROOFS · pp. 18–19

The behavior-tilted target

  • The fixed-point proof also applies to the behavior-tilted target, whose score combines the reference-policy score with the critic gradient divided by λ.The argument substitutes this target score for the critic gradient in the maximum-entropy proof.Source: A PROOFS · p. 19

B FURTHER METHOD DETAILS

The appendix fills in the interpolant and regression identities behind FFM, then details correlated exploration, related methods, and the maximum-entropy RL implementation. It also describes actor–critic updates, action bounding, and the online training procedure.

B.1 ONE-SIDED STOCHASTIC INTERPOLANTS

  • One-sided interpolants connect noise to clean actions, while a velocity–score bridge relates the fields and supports deterministic or stochastic samplers with shared marginals.The main text evaluates the reference score near, but not at, the clean endpoint.Source: §B.1 · pp. 19–20

B.2 GENERALIZED REGRESSION TARGETS AND CONTROL VARIATES

  • The appendix combines target-flow and flow-matching regression targets, preserving the Boltzmann policy as a fixed point while choosing a mixing schedule to reduce target variance.The two targets are conditionally unbiased; the variance-optimal schedule also avoids the endpoint singularity without time truncation.Source: §B.2 · pp. 20–21

B.3 CORRELATED EXPLORATION

  • Correlated exploration applies an Ornstein–Uhlenbeck chain to prior draws: each remains standard normal, while correlations decay exponentially with temporal distance.The limits recover independent draws as τou → 0 and a single draw held for an episode as τou → ∞.Source: §B.3 · pp. 21–22

C EXTENDED RELATED WORK

  • FFM belongs to iterative matching methods but obtains its velocity regression target from critic gradients at clean actions, without an adjoint ODE.Unlike action-target methods such as DIPO, its velocity target accounts for the interpolant’s noise level.Source: C EXTENDED RELATED WORK · p. 22

D MAXIMUM-ENTROPY REINFORCEMENT LEARNING

  • The appendix supplies implementation details for FFM’s online on-policy maximum-entropy RL setting, including critic and velocity-field optimization, action bounding, ablations, and per-task results.Source: D MAXIMUM-ENTROPY REINFORCEMENT LEARNING · p. 23

D.1 ACTOR-CRITIC BASED MAXIMUM-ENTROPY REINFORCEMENT LEARNING

  • Maximum-entropy actor–critic updates optimize the value function and velocity field; squashed clean actions are mapped to bounded environment actions, with critic gradients transformed by the chain rule.The experiments use the error-function squash; tanh performs slightly worse in the reported ablation.Source: D MAXIMUM-ENTROPY REINFORCEMENT LEARNING · p. 23, §D.1 · pp. 24–25

D.2 PSEUDOCODE

  • The online algorithm rolls out the policy, evaluates it with soft returns, updates the critic, then trains the actor by minimizing the FFM regression objective.It optionally correlates prior draws and includes trust-region and dual updates.Source: §D.2 · p. 26

D.3 EXPERIMENTAL DETAILS

The experimental details specify the online task suites, evaluation procedure, baselines, and FFM implementation. They also report schedule choices and a deterministic evaluation initialization that improved performance.

Environments and tasks

  • Experiments cover 20 DeepMind Control tasks and four high-dimensional MJX humanoid joystick tasks, comparing FFM with Gaussian-, diffusion-, and flow-policy baselines.DMC returns are bounded between 0 and 1000; humanoid returns are not normalized.Source: §D.3 · p. 26

Evaluation

  • Learning curves report per-task IQM returns with stratified bootstrapped 95% confidence intervals over 10 seeds, alongside aggregates across the Playground and Humanoid tasks.Source: §D.3 · p. 26

FFM implementation

  • FFM uses a geometric interpolant schedule, 16 Euler steps, error-function action squashing, and four Rademacher probes per integration node for entropy estimates.The implementation uses the variance-optimal mixed regression target with Fisher scale fixed to 1 and time weighting w(t) = 1/κ²(t).Source: §D.3 · pp. 26–27

Schedule and evaluation

  • The geometric schedule performed slightly better than the linear schedule, and evaluation from X0 = 0 improved performance; training rollouts still use stochastic prior sampling.All task suites use σ0 = 1 and r = 2.5.Source: §D.3 · p. 27

D.4 FURTHER EXPERIMENTS AND DESIGN ABLATIONS

The design ablations find that deterministic evaluation particularly benefits humanoid tasks, while integration-step count, interpolant schedule, and correlated exploration affect performance in different ways. The offline-to-online appendix describes behavior-constrained fine-tuning and its evaluation protocol across OGBench environments.

D.4 FURTHER EXPERIMENTS AND DESIGN ABLATIONS

  • The geometric schedule slightly improves aggregated humanoid results, while the Playground results show barely any difference; both schedules perform reasonably well on average.The geometric schedule is the default for online RL experiments.Source: Figure 6 · p. 28, §D.4 · p. 28
  • Humanoid performance rises with more Euler integration steps but plateaus at K = 16; tanh and erf action squashing perform on par.The experiments use K = 16 as the default and find either squashing function is a valid default.Source: §D.4 · pp. 28, 30, Figure 9 · p. 30
  • Correlated exploration can substantially improve selected Playground tasks, but no single τou setting performs best across all tasks.The reported gains include AcrobotSwingupSparse, HopperHop, and HopperStand; tuning τou requires sufficient compute.Source: §D.4 · pp. 31, 33, Figure 15 · p. 33
  • Deterministic evaluation significantly improves performance on all four humanoid tasks, with little aggregate difference on Playground tasks.The authors use deterministic evaluation throughout their experiments.Source: §D.4 · pp. 30–31, Figure 12 · p. 31, Figure 14 · p. 32

E.1 FURTHER DETAILS

  • Offline-to-online fine-tuning uses a behavior-tilted target and a trust region relative to the pretrained flow to keep the policy close to its reference.The target incorporates the reference policy’s terminal score, read from the pretrained velocity field.Source: §E.1 · p. 33

E.2 PSEUDOCODE

  • The fine-tuning procedure alternates pessimistic TD critic updates with behavior-tilted FFM actor regression, constrained by a trust region.The trust region is taken relative to the pretrained flow rather than the previous policy.Source: §E.2 · p. 34

E.3.1 ENVIRONMENTS AND TASKS

  • The OGBench evaluation covers ten domains split between goal-reaching locomotion and goal-conditioned object rearrangement, with five goals per domain.Each domain’s tasks share an offline dataset; each task fixes a goal for sparse reward and evaluation.Source: §E.3.1 · p. 34

E.3.2 OFFLINE-TO-ONLINE PROTOCOL

  • The protocol runs 1M offline gradient steps followed by 500K online environment steps after 300K behavior-cloning pretraining steps.Policies are evaluated every 50K steps over 50 episodes, and binary task-success rate is reported.Source: §E.3.2 · pp. 34–35

E.3.3 SHARED ARCHITECTURE AND HYPERPARAMETERS

The experiments use shared actor-critic architecture and mostly shared hyperparameters, with FFM’s per-domain settings organized into two regimes. The evaluation compares FFM with several named baselines under the same protocol.

E.3.3 SHARED ARCHITECTURE AND HYPERPARAMETERS

  • All methods use four-hidden-layer MLP actors and critics, with a pessimistic critic ensemble and domain-dependent discount and pessimism settings.The HumanoidMaze domains use (γ, ϱ) = (0.999, 0.0); other tasks use (0.995, 0.5).Source: §E.3.3 · p. 36

E.3.4 FFM HYPERPARAMETERS

  • FFM uses two coordinated per-domain regimes: fixed Fisher scale and 1/λ = 30 for sparse-reward locomotion, versus adaptive EMA normalization and 1/λ = 1 for manipulation.The reference score anchors both targets; locomotion retains the trust region, while manipulation switches it off.Source: §E.3.4 · pp. 36–37

E.3.5 BASELINES

  • FFM is compared with TRQAM, QAM, QAM-E, FQL, IFQL, DSRL, and CGQL-L under the identical protocol.Source: §E.3.5 · p. 37

E.4 FULL RESULTS

FFM’s offline-to-online average is best overall despite trailing the strongest offline baseline, with the largest per-environment gains on hard, long-horizon tasks and no divergence across seeds.

E.4 FULL RESULTS

  • FFM’s offline-to-online average is best overall despite trailing the strongest offline baseline, with the largest per-environment gains on hard, long-horizon tasks and no divergence across seeds.Figure 17 shows the per-environment, per-task success-rate grid across shaded offline and online phases.Source: §E.4 · p. 37, Figure 17 · p. 38
  • Figure 17 organizes offline-to-online success rates by environment rows and task columns, with offline and online phases shaded.Source: Figure 17 · p. 38
  • Figure 18 compares three Fisher normalizers against 1/λ and varies the number of Euler integration steps in its right-hand panels.The caption identifies offline success averaged over three seeds for one manipulation and one locomotion task.Source: Figure 18 · p. 39

E.5 DESIGN ABLATIONS

The ablations show that FFM’s behavior anchors and trust-region budget affect performance differently across task families and phases. They also show task-family differences in critic scaling and Euler integration, while the recipe settings were fixed across domains.

E.5 DESIGN ABLATIONS

  • The recipe and integration budget were fixed across ten domains; each ablation varies one setting on one manipulation task and one locomotion task.Unless stated otherwise, ablations reuse the pretrained flow in offline-only runs of 1M steps.Source: §E.5 · p. 39
  • Manipulation success falls from 0.40 to 0.00 as 1/λ rises; locomotion with fixed normalization rises from 0.83 to 0.95, while adaptive EMA falls from 0.72 to 0.14.The two normalizers coincide on manipulation because 1/λ = 1 applies no amplification to cancel.Source: §E.5 · p. 39
  • Adaptive-shape normalization raises locomotion success from 0.82 to 0.93 as 1/λ rises and matches the other variants on manipulation, but improves neither per-domain choice.It would remove the normalizer row from Table 4, but the authors retain the explicit split because it matches rather than improves on the per-domain choice.Source: §E.5 · p. 39
  • Two Euler steps yield 0.31 on manipulation, which saturates at ≈0.40 from five steps; locomotion scores 0.98 at two steps and 0.94 at 25.The ten steps inherited from the TRQAM/QAM protocol lie on the plateau for both task families.Source: §E.5 · p. 39
  • Removing both behavior anchors collapses offline success to 0.00 on manipulation and 0.19 on locomotion; either anchor recovers performance.At online εTR = 8, product tilt scores 0.99 on cube-double while pure tilt falls to 0.05; all locomotion variants reach ≈1.Source: §E.5 · p. 39, Figure 19 · p. 40
  • Product tilt stays ≥0.99 online across four tasks for every εTR ≥4; offline puzzle-3x3 rises from 0.19 at εTR = 1 to 0.99 at 8.Offline, a loose budget of 16 is equivalent to no trust region on puzzle-3x3: 0.99 versus 1.00.Source: §E.5 · pp. 39–40, Figure 20 · p. 40
  • Pure-tilt budgets are task-dependent: online cube-double falls from 1.0 at εTR = 1 to 0.05 at 8, while puzzle-3x3 rises from 0.23 to 1.0.Offline, cube-double and scene perform best at εTR = 1, while puzzle-3x3 needs at least 4; antmaze-large is insensitive.Source: §E.5 · p. 39, Figure 20 · p. 40

Related papers

Loading 2610.12848v1…