PaperlayerView exact revision on arXiv

Source-linked AI summary

QF3: Fast Flow RL with Filtered Q-Gradients

Chung Min Kim, Brent Yi, David McAllister, Hongsuk Choi, Himanshu Gaurav Singh, Jinkun Cao, Ken Goldberg, Pieter Abbeel, Carmelo Sferrazza, Angjoo Kanazawa

arXiv:2610.08789v1cs.ROcs.LG

TL;DR

Flow policies are widely used for robot learning, but reinforcement learning is needed both when expert trajectories are unavailable and when pretrained behavior needs improvement. QF3 trains flow policies off-policy with flow matching and critic-guided one-step predictions, filtering critic gradients near replay actions. Its evaluations include humanoid policies trained from scratch and transferred zero-shot to hardware, as well as manipulation fine-tuning.

  • Problem

    Flow policies need reinforcement learning when expert trajectories are unavailable or pretrained behavior needs improvement, while differentiating through every flow-sampling step is costly and can be unstable.

    Source: §1 · p. 2

  • Method

    QF3 combines replay-based flow matching with critic maximization through a one-step action prediction and filters critic gradients when velocity corrections move too far from replay-conditioned targets.

    Source: §1 · p. 2

  • Results

    Humanoid policies train approximately 10× faster than FPO++ and transfer zero-shot to hardware; manipulation fine-tuning improves ABC-Sim bottle gathering and performs similarly or better than OGPO on three Robomimic tasks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    QF3 can train robot policies from scratch and fine-tune pretrained policies across humanoid locomotion and manipulation evaluations.

    Source: §1 · p. 2

Abstract

from arXiv · show

Flow policies have become a standard policy class for learning robot behaviors from demonstrations, but reinforcement learning is still critical for improving pre-trained flow policies or learning them from scratch through interaction. We introduce QF3 (Fast Flow RL with Filtered Q-Gradients), an online off-policy RL algorithm that trains a flow policy with flow matching plus the critic's action gradient, backpropagated through a one-step prediction of the flow's output. To keep updates where the critic and this prediction are reliable, QF3 applies the critic gradient only to action dimensions that stay near the replay action. To our knowledge, QF3 is the first off-policy flow RL method to train humanoid locomotion policies from scratch and transfer them zero-shot to hardware. Paired with a high-throughput off-policy training recipe, it trains humanoid locomotion and motion-tracking policies with a 10x wall-clock speedup over FPO++, a recent on-policy flow RL method. We further apply QF3 to fine-tune pretrained flow-based manipulation policies on both ABC-Sim and Robomimic tasks. These results suggest that QF3 can both learn robot policies from scratch and refine those acquired from demonstrations. Website: https://qf3-rl.github.io/

1 Introduction

QF3 directly trains flow policies through online off-policy reinforcement learning, addressing both learning from rewards without expert trajectories and improving pretrained policies through interaction. It combines flow matching with critic-guided one-step action prediction and filters critic updates near replay actions; evaluations span MuJoCo, humanoid control, and manipulation.

Motivation

  • Off-policy flow RL faces a sampler-backpropagation trade-off: differentiating through every sampling step is costly and potentially unstable, while alternatives may freeze the flow or use critic values or regression targets.Reinforcement learning can also address tasks without expert trajectories and improve pretrained behavior through interaction.Source: §1 · p. 2

QF3

  • QF3 combines flow matching with critic maximization through a one-step action prediction, avoiding integration or differentiation through the flow ODE.It is designed to fit existing TD3-style off-policy training recipes and large-scale parallel simulation.Source: §1 · p. 2
  • Velocity-space clipping creates a replay-local trust region: dimensions whose critic-induced correction moves too far from the replay-conditioned target receive no critic gradient.The critic gradient is applied near replay actions, where the critic is trained and the one-step prediction is accurate.Source: §1 · p. 2

Algorithm and analysis

  • On MuJoCo, a single clipped-gradient update without sampler backpropagation performs competitively with recent off-policy flow RL methods.The paper also evaluates QF3 for training policies from scratch and fine-tuning pretrained policies.Source: §1 · p. 2

Scaling to high-dimensional, dynamic control from scratch

  • The 29-DoF Unitree G1 velocity- and motion-tracking policies transfer zero-shot to hardware and train about 10× faster than FPO++, an on-policy flow RL method.The work describes this as the first off-policy flow RL method shown to train a humanoid policy from scratch and transfer it zero-shot.Source: §1 · p. 2

Manipulation fine-tuning

  • On ABC-Sim bottle gathering, fine-tuning makes episodes 22% faster and raises success from 91% to 95%; on three Robomimic tasks, QF3 performs similarly or better than OGPO.ABC-Sim fine-tunes the ABC-VLA flow action head with LoRA; Robomimic fine-tunes the full policy.Source: §1 · p. 2

2 Related Work

Related work covers generative flow and diffusion policies, methods for optimizing them with reinforcement learning, and approaches to stabilizing off-policy Q-learning. QF3 directly trains a flow actor from replay and limits critic queries near replay-supported actions.

2.1 Diffusion and Flow Models for Robot Policies

  • Flow and diffusion policies have expanded from supervised robot learning into reinforcement learning, where prior methods freeze the generative policy or optimize it with on-policy policy gradients.Prior online on-policy methods are described as sample inefficient.Source: §2.1 · pp. 2–3

2.2 Off-Policy RL with Diffusion and Flow Actors

  • Off-policy generative-policy methods differentiate through denoising, construct action or velocity regression targets, or use PPO-style objectives; QF3 instead applies the critic action gradient to a replay-based one-step prediction.Unlike FlowRL, QF3 clips the critic-induced correction in velocity space around the replay-conditioned target, avoiding a tanh-induced mismatch with the flow-matching target.Source: §2.2 · p. 3

2.3 Stabilizing Off-Policy Q-Learning

  • Off-policy Q-learning can draw actors toward actions where critic estimates are unreliable; QF3 uses a TD3-style actor step through a one-step flow prediction and can leverage standard stabilization techniques.Its critic needs sampled flow actions, but not log-likelihoods or entropy estimates.Source: §2.3 · pp. 3–4

3 Algorithm

QF3 trains a flow actor from replay by combining flow matching with critic-guided improvement through a one-step action prediction. Its velocity-space trust region filters critic gradients when policy predictions move too far from replay-conditioned targets.

3 Algorithm

  • QF3 uses TD3-style replay storage and Bellman-trained critics while representing the actor as a velocity field whose actions are sampled by integrating the flow.The algorithm approximates the critic gradient through one Euler step instead of differentiating through the full sampler.Source: §3 · p. 4

3.1 QF3 Actor Objective

  • The actor objective combines conditional flow matching with a critic term for policy improvement.The objective is evaluated using replay transitions and a one-step prediction rather than integrating or differentiating through the full flow sampler.Source: §3.1 · p. 4, §3 · p. 4

Anchoring the flow with the replay buffer.

  • Flow matching trains the actor to reproduce the replay distribution, anchoring it toward the behavior policy and actions where critic estimates are more reliable.The actor can then use critic gradients to steer toward higher-value actions that enter the evolving replay buffer.Source: Anchoring the flow with the replay… · p. 4

Critic-driven policy improvement.

  • A single Euler step approximates the policy action for critic maximization, but an unconstrained prediction can leave replay support and chase overestimated Q-values.The one-step approximation is exact when the predicted velocity equals the replay-conditioned velocity.Source: Critic-driven policy improvement · pp. 4–5

3.2 Filtered Q-Gradients as a Trust Region

  • Velocity clipping keeps the critic query near the replay action; dimensions outside the trust region lose the critic gradient while flow matching still pulls predictions toward the replay-conditioned target.The critic gradient resumes for a component if its policy-buffer gap returns within the clipping bounds; the trust region is compared to PPO's ratio clip.Source: Figure 2 · p. 5, §3.2 · p. 6

4 Analyzing Filtered Q-Gradients

The velocity clip filters critic updates to stabilize QF3, including after the critic improves; on MuJoCo-v4, QF3 performs competitively with recent off-policy flow RL methods. The analysis also finds that action-space clipping can produce a sharply bending flow field near the end of the trajectory.

4.1 Analyzing the velocity clip

  • Without velocity clipping, Direct Q collapses on Humanoid-v4 even at a 10× lower actor learning rate, while the default QF3 clip recovers a usable policy.The unclamped update also drops a pretrained manipulation policy from task-solving to near-zero success within the first 100k steps before recovery.Source: §4.1 · p. 6, Figure 3 · p. 7
  • A fixed action-space bound makes the velocity correction unbounded as τ→1, producing a sharply bending flow field that velocity-space clipping avoids.The fixed bound α corresponds to a velocity bound of α/(1 − τ).Source: §4.1 · p. 6, Figure 3 · p. 7
  • The clip continues filtering critic corrections after the policy is performant: within-clip action dimensions are 74/83/85% on Ant and 81/83/82% on Humanoid at 50k/250k/500k steps.The authors interpret this persistent filtering as a trust region throughout training, not only as protection from an unreliable early critic.Source: §4.1 · pp. 6–7

4.2 Validation on MuJoCo-v4

  • QF3 performs competitively on Hopper, Walker2d, Ant, and Humanoid with one velocity evaluation per actor update, while DIPO uses 10–40 action-gradient steps and FlowRL trains an extra value function.QSM fails on Ant and lags on Walker2d; one QF3 setting reaches at least 79% of its per-task best return on all four tasks.Source: §4.2 · p. 7

5 Experimental Results

QF3 trains humanoid policies from scratch and fine-tunes pretrained manipulation policies. It is competitive with FastTD3, converges about 10× faster than FPO++ in wall-clock time, transfers to hardware zero-shot, and improves results on ABC-Sim and Robomimic tasks.

5.1 Evaluating QF3 for humanoid locomotion

  • QF3 trains velocity- and motion-tracking policies from scratch for the 29-DoF Unitree G1 and transfers them zero-shot to hardware.The reported hardware policies follow velocity commands and track long, dynamic motions without real-world fine-tuning.Source: §5 · p. 7, §5.1 · p. 9
  • The humanoid training setup replaces FastTD3’s deterministic actor with QF3’s flow actor while retaining the replay buffer, critic, target-update structure, and remaining hyperparameters.At deployment, the flow is integrated deterministically from ε=0 with 5 Euler steps.Source: §5.1 · p. 8
  • QF3 succeeds in humanoid locomotion and motion tracking, trains about 10× faster in wall-clock time than FPO++, and is competitive with FastTD3.Velocity tracking finishes within ∼4% of FastTD3’s final reward under the same 10-hour budget; QF3 reaches full-length motion-tracking episodes in roughly half an hour.Source: §5.1 · p. 8, Figure 5 · p. 8

5.2 Fine-tuning pretrained manipulation policies

  • For ABC-Sim, QF3 fine-tunes rank-4 LoRA adapters on the ABC-VLA flow head; for Robomimic, it fine-tunes the full pretrained policy.Both settings add a flow-matching loss toward the frozen base policy, which the authors find important for stable fine-tuning.Source: ABC-Sim · p. 9, §5.2 · p. 9, Robomimic · p. 10

ABC-Sim

  • QF3 fine-tuning improves the pretrained ABC-VLA policy on both bimanual tasks, with Bottles episodes 22% faster and success rising from 91% to 95%.On Dishrack, success and successful-episode length improve versus the base policy, with performance comparable to residual RL; changes include fewer extra grasps and fewer plates falling from the rack.Source: ABC-Sim · pp. 9–10, Figure 6 · p. 9

Robomimic

  • On three Robomimic tasks, QF3+ matches or exceeds OGPO+CA in final success rates, and QF3 reaches 96% success on Tool Hang and Transport without the success-buffer regularizer.The authors first train the critic alone on base-policy rollouts because QF3 is more sensitive to an untrained critic than OGPO’s clipped policy-gradient update.Source: Robomimic · p. 10, Figure 7 · p. 10

6 Conclusion

QF3 is an online off-policy algorithm that trains flow policies by combining flow matching with critic-driven improvement through a one-step action prediction. It supports both learning robot behaviors from scratch and fine-tuning demonstration-acquired policies.

6 Conclusion

  • QF3 combines flow matching with critic-driven improvement through a one-step action prediction, without integrating or differentiating through the flow ODE.The critic gradient is clipped around the replay-conditioned flow target.Source: §6 · p. 11
  • With FastTD3, QF3 trains Unitree G1 velocity- and motion-tracking policies at speeds close to FastTD3 and approximately 10x faster than FPO++, with zero-shot hardware transfer.Source: §6 · p. 11
  • QF3 fine-tunes a pretrained ABC-VLA policy on two bimanual simulation tasks and full policies on three Robomimic tasks, with over 95% success on Robomimic.On ABC-VLA tasks, it improves success rate and successful-episode duration over the base policy.Source: §6 · p. 11

A.1 QF3 Implementation Details for Training from Scratch

This appendix describes the flow actor and critic used for training from scratch in MuJoCo-v4 and humanoid locomotion; fine-tuning retains each pretrained policy's architecture. The implementations vary by setting while sharing the flow parameterization and Euler-step count.

A.1 QF3 Implementation Details for Training from Scratch

  • The from-scratch implementations cover MuJoCo-v4 and humanoid locomotion, while fine-tuning retains the pretrained policy architecture.Source: §A.1 · p. 17
  • The actor is an MLP that maps observation, noisy action, and time embedding to a velocity; its flow-time embedding is fixed, sinusoidal, and 64-dimensional.Source: §A.1 · p. 17
  • Rollouts use five uniform forward-Euler steps from Gaussian noise, taking the final state as the normalized action.Source: §A.1 · p. 17
  • The settings differ in network width, layer normalization, and critic head, but share the flow parameterization, time embedding, activation, and Euler-step count.Source: §A.1 · p. 17, Table A.1 · p. 17
  • MuJoCo-v4 uses a standard TD3 twin-Q critic, whereas humanoid locomotion uses a FastTD3 distributional critic with 101 atoms over [-20, 20].The actor-update Q estimate is the mean of the predicted distribution.Source: §A.1 · p. 17

A.2 Velocity Clipping as an ELBO-Surrogate Trust Region

QF3's velocity clip is interpreted as a trust region on an ELBO surrogate of the replay action. Clipping the velocity residual bounds the one-step action deviation, whereas a uniform action-space bound can induce flow velocities with sharply different magnitudes across flow times.

A.2 Velocity Clipping as an ELBO-Surrogate Trust Region

  • The likelihood connection uses the negative-ELBO interpretation of weighted denoising loss, which holds for common noise schedules including QF3's linear schedule.Source: §A.2 · p. 18
  • The clipped proposal's bound is an ELBO surrogate for replay actions, not a bound on the policy's own velocity or replay-action likelihood.The CFM loss still trains the unclipped velocity toward the replay-conditioned target, even where saturated coordinates receive no Q-gradient.Source: §A.2 · p. 18
  • QF3 evaluates the critic at a one-step reconstruction from a noised replay action using a coordinate-wise clipped velocity residual.Source: §A.2 · p. 18
  • Clipping the velocity residual makes the one-step endpoint deviation shrink linearly with 1 − τ; a uniform action-space bound instead scales the velocity bound by 1/(1 − τ).As τ approaches 1, that bound grows without limit and can push velocities toward sharply different magnitudes across τ, producing a high-curvature flow field.Source: §A.2 · p. 18

A.3 MuJoCo-v4 Settings

The MuJoCo-v4 comparisons use a shared TD3 harness, flow actor and critic, tasks, and evaluation protocol, while tuning method-specific settings per task. The tested methods include Q-Score Matching, FlowRL, DIPO, and QF3.

A.3 MuJoCo-v4 Settings

  • Each method shares a CleanRL-based TD3 harness, actor and critic, MuJoCo-v4 tasks, and evaluation protocol; runs use 500k environment steps, three seeds, and five deterministic evaluation episodes every 10k steps.Configurations are selected by mean return over the final 50k training steps, averaged over seeds.Source: §A.3 · p. 19
  • Q-Score Matching regresses the velocity field toward a scaled critic action-gradient at noised replay actions, with its score-loss weight tuned by task.Source: §A.3 · p. 19
  • FlowRL uses a Wasserstein-2 penalty toward high-value replay actions, while DIPO uses critic-gradient action improvement followed by actor refitting; their respective penalty and gradient settings are tuned.Source: §A.3 · p. 19
  • QF3's clip radius and flow-matching weight are tuned over the listed settings, with returns reported for each combination.Source: §A.3 · p. 19

A.4 Holosoma Humanoid Locomotion Settings

The locomotion experiments use Holosoma’s GPU-parallel Unitree G1 tasks and place QF3 in the FastTD3 training loop. The section specifies task metrics, shared settings, and baseline configurations.

Simulator and training stack

  • QF3 replaces the actor in Holosoma’s FastTD3 loop, sharing its replay buffer, distributional twin-Q critic, target updates, and parallel rollouts.The FastTD3 comparison uses Holosoma’s tuned configuration unchanged.Source: §A.4 · p. 19

Hyperparameter sensitivity

  • Figure A.1 reports MuJoCo-v4 tail-50k evaluation return as across-seed means with ±1σ error bars over three seeds.The caption does not provide hyperparameter-sensitivity outcomes.Source: Figure A.1 · p. 20

Tasks and command distribution

  • Single-motion tracking uses the LAFAN dance clip and reports mean episode length capped at 500 steps; velocity tracking reports per-step raw_lin_vel reward.Velocity commands are planar base velocities resampled each episode from Holosoma’s default distribution.Source: §A.4 · p. 20

QF3 hyperparameters

  • QF3 and FastTD3 share core training hyperparameters; QF3 uses task-specific α and λ_cfm values and integrates the flow with five Euler steps.For velocity tracking α=2 and λ_cfm=0.3; for motion tracking α=0.5 and λ_cfm=0.01.Source: §A.4 · p. 20

Baselines

  • FPO++ uses its released configuration and is compared on the same GPU type per task; FlowRL uses the FastTD3 recipe but runs on different GPUs across tasks.The velocity wall-clock panel omits FlowRL because its run used an L4 rather than an A10G.Source: §A.4 · p. 20

A.5 ABC Sim Manipulation Fine-tuning Settings

The appendices detail QF3’s manipulation fine-tuning setups, diagnose ABC-Sim speedups, and extend the actor update to text-to-image fine-tuning. The reported results cover dish loading, bottle gathering, and PickScore training.

A.5 ABC Sim Manipulation Fine-tuning Settings

  • ABC-Sim fine-tunes the vision-conditioned ABC-VLA flow head with rank-4 LoRA adapters while keeping its VLM and base head frozen; QF3 clips policy-buffer gaps and anchors updates to replay and base velocities.The adapters total 229k trainable parameters; the setups use base-policy replay warm-up and task-specific evaluation.Source: §A.5 · p. 21

A.6 Robomimic Fine-tuning Settings

  • Robomimic starts all methods from a behavior-cloned MLP flow checkpoint, seeds replay with base-policy rollouts, and compares QF3 variants with OGPO+CA, DSRL, and EXPO.Evaluation uses critic-selected best-of-8 samples for QF3, QF3+, and OGPO+CA; a separate figure reports deterministic single-sample ODE evaluation.Source: §A.6 · pp. 22–23, Figure A.2 · p. 23

A.7.1 Load Plates in Dishrack

  • 48%→72% success and 90→63 s mean episode length: QF3 improves Dishrack results, with about three quarters of the time reduction from seeds the base policy fails but QF3 solves.Among seeds, 21 succeed under both policies, 3 only under the base policy, and 11 under neither.Source: §A.7.1 · p. 23
  • QF3’s first-plate handover moves directly from right to left in 49/50 episodes, while the base policy typically uses a left-right-left sequence in 41/50.The base policy already uses the direct handover on four seeds at nearly the same speed.Source: §A.7.1 · p. 23
  • For Dishrack, QF3 loses 8% of placed plates versus 21% for the base policy, and its carry-and-insert takes 7.3 versus 12.0 seconds when successful.The base policy often leaves plates insufficiently seated, while QF3 learns to seat them nearly upright.Source: §A.7.1 · pp. 23, 25, Figure A.5 · p. 24

A.7.2 Throw Bottles in Bin

  • 86%→96% success and 26.9→21.1 s mean episode length: QF3 improves bottle-bin results, with about three quarters of the reduction coming from episodes both policies solve.Typical bottle handling takes 1–2 seconds less; bottles are held for 1.8 versus 2.5 seconds.Source: §A.7.2 · p. 25

A.8 Text-to-Image Fine-tuning

  • Text-to-image fine-tuning treats the prompt as observation, final image latent as action, and PickScore reward model as the critic, applying QF3 with a clipped one-step endpoint.The actor update also includes flow-matching and base-policy anchors.Source: §A.8 · p. 25
  • 19.72→22.57: mean PickScore of training samples rises during fine-tuning Stable Diffusion 3.5 Medium.Across replay configurations, qualitative trajectories differ despite similar training PickScore: recency-weighted replay retains framing variation with more consistent appearance.Source: §A.8 · p. 25, Figure A.6 · p. 26, Figure A.7 · p. 26
Loading 2610.08789v1…