PaperlayerView exact revision on arXiv

Source-linked AI summary

ArcFlow: Training Flow Policies for Multiple Solver Budgets

Hieu Tran, Hong Yu

arXiv:2610.28788v1cs.LGcs.AI

TL;DR

Fixed-budget reinforcement-learning training can leave low-step deployment performance uncertain and may erode useful one-step behavior. ARCFLOW trains and selects one flow policy across specified solver budgets, using action-aligned supervision and budget exposure. Across three control domains, it leads five flow-policy baselines in mean one-step return on 25 of 30 tasks.

  • Problem

    Fixed-budget reinforcement-learning training does not ensure strong return at cheaper deployment budgets, and updates can erode useful one-step behavior already present in a pretrained policy.

    Source: §1 · p. 1

  • Method

    ARCFLOW trains one flow field across solver budgets using action-aligned supervision, budget exposure, and worst-budget checkpoint selection.

    Source: §1 · p. 2

  • Results

    Across DM Control, manipulation, and locomotion, ARCFLOW leads five flow-policy baselines in mean one-step return on 25 of 30 tasks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The results support training and evaluating generative policies across their deployment range while retaining one architecture and paying only for the selected solver budget at deployment.

    Source: §7 · p. 9

    The evaluation covers simulated control without physical-robot experiments, and selection protects only evaluated solver budgets, not unseen counts.

    Source: ETHICS STATEMENT · p. 10, A LIMITATIONS AND SCOPE · p. 15

Abstract

from arXiv · show

Flow-matching policies adjust inference cost through the number of solver steps, but reinforcement learning at a single fixed solver budget can hide degradation at cheaper deployment budgets and even erase useful one-step behavior present in a pretrained policy. We introduce Action-Ratio-Clip Flow (ArcFlow), a deployment-aware algorithm for training one flow policy across a specified range of solver budgets. ArcFlow combines action-aligned supervision, budget exposure, and worst-budget checkpoint selection. For reconstruction objectives, its bridge aggregates temporal probes into one surrogate ratio and one PPO-style clip per action, with a boundary residual exactly equal to deployed one-step reconstruction error. Cycling exposes the shared field to the deployment range, and selection optimizes its performance floor. For native velocity objectives, multi-budget endpoint supervision applies the same principle without replacing the native update. Both realizations preserve the architecture and deployment-time cost. Across DM Control, manipulation, and locomotion, ArcFlow attains the highest mean one-step return among five flow-policy baselines on 25 of 30 tasks under each method's reported training settings. Component studies identify complementary improvements from boundary supervision, the fixed-grid bridge, budget exposure, and deployment selection. These results establish deployment-budget-aware training as an effective route to strong low-step control with one policy serving multiple inference costs.

1 INTRODUCTION

Flow policies offer adjustable inference cost, but fixed-budget reinforcement-learning training does not ensure strong low-step deployment and can erode useful one-step behavior. ARCFLOW trains and selects one field across solver budgets; across three control domains, it leads five flow-policy baselines in mean one-step return on 25 of 30 tasks.

1 INTRODUCTION

  • Flow matching adjusts inference cost through solver steps, but high return at the training budget does not assure high one-step return and may erode useful prior behavior.Fewer solver steps reduce network evaluations, which can help policies meet latency budgets.Source: §1 · p. 1
  • Reconstruction objectives can suppress the one-step boundary signal, while different solver budgets induce different action maps that single-budget updates need not improve uniformly.The cited FPO noise-reconstruction loss weights velocity error by (1 − t)^2, giving zero weight to the boundary queried by a one-step solver.Source: §1 · pp. 1–2
  • ARCFLOW combines action-aligned supervision, budget exposure, and worst-budget checkpoint selection to train one flow field across a specified solver-budget range.Its native-velocity realization applies multi-budget endpoint losses and an anchored max–min update without replacing the original objective.Source: §1 · p. 2
  • Across DM Control, manipulation, and locomotion, ARCFLOW leads five flow-policy baselines in mean one-step return on 25 of 30 tasks.The evidence combines a controlled common-stack benchmark with a released manipulation pipeline and component studies.Source: §1 · p. 2

2 PROBLEM FORMULATION

The formulation treats a policy as one velocity field whose actions are produced by a numerical solver, with solver budget controlling the deployed action map. The goal is one field that remains useful across specified deployment budgets.

2 PROBLEM FORMULATION

  • A numerical solver with K steps maps source noise and state through the velocity field to produce the action observed by the environment.Solver queries are internal computations, not environment transitions.Source: §2 · p. 2
  • The objective is to learn one field useful across specified deployment budgets without supplying K as an input to that field.Source: §2 · p. 2
  • Training-surrogate improvement or higher return at the training budget does not by itself imply better return at another deployment budget.Source: §2 · p. 2
  • A one-step collapse is a task whose specified one-to-sixteen-step return statistic falls below one half; the threshold depends on the seed pool and aggregation rule.The paper reports the graded return ratio alongside the threshold count.Source: §2 · p. 3

3 SOLVER-BUDGET MISMATCH

Across the 30-task census, ARCFLOW has the highest mean normalized return at every evaluated solver budget, with the greatest separation at one-step deployment. The section also distinguishes absolute return from within-policy cross-budget retention and links one-step degradation to boundary supervision.

3 SOLVER-BUDGET MISMATCH

  • ARCFLOW has the highest mean normalized return at every evaluated solver budget, with no marked one-step collapses among 30 tasks.Collapse counts are 0/30 for ARCFLOW, 2/30 for FPO, 4/30 for FPO++, 0/30 for DPPO, 3/30 for ReinFlow, and 0/30 for PolicyFlow; the mean-return separation is largest at one step.Source: §3 · p. 3
  • In the cited reconstruction formulation, the loss and actor gradient vanish at t = 1, the node queried by a one-step reverse Euler solver.Nearby supervision is quadratically down-weighted, while velocity supervision removes this factor; the ablations attribute most one-step recovery to this change.Source: §3 · p. 3
  • The collapse count and mean return capture different aspects: zero marked collapses alone does not indicate an effective low-latency policy.DPPO and PolicyFlow have no marked collapses, yet their mean returns remain below ARCFLOW at every displayed budget.Source: §3 · p. 3

4 THE ARCFLOW ALGORITHM

ARCFLOW trains one shared field across solver budgets using cycling rollouts, a boundary-aware fixed-grid bridge for reconstruction objectives, and worst-budget checkpoint selection. A native-velocity realization instead adds multi-budget endpoint supervision while retaining the native actor objective.

4.1 BUDGET-CYCLED ROLLOUTS

  • Rollouts cycle through eight solver budgets in increasing order, exposing the shared field equally to each supported budget over a cycle.The field is not conditioned on K.Source: §4.1 · p. 4

4.2 BOUNDARY-AWARE BRIDGE

  • At the bridge boundary, the residual equals one-step action-reconstruction error, so the fixed-grid probes include an exact one-step reconstruction signal.The identity links one quarter of the sampled-node distribution to the reconstruction error of a one-step action.Source: §4.2 · p. 4
  • The bridge realization scores collected actions using fresh noise and a fixed-grid reference path; averaged probes yield one energy-based surrogate weight and PPO-style clip per action.That weight is not the density ratio of the executed solver policy.Source: §4.2 · p. 4

4.3 TRAINING AND SELECTION

  • Worst-budget selection retains the checkpoint with the highest minimum validation return over K = {1, 4, 16}.Taking the minimum prevents a high validation return at one evaluated budget from compensating for a low return at another.Source: §4.3 · p. 5, §4.2 · p. 5
  • The native-objective realization retains the native actor objective and adds multi-budget endpoint supervision, while both realizations deploy one field.In the released ReinFlow pipeline, it collects at K = 4 and supervises endpoint views at V = {1, 2, 4, 8, 16}.Source: §4.3 · pp. 5–6

5 EXPERIMENTS

Across the reported settings, ARCFLOW combines broad one-step gains with performance across solver budgets, while component studies and a native-velocity pipeline test its mechanisms and scope.

5.1 EXPERIMENTAL SETUP

  • The benchmark uses a common control stack, with five flow-policy baselines sharing the normalizer, architecture, initialization, critic, and interaction budget.It covers 21 DM Control tasks, 64 outer iterations, and 62.9 million interactions; separate component studies use five matched training seeds.Source: §5.1 · p. 6

5.2 MULTI-BUDGET PERFORMANCE

  • At K = 1, ARCFLOW has the highest displayed mean on 25 of 30 tasks, including all three manipulation tasks and 18 of 21 DM Control tasks.On 12 of the 25 wins, the displayed mean improves on the strongest baseline by at least 10%.Source: §5.2 · p. 6
  • ARCFLOW leads all methods in mean normalized return at every evaluated solver count, with its mean nearly constant from two through sixteen steps.It is best at all three budgets on the three manipulation tasks; across the full table it leads on 13 tasks at K = 4 and 14 at K = 16.Source: §5.2 · pp. 7–8

5.4 CONTROL COMPONENT ANALYSIS

  • On the 25-task component pool, direct velocity supervision raises normalized one-step return from 0.350 to 0.534, and adding the fixed-scale bridge at K = 4 raises it to 0.612.Cycling over {1, 4, 16} raises worst-budget return to 0.635; broader support reaches 0.668 at one step, and full ARCFLOW reaches 0.711 at one step and 0.685 at its weakest budget. The fixed-scale bridge row has the highest training AUC, 0.700.Source: Table 2 · p. 8, §5.4 · p. 8

5.5 NATIVE-PIPELINE VALIDATION

  • In the released ReinFlow manipulation pipeline, ARCFLOW scores between 91.1% and 97.1% across five budgets, with the highest worst-budget score (91.1%) and mean score (94.7%).Official fine-tuning reaches 92.5% at four steps but 1.5% at one step; the one-step specialist reaches 98.0% at one step but has a worst-budget score of 74.6%.Source: Table 3 · p. 8, §5.5 · pp. 8–9

5.3 TRAINING DYNAMICS

  • On many tasks, one-step gains emerge by roughly mid-training; other tasks show stable early gains, continued improvement, or selected checkpoints above the preceding trajectory.Early separation appears on CheetahRun, FingerSpin, HopperHop, HopperStand, Walker tasks, and manipulation tasks; several other tasks improve after mid-training.Source: §5.3 · p. 8

6 RELATED WORK

Related work spans generative transport and control, reinforcement learning for generative policies, and accelerated or one-step samplers. ARCFLOW differs by training one RL flow field for a declared set of deployment budgets.

Generative transport and control

  • Generative transport methods learn noise-to-data transformations, while control applications include planning, return conditioning, and visuomotor imitation.Flow matching learns velocity fields directly; AdaFlow adapts integration to action variance, and FlowPolicy learns one-step consistency.Source: §6 · p. 9

Reinforcement learning of generative policies

  • RL methods for generative policies include denoising-chain policy gradients, flow-loss PPO-style ratios, stochastic flow transitions, and velocity- or transition-based objectives.Offline and critic-based approaches form another related group; ARCFLOW focuses on the return floor of one policy across solver budgets.Source: §6 · p. 9

Few-step and one-step policies

  • Accelerated samplers and one-step policies target faster generation, whereas ARCFLOW measures one RL-trained field at every declared solver budget.Prior approaches include numerical acceleration, distillation, consistency, shortcut, and one-step RL methods.Source: §6 · p. 9

7 CONCLUSION

ARCFLOW aligns flow-policy learning with deployment budgets through boundary supervision, budget exposure, and worst-budget selection. The conclusion reports gains in one-step performance across three control domains while retaining competitive higher-budget returns and unchanged deployment architecture and per-step solver cost.

7 CONCLUSION

  • ARCFLOW improves one-step performance across three control domains while retaining competitive higher-budget returns and unchanged deployment architecture and per-step solver cost.The native ReinFlow study extends the principle to a released image-policy pipeline and five budgets; component studies separate gains from supervision, exposure, and selection.Source: §7 · p. 9

AI USE STATEMENT

  • The authors report using generative AI across conceptual, mathematical, experimental, software, interpretive, and manuscript work, while taking responsibility for the final claims and artifacts.AI-generated outputs were not treated as empirical evidence, and AI was not used to generate or clean experimental datasets.Source: AI USE STATEMENT · p. 10

ETHICS STATEMENT

  • The evaluation concerns simulated control and includes no human participants, personal data, or physical-robot experiments.The authors note that improved optimization does not itself provide safety guarantees; physical or safety-critical use would require independently validated objectives, constraints, monitoring, and fail-safes.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

  • The anonymous repository provides the ARCFLOW implementation and experiment configurations.The paper and appendices specify algorithms, solver budgets, checkpoint selection, evaluation conventions, and the matched five-seed manipulation protocol.Source: REPRODUCIBILITY STATEMENT · p. 10

A LIMITATIONS AND SCOPE

The evidence is limited to simulated control, controlled baseline adaptations, and the solver budgets and objectives evaluated. Training and selection add compute, while the theory does not establish monotone return improvement or generalize to several untested settings.

Empirical scope

  • The evidence is limited to simulated DM Control, Panda manipulation, locomotion, and one native image-policy manipulation task, not real-robot robustness or larger policy and action spaces.The native study uses endpoint supervision rather than the bridge objective, so it supports transfer of the principle but is not an independent test of the same update rule.Source: A LIMITATIONS AND SCOPE · p. 15

Comparison and statistical scope

  • Most common-stack baselines are controlled adaptations, not reproductions of their native architectures, pretraining, or published benchmarks..Source: A LIMITATIONS AND SCOPE · p. 15

Training and selection cost

  • Training adds cached bridge probes, multi-budget rollouts, and validation; validation increases environment interactions by 2.75% in the reported control protocol.Selection protects only evaluated budgets and gives no guarantee for unseen solver counts.Source: A LIMITATIONS AND SCOPE · p. 15

Objective and theory

  • The bridge weight is an energy-based surrogate, and PPO-style clipping has no exact trust-region interpretation; the appendix does not prove monotone environment-return improvement.The fixed four-node bridge and budget sets are empirical choices; effectiveness with adaptive or higher-order solvers, continuous budgets, longer horizons, or other parameterizations remains open.Source: A LIMITATIONS AND SCOPE · p. 15

B IMPLEMENTATION DETAILS

The supplement provides implementation-related material that complements the main-text algorithm description.

B IMPLEMENTATION DETAILS

  • The complete bridge-based control realization and Algorithm 1 appear in Section 4.3.The supplement adds mathematical details, baseline scope, an energy-scale diagnostic, and the exact native-anchored manipulation update.Source: B IMPLEMENTATION DETAILS · p. 15
  • The supplement covers the scope of the baselines used for comparison.Source: B IMPLEMENTATION DETAILS · p. 15
  • The supplement specifies the energy-scale diagnostic and native-anchored manipulation update.Source: B IMPLEMENTATION DETAILS · p. 15

B.1 NATIVE-OBJECTIVE MANIPULATION UPDATE

The native manipulation update adds multi-budget endpoint supervision while retaining the native objective as an anchor. The accompanying mathematical basis explains boundary mismatch and finite-solver endpoint control, while explicitly limiting what those results establish.

B.1 NATIVE-OBJECTIVE MANIPULATION UPDATE

  • The native optimizer backtracks over candidate step fractions, accepting a finite step only when endpoint losses and cross-budget common-noise endpoints remain within stated tolerances.The check reuses training samples, makes no extra reward queries, and is not an independent return certificate; a separate cycling control uses K = 1, 2, 4.Source: §B.1 · p. 16

C MATHEMATICAL BASIS

  • For the deterministic reverse Euler field, the stated mathematical results identify the supervised deployment boundary and bound action reconstruction under explicit regularity assumptions.They do not assert monotone environment return or directly cover the clipped ShortCut operator.Source: C MATHEMATICAL BASIS · p. 16

C.1 BOUNDARY NON-COERCIVITY

  • The continuous noise-reconstruction risk is c^2δ^3/30, so arbitrarily small continuous surrogate risk does not control the deployed one-step endpoint error, even over continuous fields.Source: §C.1 · p. 16

C.3 ONE-STEP IDENTITY AND LOCAL GRADIENT

  • With common probes, the energy-based actor weight has a local gradient tied to the residual difference, while its finite clipped form has no exact likelihood-ratio or trust-region interpretation.The text also notes the energy scale changes gradient scale and when the numerical ratio reaches PPO clipping thresholds.Source: §C.3 · p. 17

C.4 CROSS-BUDGET BOUND

  • The cross-budget endpoint bound relies on time-regular residuals and can become loose for large H or a shifted reference bridge; it is not a reward theorem.The stated endpoint results keep Euler fixed and do not claim invariance to changes in flow geometry.Source: §C.4 · p. 18

D EXPERIMENTAL PROTOCOL AND BASELINES

The protocol evaluates checkpoints across three solver budgets, using disjoint episodes for checkpoint selection and reporting. Additional controls separate the collecting-grid weight’s effect on bridge energy from node placement.

D EXPERIMENTAL PROTOCOL AND BASELINES

  • Each checkpoint is evaluated at K = {1, 4, 16}; 32 of 64 episodes per budget select the checkpoint, while 32 disjoint episodes report its return.The nine evaluations add 1,728,000 environment steps (2.75% of the 62,914,560-interaction training budget) on tasks with 1,000-step episodes; selection is an empirical rule, not a formal robustness certificate.Source: D EXPERIMENTAL PROTOCOL AND BASELINES · p. 18
  • The common-stack comparison adapts FPO++, DPPO, ReinFlow, and PolicyFlow to a shared architecture, initialization, critic, and interaction budget, rather than reproducing their native benchmark settings.The DPPO row applies a denoising-MDP objective to Gaussian-noised Euler transitions on a flow backbone; native image-based ReinFlow results are reported separately.Source: D EXPERIMENTAL PROTOCOL AND BASELINES · p. 18, Table 4 · p. 19

E.1 ENERGY SCALE AND NODE PLACEMENT

  • The collecting-grid controls distinguish energy scale from node placement: weight 1/K makes energy depend on the collecting solver, while fixed 1/4 removes that dependence.Figure 4 shows bridge energy falling with increasing collecting budget under 1/K, while fixed-1/4 variants keep bridge energy and log-weight spread comparable; this diagnostic does not compare episodic return.Source: §E.1 · p. 19, Figure 4 · p. 19, E ADDITIONAL CONTROLS · p. 19

Related papers

Loading 2610.28788v1…