PaperlayerView exact revision on arXiv

Source-linked AI summary

Variance-Averse $n$-Step Offline Reinforcement Learning for Sparse Long-Horizon Environments

Guhyeon Kang, Minhae Kwon

arXiv:2610.07899v1cs.LGcs.AI

TL;DR

Generative actors can reproduce unreliable action modes in heterogeneous offline datasets, making expected Q-values alone inadequate for identifying consistent behavior. VAN-Flow combines categorical distributional value estimation, a variance-averse expectation, and rejection-sampled flow actions; across challenging benchmarks it consistently outperforms strong baselines, especially in long-horizon and high-variance regimes.

  • Problem

    In heterogeneous offline datasets, generative actors can reproduce high-variance action modes, while expected Q-values alone do not distinguish return consistency.

    Source: §1 · p. 1

  • Method

    VAN-Flow combines a categorical distributional critic with a variance-averse expectation and a flow-matching actor guided by rejection sampling.

    Source: §1 · p. 2

  • Results

    Across challenging benchmarks, VAN-Flow consistently outperforms strong baselines, with the largest gains in high-variance and long-horizon regimes.

    Source: §1 · p. 2, §7 · p. 10

  • Takeaways & Limitations

    The results support variance-averse aggregation over categorical return distributions as a way to select reliable behaviors for generative actors in offline RL.

    Source: §1 · p. 2, §7 · p. 10

    The method relies on categorical distributional critics; extending its aggregation operator to quantile-based critics remains future work.

    Source: M Broader Impact and Limitation · p. 34

Abstract

from arXiv · show

Generative actors are transforming offline reinforcement learning (RL) by enabling expressive policy classes that model complex action distributions. However, this expressiveness also exposes a key challenge in heterogeneous datasets: generative policies can reproduce unreliable action modes whose return distributions exhibit high variance, occasionally yielding high returns by chance but lacking consistency. Consequently, maximizing the expected $Q$-value alone is insufficient for identifying reliable actions. We propose VAN-Flow (Variance-Averse $n$-step Flow), a framework that promotes reliable actions in generative offline RL. VAN-Flow combines (i) a categorical distributional critic, (ii) a variance-averse expectation operator that smoothly reweights atom probabilities to favor actions with both high returns and low dispersion, and (iii) a flow-matching generative actor guided via rejection sampling. Unlike CVaR or mean-variance objectives, the operator redistributes probability mass over the categorical return distribution without hard truncation or auxiliary penalty terms. Across more than 40 tasks from D4RL and OGBench, VAN-Flow consistently outperforms strong baselines, with the largest gains in long-horizon and high-variance regimes where reliable action selection becomes critical.

1 Introduction

Generative actors can reproduce unreliable modes in heterogeneous offline datasets, so expected return alone cannot identify dependable actions. VAN-Flow combines distributional value estimation, variance-averse action scoring, and rejection sampling to favor reliable behaviors.

1 Introduction

  • Generative actors fit multimodal dataset actions, including unreliable modes that can yield high returns by chance, especially when sparse-reward successes are rare.Heterogeneous behavior policies can produce sharply different outcomes for the same state–action pair.Source: §1 · p. 1
  • The variance-averse expectation smoothly reweights categorical return probabilities to favor actions with both high returns and low variance, unlike hard-tail truncation.It is designed for selecting among candidate actions from a multimodal policy rather than defining a new risk-sensitive control objective.Source: §1 · p. 2
  • On high-variance explore, FQL-n has a markedly larger performance drop than ReBRAC-n, while reliability-driven n-step training degrades only marginally.Figure 1 compares antmaze-large success rates under low-variance navigate and high-variance explore datasets.Source: §1 · p. 2, Figure 1 · p. 2
  • VAN-Flow combines a categorical distributional critic, variance-averse expectation, and flow-matching actor guided by rejection sampling to select reliable actions.The rejection-sampling selection aligns final actions with the reliable subset of the dataset.Source: §1 · p. 2
  • Across more than 40 D4RL and OGBench tasks, VAN-Flow consistently outperforms strong baselines, with largest gains in high-variance and long-horizon regimes.The paper also reports ablations and sensitivity analyses supporting the components and reliability-aware action selection.Source: §1 · p. 2

2 Preliminaries

The preliminaries set up offline n-step actor–critic learning and categorical return-distribution estimation, alongside flow matching as a stochastic policy parameterization.

2 Preliminaries

  • The setting is an MDP with states, actions, transitions, rewards, and discount, where the policy objective is expected return.The passage defines M = (S, A, P, r, γ) and an infinite-horizon return objective.Source: §2 · p. 3
  • Offline learning uses a fixed dataset and trains the critic against an n-step Bellman target; n-step returns have improved sparse-reward performance in recent work [17] [18].The distributional target shifts each return atom by the n-step reward and discounted continuation, then projects it back onto the fixed support.Source: §2 · p. 3
  • A categorical critic models the full return distribution over fixed atoms, exposing variance structure unavailable from a regression-based critic.Its expected return is recovered from atom probabilities, and training uses cross-entropy against the projected n-step distributional target.Source: §2 · p. 3
  • Flow matching parameterizes a stochastic policy through a state-conditioned velocity field, with actions formed by following the learned flow.The described interpolation begins from noise and reaches the action at the end of the flow.Source: §2 · p. 3

3 Variance-averse Expectation with Categorical Value Estimation

The section introduces a variance-averse expectation over categorical return distributions to prefer high-return, less-dispersed actions. It relates the operator to convex-order dispersion and bounds its discretization gap.

3 Variance-averse Expectation with Categorical Value Estimation

  • For categorical returns, the variance-averse expectation reweights atom probabilities to penalize high-variance distributions; δ controls aversion, and δ = 0 recovers ordinary expectation.Increasing δ strengthens the penalty on high-variance returns; Figure 2 illustrates the operator for varying δ and return variance.Source: §3 · pp. 3–4, Figure 2 · p. 3
  • For equal-mean distributions, the spectral form ranks the less dispersed distribution higher under convex order, while the implemented operator approximates that ordering within a concentration-dependent error.The certificate is computable from critic output; on trained critics, the median gap was 0.37% of R and the two operators selected the same candidate in 96.9% of states (Kendall’s τ = 0.99).Source: §3 · p. 4
  • Under a supported C51 projection with a Lipschitz-continuous CDF, the maximum atom probability and the operator’s error εδ(Z) are O(1/I), so the gap vanishes as atom count I grows.The stated result assumes the target is supported on [Vmin, Vmax].Source: §3 · p. 4

4 VAN-Flow: Variance-averse n-step RL with Flow-based Policy

VAN-Flow uses a flow-based actor and variance-averse distributional value estimates to apply n-step learning on high-variance datasets. Its action selection, critic targets, and actor guidance are coordinated around reliable actions.

4 VAN-Flow: Variance-averse n-step RL with Flow-based Policy

  • The framework pairs n-step learning with categorical distributional value estimation and a variance-averse expectation to select reliable actions under high-variance datasets.The flow-based actor is guided by variance-averse Q estimates and rejection sampling.Source: §4 · p. 4

4.1 Flow-based Actor Design

  • The flow actor generates candidate actions and uses variance-averse rejection sampling to retain reliable, in-distribution candidates.Action construction advances the interpolated flow state with an Euler update rather than explicitly solving the continuous-time ODE.Source: §4.1 · p. 5

4.2 Variance-averse Q-guided Flow Matching

  • The categorical critic learns from the projected n-step distributional Bellman target using the variance-averse rejection-sampled action as its target action.The flow-matching actor also receives critic guidance through a variance-averse loss; a single Euler-step action proxy propagates the critic gradient to the velocity field.Source: §4.2 · pp. 5–6

5 Simulation Result

VAN-Flow is evaluated on sparse, long-horizon D4RL and OGBench tasks, where it remains strong across difficult settings. Ablations examine its actor, critic, value aggregation, Q guidance, parameter sensitivity, and runtime.

5.1 Simulation Setup

  • Evaluation covers D4RL AntMaze and state-based OGBench tasks with sparse rewards, long horizons, and noisy or suboptimal datasets.Results are averaged over five random seeds.Source: §5.1 · p. 6, §5 · p. 6

5.2 Performance Comparison

  • VAN-Flow remains strong from umaze to large-diverse and robust on noisy, long-horizon tasks, while n-step flow baselines drop sharply in high-variance settings.The comparisons cover OGBench and D4RL AntMaze; Gaussian policies also degrade on long-horizon and noisy tasks.Source: Table 1 · p. 7, Table 2 · p. 7, §5.2 · pp. 7–8

5.3 Ablation Studies

  • Flow+Categorical consistently outperforms variants using regression-based critics or Gaussian actors on humanoidmaze-giant-navigate.Flow-based actors also outperform Gaussian counterparts on long-horizon, sparse-reward problems; Appendix G reports over 90% success with three flow steps versus 20 diffusion steps for comparable performance.Source: §5.3 · p. 8, Figure 3 · p. 8
  • E(Z) achieves the lowest Var[Z] across tasks and the highest success rates, using fixed δ = 2 across nearly all tasks.Unlike CVaR or mean–variance and entropic-risk rules, it reweights categorical return probabilities without hard truncation or auxiliary penalties.Source: Table 3 · p. 8, §5.3 · p. 8
  • Q-guided actors achieve high success rates and time efficiency, while unguided actors can solve tasks yet remain inefficient.Time efficiency is based on episode length, normalized against the maximum episode horizon.Source: Figure 4 · p. 9, §5.3 · p. 9

5.4 A Close Look at VAN-Flow

  • Larger n can improve long-horizon performance, but excessive n risks diminishing or negative returns; rejection-sampling gains saturate, and three to five flow steps already perform well.Most tasks are stable with fewer than 32 rejection candidates; performance changes only mildly beyond three to five flow steps.Source: Figure 5 · p. 9, §5.4 · p. 9

5.5 Computational Complexity Analysis

  • VAN-Flow has O(UMKd^2) training complexity, yet runs faster than QC in step and wall-clock time and remains comparable to other generative-policy baselines.The M and K factors arise from evaluating candidate actions generated through flow transformations; small values keep practical cost low.Source: Table 4 · p. 9, §5.5 · pp. 9–10

6 Related Works

Related work addresses offline n-step returns, generative policies, and distributional or risk-sensitive value aggregation. VAN-Flow’s stated distinction is its categorical, variance-averse aggregation for unreliable high-variance targets in offline n-step learning.

Offline RL with n-step Returns

  • Offline n-step methods address bootstrapping bias through compound estimators, decoupled horizons, λ-returns, temporal-distance augmentation, or foresight.The cited approaches include PiLaR, Decoupled Q-chunking [45], LEQ [17], and methods in [46], [47].Source: §6 · p. 10

Generative Policies for Reinforcement Learning

  • Generative offline policies use diffusion or flow models to represent complex action or trajectory distributions, with some methods adding critic guidance.Examples include Diffusion Policy, Diffuser [49], Diffusion-QL [1], IDQL [50], Flow Q-Learning [3], Q-chunking [4], and energy-weighted flow matching [20].Source: §6 · p. 10

Distributional and Risk-sensitive RL

  • Distributional critics model full return distributions, while common risk-sensitive rules alter aggregation; VAN-Flow instead reweights all atoms of categorical critics without hard truncation.The related work frames its target as suppressing unreliable, high-variance targets in offline n-step learning.Source: §6 · p. 10

7 Conclusion

VAN-Flow combines n-step learning, categorical distributional value estimation, variance-averse action evaluation, and a flow-based actor with rejection sampling. It consistently outperforms strong baselines across challenging benchmarks, with the conclusion emphasizing high-variance datasets as a setting where naive n-step horizon reduction breaks down.

7 Conclusion

  • VAN-Flow combines n-step learning, categorical distributional value estimation, variance-averse Q estimates, and a flow-based actor guided by rejection sampling.The paper reports consistent gains over strong baselines across challenging benchmarks.Source: §7 · p. 10

A Notation Tables

  • The notation distinguishes the scalar critic used in standard one-step formulations from VAN-Flow’s categorical return distribution critic and defines E(Z) as its variance-averse expectation.The fixed support atoms and probabilities represent the categorical return distribution.Source: A Notation Tables · p. 16

B.1 Proof of Theorem 3.2(i)

  • The implemented variance-averse operator preserves first-order stochastic dominance, while variance affects rankings only when return distributions are not dominance-ordered.Its guarantee holds up to a discretization gap for the implemented operator.Source: §B.1 · p. 17

B.2 Design Considerations

  • The variance-averse reweighting gives zero weight to the largest support atom, with the resulting value gap bounded by R εδ(Z).On trained critics, E and Esp selected the same rejection-sampling candidate in 96.9% of states.Source: §B.2 · p. 18
  • A numerical stabilizer η = 10^-8 is added to the denominator; the paper reports no resulting numerical instability in its experiments.For nearly deterministic returns, the spectral form Esp is presented as a drop-in alternative that returns the support value for a point mass.Source: §B.2 · p. 18

B.3 Proof of Theorem 3.2(ii): Discretization Error of the Implemented Operator

The implemented operator approximates the spectral form with an error controlled by return support and atom-mass concentration, and its ordering agrees when the convex-order margin exceeds the certified error budget. The appendix also specifies the categorical projection, benchmark variance regimes, and reward preprocessing used by VAN-Flow.

B.3 Proof of Theorem 3.2(ii): Discretization Error of the Implemented Operator

  • The implemented operator’s value error is bounded by R εδ, and its ordering matches the spectral form when the convex-order margin exceeds the combined error budget.The certificate εδ is bounded using the largest atom probability ρ and, under an additional condition, the sum of squared atom probabilities.Source: §B.3 · pp. 18–19
  • On trained critics, the realized discretization gap was a median 0.37% of the value range, and both operators selected the same candidate in 96.9% of states.Candidate rankings had Kendall’s τ = 0.99 across 256 dataset states and 32 candidate actions per state on antmaze-teleport-navigate.Source: §B.3 · p. 19, Table 6 · p. 19

C Projection Operator

  • The categorical projection clamps target returns to the fixed support and distributes each categorical mass between the two nearest atoms.The critic represents values with probabilities over uniformly spaced atoms; rewards and discounted next-state returns form the unprojected target distribution.Source: C Projection Operator · p. 20

D.1 D4RL antMaze

  • D4RL AntMaze varies maze scale and dataset type: larger mazes extend planning horizons, while diverse behavior datasets induce more return variance than play datasets.Sparse goal rewards make long-horizon value propagation central; increasing scale and heterogeneity amplifies variance in n-step targets.Source: §D.1 · p. 21

D.2 OGBench

  • OGBench spans diverse long-horizon tasks and dataset variants, enabling comparisons of low- and high-return-variance data under the same task dynamics.The evaluated task families include AntMaze, HumanoidMaze, Scene, and Puzzle; explore and noisy datasets are categorized as high variance, while navigate and play are low variance.Source: §D.2 · pp. 21–22

E Implementation Details of VAN-Flow

  • Reward preprocessing rescales rewards to {−1, 0}, keeping discounted returns within the critic support while adding a state-independent constant to Q-values.The lower support bound approximates the worst-case return when the goal is never reached; the transformation does not affect the optimal policy.Source: E Implementation Details of VAN-Flow · p. 23

F Component-wise Ablation Study

The ablations show that the variance-averse expectation and flow actor contribute complementary benefits, while flow matching reaches strong performance with fewer inference steps than the diffusion alternative. Additional sections describe evaluation protocols, learning-curve presentation, and Adroit comparisons.

F Component-wise Ablation Study

  • On humanoidmaze-large, removing Flow drops success from 87 to 0, while removing VE drops it from 87 to 77; on low-dimensional tasks, removing VE causes the larger drops.The ablation authors attribute the complementary roles to Flow’s action expressiveness and VE’s reliability control over value estimates.Source: F Component-wise Ablation Study · p. 24, Table 8 · p. 24

G Empirical Comparison of Flow Matching and Diffusion Policies

  • VAN-Flow exceeds 90% success with 3 flow steps (3.3ms), while VAN-DDPM needs 20 denoising steps (11.9ms) for comparable success, reducing inference cost by roughly 3.6×.VAN-DDPM has near-zero success at five or fewer denoising steps, whereas flow matching performs strongly even at low step counts.Source: G Empirical Comparison of Flow… · p. 25, Table 9 · p. 25

H.1 Baselines Description

I.1 Learning Curve of VAN-Flow

  • The learning curves average evaluation success over five seeds, show offline training followed by online fine-tuning, and mark the phase switch with a vertical dashed line.The switch is at 1M steps for OGBench and 0.5M for D4RL.Source: §I.1 · p. 26

I.2 Full Performance Table

  • The full OGBench performance table reports normalized scores for every evaluated task and compared baseline under the corresponding benchmark protocol.The supplied table caption identifies the singletask setting but provides no score values.Source: §I.2 · p. 29, Table 10 · p. 29

I.3 Performance Comparison in Denser Reward Environments

  • VAN-Flow is competitive or superior across multiple Adroit tasks, particularly cloned and expert datasets, and outperforms flow-based FQL and IFQL on most tasks.It remains competitive with strong diffusion-based methods; the authors attribute gains over flow baselines to variance-aware n-step value learning rather than generative policy use alone.Source: §I.3 · p. 30, Table 11 · p. 30

I.4 D4RL Locomotion Tasks

On dense-reward D4RL locomotion, VAN-Flow is competitive with strong baselines, matching or outperforming ReBRAC on all medium-replay datasets. Its gap is small on more homogeneous medium datasets, consistent with the method’s intended setting.

I.4 D4RL Locomotion Tasks

  • VAN-Flow’s mean score is 80.0, compared with 79.4 for ReBRAC and 78.9 for Diffusion-QL, and it matches or exceeds ReBRAC on all three medium-replay datasets.The gap is small on the more homogeneous medium datasets; the experiments used medium and medium-replay data, neither containing expert demonstrations.Source: §I.4 · pp. 30–31, Table 12 · p. 30
  • The authors report that variance adds little beyond expected return on datasets from a single behavior policy, while mixed-quality behavior modes make return variance informative for separating reliable from unreliable actions.This interpretation accompanies the contrast between medium-replay and more homogeneous medium datasets.Source: §I.4 · pp. 30–31

J Off-to-Online Fine-Tuning Results

VAN-Flow adapts stably during online fine-tuning, improving on unsaturated tasks and matching or slightly surpassing QC in several settings. Additional comparisons favor its variance-averse expectation over tested risk-sensitive alternatives, and its performance remains robust across a range of n-step horizons.

J Off-to-Online Fine-Tuning Results

  • VAN-Flow improves with online interaction on every task not already saturated offline and matches or slightly surpasses QC in several settings.The results compare offline-trained initializations across long-horizon, noisy, and D4RL AntMaze tasks.Source: J Off-to-Online Fine-Tuning Results · p. 31, Table 13 · p. 31

K Additional Comparisons with Risk-Sensitive Objectives

  • VE outperforms the tested risk-sensitive alternatives on antmaze-teleport-navigate and antmaze-large-explore, including CVaR, mean–variance, and entropic risk.The authors note that CVaR can be overly conservative, while scalar penalty objectives do not fully use the categorical return structure; VE smoothly reweights the full distribution.Source: K Additional Comparisons with… · p. 31, Table 14 · p. 31

L.1 n-step Horizon

L.2 Number of Rejection Samples

Sensitivity analyses show that rejection-sample performance quickly saturates and flow-step needs vary by task. The default aversion coefficient and categorical support choices balance performance with robustness, while atom-count performance is stable once resolution is sufficient.

L.2 Number of Rejection Samples

  • Performance generally improves as rejection candidates increase, then quickly saturates, so strong results do not require excessive sampling.Figure 14 reports additional rejection-sample sensitivity results.Source: Figure 14 · p. 32, §L.2 · p. 32

L.3 Number of Flow Steps

  • A single flow step is insufficient; maze tasks approach their best at 3–5 steps, while scene-play and puzzle-3x3-play need about 10, and larger K adds little.Humanoidmaze-large shows a mild decline for large K; K = 10 is a safe cross-task default.Source: Figure 15 · p. 32, §L.3 · p. 32

L.4 Aversion Coefficient δ

  • The default δ = 2 is at or near best on three environments, while δ ≥5 collapses success on antmaze-large-explore and δ = 7 works better on antmaze-teleport-navigate.Table 15 sweeps δ on the first task of three environments; stochastic teleporters leave irreducible variance in multi-step targets.Source: Table 15 · p. 33, §L.4 · p. 33

L.5 Categorical Support

  • Underestimating categorical support clips Bellman targets and breaks training, whereas overestimating it only lowers resolution; the support should cover the return range, erring wide.The support bounds are set by [rmin/(1 −γ), rmax/(1 −γ)] rather than tuned per task.Source: Table 16 · p. 33, §L.5 · p. 33

L.6 Number of Atoms

  • Performance is stable across roughly a tenfold atom-count range once resolution is sufficient; scene-play needs more than I = 51, while I = 101 costs little extra wall-clock time.Doubling the atom count restores scene-play performance; wall-clock time barely changes across the tested range.Source: Table 17 · p. 33, §L.6 · p. 33

M Broader Impact and Limitation

The paper identifies possible benefits of more reliable offline RL in expensive- or unsafe-to-explore applications, while cautioning that real-world use still depends on data quality and safeguards. Its limitations include added computation, reliance on categorical critics, and possible over-conservatism when dynamics create irreducible variance.

Broader Impact

  • Improved reliability in sparse, long-horizon offline RL may benefit applications where online exploration is expensive or unsafe, including robotics, navigation, UAV control, and decision support.These possible applications use policies trained from logged data.Source: M Broader Impact and Limitation · pp. 33–34

Limitations

  • Biased or incomplete datasets may cause stronger offline RL systems to inherit or amplify undesirable behaviors, so safety-critical use still requires curation, evaluation, and deployment safeguards.Source: M Broader Impact and Limitation · p. 34
  • Rejection sampling and distributional aggregation add modest computation, although the operations parallelize and practical overhead is small in typical training.Source: M Broader Impact and Limitation · p. 34
  • The approach relies on categorical distributional critics; extending its aggregation operator to quantile-based critics remains future work.Source: M Broader Impact and Limitation · p. 34
  • When dynamics are strongly stochastic, irreducible return variance can make indiscriminate penalization overly conservative even for well-supported actions.The proposed state-dependent or annealed aversion extensions would require additional uncertainty machinery.Source: M Broader Impact and Limitation · p. 34

Related papers

Loading 2610.07899v1…