PaperlayerView exact revision on arXiv

Source-linked AI summary

Horizon Reduction Makes RL Scalable

Seohong Park, Kevin Frans, Deepinder Mann, Benjamin Eysenbach, Aviral Kumar, Sergey Levine

arXiv:2506.04168v3cs.LGcs.AI

TL;DR

The paper asks whether offline RL can solve increasingly complex, long-horizon tasks by scaling data and compute. It studies this question with billion-transition datasets, analyzes horizon-related barriers, and evaluates horizon-reduction methods including SHARSA. Standard methods often saturate below optimal performance, while SHARSA generally achieves the strongest scaling behavior and asymptotic performance among the evaluated methods.

  • Problem

    It remains unclear whether current offline RL algorithms can solve more challenging, longer-horizon tasks simply by scaling data, compute, and model capacity.

  • Method

    The paper evaluates offline RL on challenging robotics tasks with up to 1B transitions, analyzes horizon-related barriers, and tests horizon-reduction techniques including SHARSA.

  • Results

    Many standard offline RL methods saturate below optimal performance even with 1B-sized datasets, while SHARSA generally achieves the best scaling behavior and asymptotic performance among evaluated methods.

  • Takeaways & Limitations

    Explicitly reducing the value or policy horizon substantially improves offline RL scalability on challenging tasks.

  • Takeaways & Limitations

    The analysis uses idealized environments that remove visual representation learning, distributional shift, and some out-of-distribution generalization challenges.

Abstract

from arXiv · show

In this work, we study the scalability of offline reinforcement learning (RL) algorithms. In principle, a truly scalable offline RL algorithm should be able to solve any given problem, regardless of its complexity, given sufficient data, compute, and model capacity. We investigate if and how current offline RL algorithms match up to this promise on diverse, challenging, previously unsolved tasks, using datasets up to 1000x larger than typical offline RL datasets. We observe that despite scaling up data, many existing offline RL algorithms exhibit poor scaling behavior, saturating well below the maximum performance. We hypothesize that the horizon is the main cause behind the poor scaling of offline RL. We empirically verify this hypothesis through several analysis experiments, showing that long horizons indeed present a fundamental barrier to scaling up offline RL. We then show that various horizon reduction techniques substantially enhance scalability on challenging tasks. Based on our insights, we also introduce a minimal yet scalable method named SHARSA that effectively reduces the horizon. SHARSA achieves the best asymptotic performance and scaling behavior among our evaluation methods, showing that explicitly reducing the horizon unlocks the scalability of offline RL. Code: https://github.com/seohongpark/horizon-reduction

1 Introduction

The paper asks whether current offline RL algorithms can solve increasingly complex, long-horizon tasks by scaling data and compute. It finds that long horizons impede scaling and that horizon reduction, particularly SHARSA, substantially improves performance and scalability.

  • Prior evidence leaves unclear how offline RL scales to harder tasks requiring complex, longer-horizon sequential decision making.
  • The paper asks whether current offline RL algorithms can solve complex tasks simply by scaling data and compute.
  • Datasets reach 1B transitions per environment, up to 1000× larger than standard 1M-sized offline RL datasets.
  • Many existing algorithms either fail on complex tasks or require excessive compute and model capacity, with performance saturating below maximum performance.
  • Bias accumulation in TD targets and complex state-to-action mappings are identified as barriers associated with long-horizon value and policy learning.
  • Horizon reduction techniques improve scaling, while SHARSA reduces both value and policy horizons and generally achieves the best scaling behavior and asymptotic performance among evaluated methods.

2 Related work

Related work has studied offline RL across broader task collections and developed horizon-reduction techniques, but this paper focuses on solving harder sequential tasks as data and compute increase. It therefore examines a complementary depth-oriented scaling axis.

  • Offline RL: Offline RL methods address distribution shift through behavioral regularization, conservatism, weighted regression, in-sample maximization, uncertainty minimization, one-step RL, and model-based RL.
  • Scaling RL: Prior scaling studies often train multi-task agents to solve more diverse but not necessarily harder tasks.
  • Scaling RL: This paper studies whether more data and compute enable offline RL to solve more challenging tasks requiring complex sequential decision making.
  • Scaling RL: The paper frames this as scalability along a depth axis, complementary to prior width-oriented scaling across task diversity.
  • Horizon reduction and hierarchical RL: Prior horizon-reduction work includes multi-step or hierarchical value functions, hierarchical policy extraction, and high-level planning, often emphasizing exploration.

3 Experimental setup

The study evaluates offline goal-conditioned RL on challenging robotics tasks using large, task-agnostic datasets. It idealizes the setting to isolate sequential decision-making from visual learning, distribution shift, and coverage challenges.

  • Problem setting: The problem is to train agents that reach any goal state from any initial state in few steps using a static dataset of behaviors.
  • Problem setting: The setting uses sparse binary rewards and an unlabeled, reward-free dataset, creating a challenging multi-task learning problem.
  • Problem setting: The dataset contains N length-H state-action trajectories collected in an unsupervised, task-agnostic manner.
  • Environments and datasets: Experiments use four challenging OGBench robotics tasks, including sequential cube manipulation, Lights Out puzzle solving, and humanoid maze navigation.
  • Environments and datasets: Datasets contain up to 1B transitions from play-style trajectories designed to provide coverage and diversity.
  • Idealization: The analysis uses low-dimensional states, oracle goals, in-distribution evaluation goals, and datasets verified to have sufficient coverage and optimality.
  • Methods we evaluate: The evaluated methods include IQL, CRL, SAC+BC, and flow behavioral cloning, representing several offline model-free RL approaches.

4 Standard offline RL methods struggle to scale

Four standard offline RL methods are evaluated with datasets from 1M to 1B transitions on four complex, long-horizon tasks. Their performance often plateaus below optimal success, and all fail on the hardest cube-octuple task despite the largest datasets.

  • The study trains flow BC, IQL, CRL, and SAC+BC with 1M, 10M, 100M, and 1B-sized datasets.
  • None of the four standard methods solves all four tasks even with 1B-sized datasets, and all completely fail on cube-octuple.
  • Performance often quickly plateaus well below the optimal success rate of 100% despite scaling up data.
  • The dataset distribution provides sufficient coverage to learn a near-optimal policy, and the tasks are solvable or support non-trivial performance under further analyses.

Q: Have you tried further increasing the model size?

Larger models can improve performance somewhat, but increasing model capacity alone does not reliably master the hardest tasks within a practical compute budget. Performance often saturates or degrades despite substantially larger networks.

  • Performance often saturates or degrades as model size increases, including across residual MLP and Transformer architectures.These architectural ablations showed similar trends.
  • The study evaluates scalability within a reasonably bounded total compute budget rather than assuming unlimited training resources.The authors report that 591M-parameter models already required 8 days of training.
  • The authors could not rule out further gains from an even larger network with a smaller learning rate, but regard this as impractical for their compute setting.
  • Despite extensive adjustments to hyperparameters and design choices, the tested methods did not achieve promising scaling results.The ablations covered policy classes, architectures, ensembles, regularization, learning rates, target updates, batch sizes, and gradient settings.

5 The curse of horizon

The paper identifies horizon length as a central obstacle to offline RL scalability in both value and policy learning. Controlled experiments link long horizons to accumulating value errors and show that reducing the effective horizon can improve scaling.

  • 5 The curse of horizon: Long horizons are hypothesized to obstruct offline RL scaling through separate value-learning and policy-learning difficulties.The analysis treats the curse of horizon along two orthogonal axes.
  • 5.1 The curse of horizon in value learning: TD targets are biased at each gradient step, and these biases accumulate over the horizon in many offline RL algorithms.The paper contrasts this with objectives such as next-token prediction, where such bias accumulation does not occur or accumulate in the same way.
  • 5.1 The curse of horizon in value learning: 64-step DQN degrades more slowly than 1-step DQN as the combination-lock horizon increases from H = 256 to H = 4096.The two methods use datasets with uniform coverage for their respective trajectory-segment lengths.
  • 5.1 The curse of horizon in value learning: 1-step DQN has significantly larger Q errors than 64-step DQN despite similar TD errors, indicating compounded error in the learned Q function.The results support a link between bias accumulation and poorer long-horizon performance.
  • 5.1 The curse of horizon in value learning: Increasing model size or decreasing learning rate or target-network update rate provides limited or no improvement in 1-step DQN performance and Q-error accumulation.The comparison suggests that hyperparameter tuning alone does not resolve the long-horizon difficulty.
  • 5.1 The curse of horizon in value learning: The evidence does not eliminate the possibility that 1-step TD learning could converge with a very low learning rate and much larger network.The authors instead characterize its scaling as poor because good performance may require excessive compute, capacity, and practitioner time.
  • 5.2 The curse of horizon in policy learning: Hierarchical policies reduce policy-learning horizon by decomposing a goal-conditioned policy into high-level subgoal selection and low-level action selection.The individual hierarchical mappings are often less complex than the corresponding flat policy.

6 Horizon reduction makes RL scale better

The section evaluates value, policy, and combined horizon reduction on challenging offline RL tasks, finding that explicitly shortening horizons improves scalability and asymptotic performance. SHARSA combines both reductions with simple behavioral cloning and SARSA ingredients.

  • Evaluation setup: Horizon reduction techniques are evaluated across four challenging benchmark tasks to assess their effects on offline RL scalability.The methods separately reduce value horizons, policy horizons, or both.
  • SHARSA: SHARSA reduces both value and policy horizons using behavioral cloning and SARSA, aiming to improve scalability without extensive hyperparameter tuning.Its high-level component uses SARSA, while expressive flow policies support horizon reduction and policy extraction.
  • Value horizon reduction: n-step SAC+BC substantially improves scalability and asymptotic performance over SAC+BC on many tasks despite identical network sizes and training objectives apart from n-step returns.This comparison isolates value horizon reduction as the principal difference.
  • Policy horizon reduction: Hierarchical flow BC significantly improves performance over flow BC on a different set of tasks, with some tasks showing near-zero performance without policy horizon reduction.The cube-octuple task is given as an example where policy horizon reduction is especially important.
  • Combined horizon reduction: (Double) SHARSA is the only evaluated method achieving non-trivial performance on all four tasks, indicating the strongest combined value- and policy-horizon results.The section characterizes joint horizon reduction as combining the benefits of the two separate approaches.

7 Call for research: offline RL algorithms should be evaluated for scalability

The paper argues that offline RL should be tested for scalability on increasingly complex tasks and datasets, not only on standard small benchmarks. Its results motivate continued work because current techniques improve scalability but do not yet solve all tasks.

  • Findings: Standard non-hierarchical offline RL methods struggle to scale on complex tasks, while explicit horizon reduction, including SHARSA, can unlock scalability.This conclusion summarizes the empirical findings across the paper’s challenging benchmark setting.
  • Limitations: None of the evaluated horizon-reduction techniques masters all four tasks, even with 1B data, and some performance curves are non-monotonic with dataset size.The paper also notes that two-level hierarchies only mitigate, rather than fundamentally solve, TD-learning error accumulation.
  • Research agenda: The paper calls for scalable offline RL research using large datasets and complex tasks because success on small-scale benchmarks does not guarantee scaling to datasets 1000× larger.It frames scalability to harder tasks as complementary to prior work focused mainly on scaling across more tasks.

B Other attempts to fix the scalability of offline RL

The authors test whether scaling model capacity, architecture, optimization, regularization, or training can fix offline RL scalability without reducing the horizon. These alternatives generally help little or inconsistently, whereas horizon reduction achieves better asymptotic performance.

  • Scope: Nine non-horizon-reduction approaches were evaluated because standard offline RL struggles on complex, long-horizon tasks despite large datasets.The attempts included larger networks, Transformers, expressive policies, larger Q ensembles, regularization, hyperparameter changes, larger batches, longer training, and alternative losses.
  • Model capacity: Larger networks improve performance to some degree, but increasing capacity alone is insufficient; horizon reduction performs better even with default-sized models.The largest tested model had 591M parameters.
  • Architecture: Transformers improve performance on some tasks but still often fall significantly short of horizon reduction techniques.
  • Optimization and ensembles: Using 2 versus 10 Q networks produces nearly identical performance, while smaller learning rates and target network update rates do not substantially improve benchmark performance.
  • Training scale: Larger batches and 5× longer training help humanoidmaze-giant but do not improve the other three tasks.Extended training totals 25M gradient steps.
  • Alternative losses: Classification-based losses failed to produce significant improvement, and residual TD-error minimization failed to achieve non-trivial performance.

C Ablation studies of SHARSA

The ablations examine SHARSA’s value learner, high-level policy extraction, and value loss. SARSA, rejection sampling, and BCE loss provide the strongest reported choices within the tested framework.

  • Experimental setup: SHARSA ablations compare value learning methods, policy extraction methods, and value losses on the largest 1B datasets.
  • Value learning methods: SARSA achieves the best benchmark performance among SHARSA value-learning choices, outperforming IQL variants with κ = 0.7 and κ = 0.9.
  • Policy extraction methods: Reparameterized gradient-based extraction methods perform worse than rejection sampling, especially on puzzle tasks containing discrete state information.
  • Value losses: BCE value losses produce better performance and faster convergence than regression losses in the SHARSA comparisons.The authors also report generally better performance with BCE across underlying algorithms.

D Additional results

Additional results extend the horizon-reduction analysis to reward-based tasks and provide algorithmic and implementation details for the evaluated methods. They also describe SHARSA and related variants.

  • Reward-based tasks: The reward-based evaluation compares SARSA, IQL, SAC+BC, n-step SAC+BC, SHARSA, and an IQL variant of SHARSA on four OGBench tasks.
  • Horizon-reduction methods: n-step SAC+BC reduces the value horizon, whereas SHARSA reduces both the value and policy horizons.
  • Reward-based tasks: Horizon reduction significantly improves performance in reward-based, non-goal-conditioned offline RL as well.The reported performance is measured at the 1M epoch.
  • Algorithm details: The method descriptions cover flow behavioral cloning, SAC+BC, FQL, IQL, CRL, n-step SAC+BC, and their associated objectives and implementation choices.
  • Losses: The BCE loss generally performs and scales better than regression loss in the n-step SAC+BC experiments.
  • SHARSA variants: SHARSA is based on hierarchical flow behavioral cloning and n-step SARSA, while Double SHARSA adds another rejection-sampling round to the low-level policy.

F.1 Didactic experiments

The didactic experiments use controlled combination-lock environments and matched-coverage datasets to compare 1-step and 64-step DQN while measuring success, TD, and Q errors.

  • Task: The combination-lock task has H states, two discrete actions, and randomly ordered binary-vector state representations.
  • Algorithms: The study compares 1-step DQN with n-step DQN using n = 64, with double Q-learning used for training stability.
  • Datasets: The two datasets provide uniform state-action coverage tailored to 1-step DQN and 64-step DQN, respectively.
  • Comparison: Because 1-step DQN performs worse on the 64-step uniform dataset and 64-step DQN is incompatible with the 1-step dataset, the setup compares algorithm performance with the dataset factor marginalized out.
  • Metrics: Agents train for 5M gradient steps and are evaluated every 100K steps using success rate, TD error, and Q error.
  • Broader evaluation: The broader experiments use four OGBench tasks, 1B-sized datasets, task-specific tuning, and repeated rollout-based success-rate evaluation.

G Result tables

This section presents evaluation-goal figures, experiment hyperparameter tables, and full result tables for the reported methods and environments. Results are reported at 1M epoch and averaged over later epochs, with standard deviations and performance abbreviations specified.

  • Results: Result tables report standard deviations with “±” and abbreviate flow BC, hierarchical flow BC, n-step SAC+BC, and double SHARSA.The abbreviations are FBC, HFBC, n-SAC+BC, and DSHARSA, respectively.
  • Results: Values at or above 95% of the best performance are highlighted in bold.The section states that this follows Park et al. [75].
  • Hyperparameters: Tables 2–4 provide hyperparameters for didactic experiments, OGBench experiments, and policy extraction.Tables 3 and 4 separate common OGBench settings from policy-extraction settings.
  • Evaluation goals: Figures 16–21 define evaluation goals for six environments, including puzzle, cube, and humanoidmaze tasks.The cube-double evaluation omits task4, as stated in Section F.2.
Loading 2506.04168v3…