PaperlayerView exact revision on arXiv

Source-linked AI summary

Training Agents using Upside-Down Reinforcement Learning

Rupesh Kumar Srivastava, Pranav Shyam, Filipe Mutz, Wojciech Jaśkowski, Jürgen Schmidhuber

arXiv:1912.02877v2cs.LGcs.AIcs.RO

TL;DR

The paper asks whether agents can learn to act efficiently using only supervised learning, rather than relying on traditional reinforcement-learning objectives. It introduces UDRL, which trains agents to follow commands from past experience; experiments report high empirical returns and useful behaviors across episodic environments. The authors argue that alternative approaches to expected-return maximization merit study.

  • Problem

    The paper asks whether high-dimensional agents can learn efficiently using only supervised learning, given that reinforcement learning supplies evaluation signals rather than action-level error signals.

    Source: §1 · p. 1

  • Method

    UDRL trains a behavior function with supervised learning on past interactions, mapping commands such as a desired return within a horizon to actions.

    Source: §1 · p. 2, §2.1 · p. 3

  • Results

    UDRL achieves high empirical returns without formally maximizing expected returns, and agents can learn from delayed rewards and adjust behavior to varying desired returns.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The behavior function encodes past behaviors to produce actions conditioned on desired future outcomes, opening the possibility of importing supervised-learning techniques into reinforcement learning.

    Source: §5 · p. 12

    In stochastic environments, hindsight-generated examples do not provide optimal trajectories for fulfilled commands, and theoretical analysis—especially with behavior-function generalization—remains open.

    Source: §5 · p. 12

Abstract

from arXiv · show

We develop Upside-Down Reinforcement Learning (UDRL), a method for learning to act using only supervised learning techniques. Unlike traditional algorithms, UDRL does not use reward prediction or search for an optimal policy. Instead, it trains agents to follow commands such as "obtain so much total reward in so much time." Many of its general principles are outlined in a companion report; the goal of this paper is to develop a practical learning algorithm and show that this conceptually simple perspective on agent training can produce a range of rewarding behaviors for multiple episodic environments. Experiments show that on some tasks UDRL's performance can be surprisingly competitive with, and even exceed that of some traditional baseline algorithms developed over decades of research. Based on these results, we suggest that alternative approaches to expected reward maximization have an important role to play in training useful autonomous agents.

1 Introduction

UDRL reframes learning to act as following commands about desired outcomes, using supervised learning on past experience. The paper develops this approach for episodic tasks and reports useful behaviors despite not formally maximizing expected returns.

1 Introduction

  • The motivating challenge is whether agents can learn efficiently in high-dimensional environments using only supervised learning, despite reinforcement learning providing evaluation signals rather than action-level error signals.The passage notes prior discussion of this challenge by Rosenstein and Barto [2004] and Barto and Dietterich [2004].Source: §1 · p. 1
  • UDRL trains agents to map commands—such as a desired return within a time horizon—to actions, using past experiences as examples of successful command-following.Cumulative reward is an input rather than a prediction, and training uses a supervised-learning objective.Source: §1 · p. 2
  • UDRL can achieve high empirical returns without formally maximizing expected returns, using minimal additional algorithmic ingredients to steer learning toward higher-return commands.The paper develops concrete implementations and discussion for episodic tasks.Source: §1 · p. 2
  • The behavior-function representation is realized in several non-trivial environments and is intended to use neural-network generalization without target networks or other stabilization techniques.The paper also reports that agents naturally learn from delayed rewards and can adjust behavior after training to commands specifying different desired returns.Source: §1 · p. 2
  • Agents can naturally learn from delayed rewards and adjust their behavior after training in response to commands specifying varying desired returns.Source: §1 · p. 2

2 Upside-Down Reinforcement Learning

Upside-Down Reinforcement Learning trains agents to follow commands about desired returns and time horizons rather than predict values or explicitly maximize them. The section describes a supervised-learning algorithm that learns from past experience, biases training toward high-return trajectories, and uses the behavior function for exploration and evaluation.

2.1 Knowledge Representation

  • UDRL replaces value prediction with a command-conditioned behavior function mapping state, desired return, and horizon to actions, learned to generalize beyond expensive trajectory-table lookup.Training examples pair a segment’s first state and action with its cumulative reward and duration, computed retrospectively.Source: §2.1 · pp. 3–4

2.2 Maximizing Episodic Returns using Upside-Down RL

  • The algorithm repeatedly trains on replay-buffer experience, uses the behavior function to gather new episodes, and continues until a stopping criterion is met.The section frames this practical algorithm as a test of whether supervised learning can solve interesting RL problems while targeting high average episodic returns.Source: §2.2 · p. 5

2.2.1 Replay Buffer

  • Training uses the replay buffer’s best-return trajectories, which biases learning toward higher returns but may weaken obedience to low-return commands later.The buffer is populated initially with episodes generated by random actions.Source: §2.2.1 · p. 5

2.2.2 Behavior Function

  • At each step, the behavior function conditions an action distribution on the current state and command, and the command is updated using rewards and time remaining.The behavior function induces a policy at each step rather than acting as a typical fixed policy.Source: §2.2.2 · p. 5

2.2.3 Training the Behavior Function

  • Supervised training samples an episode segment’s initial state and action, labeling them with the segment’s cumulative reward and duration.Experiments use trailing segments ending at T − 1, reducing the examples and commands the behavior function must learn for episodic tasks.Source: §2.2.3 · p. 6

2.2.4 Sampling Exploratory Commands

  • Exploratory commands use the mean length of selected high-return episodes as the horizon and sample desired returns from U[M, M + S].M and S are the mean and standard deviation of those episodes’ returns; command selection should produce meaningfully different experience to drive learning progress.Source: §2.2.4 · pp. 6–7

2.2.5 Generating Experience

  • The agent generates a fixed number of episodes per learning iteration by sampling actions from the behavior function and adding the episodes to the replay buffer.Source: §2.2.5 · p. 7

2.2.6 Evaluation

  • Evaluation commands are derived from recent exploratory commands, relying on similar return-and-horizon commands being repeatable; the strategy may not apply in arbitrarily stochastic environments.For continuous-valued actions, evaluation uses the mode of the action distribution.Source: §2.2.6 · p. 7

3 Experiments

Across benchmark tasks, UDRL was competitive with or outperformed traditional baselines on all but InvertedDoublePendulum-v2. It also retained much of its performance with delayed sparse rewards and showed command-responsive behavior.

3 Experiments

  • Simple state-command concatenation produced inconsistent results, while first-layer fast-weight architectures improved reliability across runs.The experiments tested gating and bilinear fast-weight designs.Source: §3 · pp. 7–8
  • The experiments compared discrete- and continuous-action tasks using tuned algorithms, separate training and evaluation seeds, and 20 independent runs for final comparisons.Tasks included low- and high-dimensional observations, and the benchmark environments were described as solved problems used to enable rigorous comparisons.Source: §3 · pp. 7–8

3.1 Results

  • All algorithms solved LunarLander-v2, reaching average returns above 200; DQN and UDRL were behind A2C in sample complexity and final returns.The authors say A2C benefits from its use of multi-step returns.Source: §3.1 · p. 8
  • On TakeCover-v0, UDRL comfortably outperformed A2C and DQN, whose evaluation scores fluctuated substantially.The authors note that additional techniques or environment modifications might address the baselines’ instabilities.Source: §3.1 · p. 8
  • On Swimmer-v2, UDRL outperformed TRPO and PPO and was on par with DDPG, whose evaluations were erratic and sensitive to hyperparameters.DDPG stalled at low returns for some random seeds, while UDRL showed consistent progress.Source: §3.1 · p. 8
  • UDRL was competitive with or outperformed traditional baselines on every task except InvertedDoublePendulum-v2.Figure 3 aggregates evaluation scores across benchmark tasks; the results also describe UDRL as simple and relatively immature.Source: §3.1 · pp. 8–9, Figure 3 · p. 9

3.2 Sparse Delayed Reward Experiments

  • In sparse delayed-reward settings, UDRL retained much of its performance without modification, while baseline methods often became unstable, slow, or failed.The best baseline on LunarLanderSparse-v2, A2C with 20-step returns, was highly sensitive to hyperparameters; UDRL also improved on Inverted Double Pendulum, approaching PPO's dense-reward performance.Source: §3.2 · p. 9, Figure 4 · p. 10

3.3 Different Returns with a Single Agent

  • A LunarLander-v2 agent’s obtained and desired returns had a strong correlation (R ≈0.98), indicating command-responsive behavior after training.Some agents did not produce very low returns when commanded to do so, which the authors attribute to occasional training stochasticity.Source: §3.3 · p. 10, §3.2 · p. 10

4 Related Work

The related work situates UDRL among supervised-learning, goal-conditioned, and alternative reinforcement-learning approaches. It distinguishes UDRL by conditioning one learned behavior function on time-varying return and horizon commands, without value functions or policy gradients.

4 Related Work

  • Many model-free methods combine value functions and policy search, but bootstrapping can create changing targets for the same inputs.The discussion also notes experience replay and supervised-learning subroutines in these methods.Source: §4 · p. 10
  • Unlike goal-conditioned policies, UDRL’s behavior function accepts time-varying desired returns and horizons and is trained without value functions or policy gradients.The authors frame this as a simpler setup for potentially using a variety of supervised-learning techniques.Source: §4 · p. 10
  • PowerPlay also uses task inputs and supervised learning on past trajectories, but its task inputs are not systematically constructed from previous behaviors as UDRL’s are.RUDDER instead uses supervised learning for contribution analysis mapping rewards to state-action pairs.Source: §4 · p. 11
  • Policy search methods were adapted into weighted regression for immediate rewards by Peters and Schaal [2007], then extended to episodic settings by Wierstra et al. [2008].The paper describes this as a more supervised-learning-like setup within each iteration.Source: §4 · p. 11
  • Other pure-supervised-learning methods condition on fixed goals, whereas UDRL uses time-varying desires and horizons and its learned behavior function for exploration.Lynch et al. [2020] use human-generated play data; Ghosh et al. [2019] use agent-generated data. RCP-R [Kumar et al., 2019] differs in using discount factors rather than explicit horizon conditioning.Source: §4 · p. 11

5 Discussion

The discussion presents behavior functions as an alternative way to encode knowledge and act, while acknowledging that UDRL lacks traditional RL’s theoretical guarantees for arbitrary stochastic environments. It identifies potential advantages, practical constraints, future directions, and broader-impact concerns.

5 Discussion

  • Behavior functions produce immediate actions conditioned on desired outcomes, rather than predicting long-term horizons conditioned on a policy.Source: §5 · p. 12
  • Plain neural networks with concatenated state and command inputs were difficult to train reliably; useful behavior functions require actions to depend strongly on commands.Source: §5 · p. 12
  • UDRL represents rewards and horizons directly in commands, avoiding discount-related distortions and allowing real-time horizons to be command inputs.Source: §5 · p. 12
  • In stochastic environments, hindsight-generated examples are not optimal trajectories, and evaluating command feasibility generally requires additional algorithmic ingredients.Source: §5 · pp. 12–13

5.1 Further Research Directions

  • Future work includes recurrent models for partially observable settings, varied command formats and history sampling, and additional exploration in some environments.Source: §5.1 · p. 13

5.2 Broader Impact

  • The approach may have positive applications such as reducing industrial waste, but its use also raises risks including job losses and harmful recommendation feedback loops.Source: §5.2 · p. 13
  • Because neural-network function approximators are poorly understood theoretically, the authors call for rigorous safe-environment testing and safeguards against unintended behavior.Source: §5.2 · p. 13

Contributions

The paper distinguishes its behavior-function approach from world models and fixed-dataset learning. It also records the authors’ contributions to the work.

Contributions

  • JS proposed the basic principles; WJ developed the baseline code setup, while PS, FM, and RKS contributed implementations, experiments, supervision, and writing.Source: Contributions · p. 14

A Further Discussion of Related Work

A.1 Comments on RCP-R

The RCP-R discussion identifies a mismatch between how reward-to-go values are assigned during training and how they are supplied during rollout. The environment descriptions cover tasks with different observations, actions, and objectives.

A.1 Comments on RCP-R

  • RCP-R trains on state-action pairs labeled with observed reward-to-go, but samples one fixed value at rollout, potentially creating different training and evaluation distributions.Source: §A.1 · p. 21

B Description of Environments Used

C Network Architectures

Fast-weight architectures encourage command-dependent behavior, and the authors find them more reliable than simple concatenation under limited tuning. The section describes gated and bilinear variants that condition processing on commands.

C Network Architectures

  • Fast-weight architectures were more reliable under limited tuning because concatenation networks often showed high variation across random seeds.The authors report that concatenation networks could still perform well in some experiments, but often required more extensive tuning.Source: C Network Architectures · p. 23
  • Fast weights make command inputs harder to ignore and enable multiplicative interactions suited to behavior that varies with the command.With simple concatenation, a network can assign command inputs low weights and reduce loss, especially early in training when experience is less diverse.Source: C Network Architectures · p. 23

Gated

  • The gated variant multiplies contextual representation x by gate g to produce y; its parameters include command-conditioned weights and biases.The section defines σ as the sigmoid nonlinearity and gives U and p as parameters for the gate.Source: Gated · p. 24

Bilinear

  • The bilinear variant applies a linear transform to observation o whose transformation parameters are produced through linear transforms of command c.The implementation uses multiplicative interactions in the first layer, except for TakeCover-v0, where a bilinear transform is used only in the last layer and the gated variant is not tuned.Source: Bilinear · p. 24
  • For TakeCover-v0, the authors used a bilinear transformation only in the last layer and did not tune the gated variant.Other network layers were fully connected, and an activation function was used with the bilinear transformation.Source: Bilinear · p. 24

C.1 Continuous Control Parameterization

For continuous actions, the behavior function outputs parameters of independent normal distributions, with squashing choices intended to respect bounded actions and avoid numerical issues.

C.1 Continuous Control Parameterization

  • For continuous actions, the behavior function outputs independent normal distributions using a vector of means and a vector of log standard deviations.Means are squashed with tanh, while log standard deviations are squashed with sigmoid and scaled to [-6, 2].Source: §C.1 · p. 24

D Upside-Down RL Hyperparameters

E Benchmarking Setup and Hyperparameter Tuning

The benchmark setup uses random hyperparameter search and 20 final evaluation seeds, while tuning uses only two or three seeds per configuration. The search is a moderate-tuning proxy and may not find each algorithm’s maximum performance.

E Benchmarking Setup and Hyperparameter Tuning

  • Final experiments used 20 seeds per environment and algorithm, after random hyperparameter configurations were evaluated with two or three seeds each.The small tuning-seed count can over- or underestimate performance; comparisons based on fewer final seeds were often inaccurate or misleading.Source: E Benchmarking Setup and… · p. 25

E.1 Grids for Random Hyperparameter Search

  • Traditional algorithms used predefined random-search grids, with untuned hyperparameters left at their Stable-Baselines defaults.The DQN implementation used Double Q-learning by default, while additional tricks such as prioritized replay were disabled.Source: §E.1 · p. 26

E.1.1 LunarLander-v2 & LunarLanderSparse-v2

  • LunarLander-v2 and LunarLanderSparse-v2 searches covered network architectures, activation functions, and algorithm-specific training and optimization settings.The listed configurations include batch and buffer sizes, learning rates, update settings, and UDRL-specific options.Source: §E.1.1 · p. 26

E.1.2 TakeCover-v0

  • TakeCover-v0 used sampled convolutional and fully connected network architectures, with additional algorithm settings listed for the environment.The excerpt lists alternative channel counts and fully connected layer layouts, along with selected update, replay, and return-scale choices.Source: §E.1.2 · p. 27

E.1.3 Swimmer-v2 and InvertedDoublePendulum-v2

  • Swimmer-v2 and InvertedDoublePendulum-v2 searches included architecture, activation, learning, update, and algorithm-specific control settings.The listed choices include TRPO/PPO update parameters, DDPG exploration and buffer settings, and UDRL batch, replay, and return-scale options.Source: §E.1.3 · pp. 27–28

E.2 Tuned Hyperparameter Values

  • Table 4 reports tuned hyperparameter values, but code, library-version, or hardware differences can vary the obtained results.The setup is described as a proxy for moderate tuning effort and is unlikely to discover each algorithm’s maximum performance.Source: E Benchmarking Setup and… · p. 25, Table 4 · p. 28, §E.2 · p. 28

F.1 SwimmerSparse-v2 results

On SwimmerSparse-v2, UDRL retained much of its performance without modification, using the same hyperparameters as in the dense-reward environment.

F.1 SwimmerSparse-v2 results

  • UDRL retained much of its performance on delayed-reward SwimmerSparse-v2 without modification, using the dense-environment hyperparameters.Source: §F.1 · p. 29

F.2 Sensitivity to Initial Commands

Randomly selected LunarLander agents generally track desired returns, but command sensitivity varies between agents and environments. Higher desired returns can increase variability rather than mean returns, while TakeCover makes precise targets difficult.

F.2 Sensitivity to Initial Commands

  • Randomly selected agents on LunarLander-v2 and LunarLanderSparse-v2 show strong correlations between desired and obtained returns.Figures 7a, 7b, 7e, and 7f show these relationships.Source: §F.2 · p. 29
  • Figure 7c’s LunarLanderSparse-v2 agent obtains returns above 200 for most commands, unlike another agent trained with a different seed.The difference suggests training stochasticity can affect how agents generalize to commands.Source: §F.2 · p. 29
  • The initial desired horizon is difficult to set consistently; the experiments use environment-specific heuristics or the end-of-training value across desired returns.The lifelong experience of an agent can track realistic initial desired horizons and returns, which may depend on the initial state.Source: §F.2 · p. 29
  • TakeCover-v0 makes precise desired returns difficult: stochastic enemy behavior and rising difficulty constrain low returns and make higher mean returns progressively harder.The passage says returns below 200 are not achievable in this setting.Source: §F.2 · p. 29
  • Higher desired returns increase return standard deviation rather than mean returns in the reported evaluation.Each evaluation used 100 episodes, with error bars showing standard deviation from the mean.Source: §F.2 · p. 30

G Software & Hardware

The experiments used a set of open-source software tools and environment-specific package versions, with cloud computing and some local GPU hardware.

G Software & Hardware

  • The setup lists Gym, Matplotlib, NumPy, OpenCV, PyTorch, Ray Tune, Sacred, Seaborn, Stable-Baselines, VizDoom, and VizDoomGym.The passage gives software citations, including Gym [Brockman et al., 2016], PyTorch [Paszke et al., 2017], and Stable-Baselines [Hill et al., 2018].Source: G Software & Hardware · p. 30
  • LunarLander and TakeCover used gym==0.11.0, stable-baselines==2.5.0, and pytorch==1.1.0; continuous-control tasks used Mujoco 1.5.Source: G Software & Hardware · p. 30
  • Experiments ran on cloud Intel Xeon resources; TakeCover used Nvidia P100 GPUs in the cloud or V100 GPUs on local hardware.Each experiment occupied one or two vCPUs and 33% GPU capacity when a GPU was used.Source: G Software & Hardware · p. 30
Loading 1912.02877v2…