Source-linked AI summary
Reinforcement Learning Upside Down: Don't Predict Rewards -- Just Map Them to Actions
Juergen Schmidhuber
TL;DR
The paper addresses how reinforcement learning can avoid predicting rewards by treating rewards and time horizons as task-defining commands. It proposes Upside Down RL, which learns action mappings through supervised learning and reports that a pilot version can outperform traditional RL methods on certain challenging problems. It also outlines an Imitate-Imitator approach for learning robot behavior from demonstrations.
Problem
Traditional RL predicts rewards and transforms those predictions into actions, motivating an alternative that directly uses desired rewards and horizons as inputs.
Method
UDRL uses supervised learning on past experience to map reward, time-horizon, and other task-defining inputs to actions or action probabilities.
Results
A pilot version of UDRL can outperform traditional RL methods on certain challenging problems.
Takeaways & Limitations
The paper presents direct command-to-action mapping as a way to generalize toward high rewards or other goals, and extends the idea to robot imitation from demonstrations.
Takeaways & Limitations
In practice, the controller can be trained on only finitely many commands, which must be chosen wisely, and success depends on its generalization capability.
Abstract
from arXiv · showhide
We transform reinforcement learning (RL) into a form of supervised learning (SL) by turning traditional RL on its head, calling this Upside Down RL (UDRL). Standard RL predicts rewards, while UDRL instead uses rewards as task-defining inputs, together with representations of time horizons and other computable functions of historic and desired future data. UDRL learns to interpret these input observations as commands, mapping them to actions (or action probabilities) through SL on past (possibly accidental) experience. UDRL generalizes to achieve high rewards or other goals, through input commands such as: get lots of reward within at most so much time! A separate paper [63] on first experiments with UDRL shows that even a pilot version of UDRL can outperform traditional baseline algorithms on certain challenging RL problems. We also also conceptually simplify an approach [60] for teaching a robot to imitate humans. First videotape humans imitating the robot's current behaviors, then let the robot learn through SL to map the videos (as input commands) to these behaviors, then let it generalize and imitate videos of humans executing previously unknown behavior. This Imitate-Imitator concept may actually explain why biological evolution has resulted in parents who imitate the babbling of their babies.
23 June 2020 (based on version v1 of 5 Dec 2019)
This section notes that the work is a minor update of recent work and refers to results from a pilot version.
- It refers to results from a pilot version.
- The result statement is presented only as a fragment in this passage.
- The text identifies the work as a minor update of recent work.
1 Basic Ideas
UDRL reframes reinforcement learning as supervised learning by using rewards, horizons, and related inputs as commands for mapping observations to actions. The paper reports that a pilot version can outperform traditional RL methods on certain challenging problems.
- UDRL takes rewards as inputs rather than predicting them, using commands that specify desired rewards and time horizons.The commands can request a target reward within a specified time.
- UDRL can generalize from self-acquired command knowledge to new requests for more reward in less time.The paper describes commands such as achieving more than half an observed reward.
- A pilot version of UDRL can outperform traditional RL methods on certain challenging problems.
- The controller learns through supervised learning to map command-augmented observations to known action sequences or action probabilities.Training uses past experience, including retrospectively generated commands for observed intervals.
- The approach is intended to use accumulated experience to solve user-given problems under resource constraints such as time, energy, and pain.An example is reaching a charging station quickly while avoiding obstacles.
2 Notation
The notation defines variables, trials, inputs, traces, and controller outputs for an agent interacting with an environment over discrete time steps. Inputs combine sensory, reward or cost, prior-action, and task-defining information, while outputs select actions or action distributions.
- The notation uses positive integer constants m, n, o, p, q, u and positive integer variables h, i, j, k, t, τ.The i-th component of a real-valued vector v is denoted v_i.
- An agent controller may be a recurrent neural network operating across multiple trials or one lifelong trial in an initially unknown environment.Each trial consists of discrete time steps t = 1, 2, . . . , T.
- At time t, the controller receives sensory input, a vector-valued cost or reward, the previous action, and task-defining horizon inputs.Sensory input may include pixels, while costs can represent pain, hunger, or external rewards.
- The controller computes an output vector used to select a final action, either as discrete-action probabilities or parameters of a Gaussian distribution for real-valued actions.The latter supports sampling high-dimensional actions such as multi-joint robot controls.
- all(t) concatenates the previous action, current sensory input, and current cost or reward vector, while trace(t) records these inputs through time t.
3 Deterministic Environments With Markovian Interfaces
In deterministic Markovian environments, UDRL uses supervised replay to map observations, desired rewards, and time horizons to actions, learning multiple cost-aware policies from one behavioral trace. The method avoids discount-factor distortions, supports probabilistic and high-dimensional actions, and can trade training cost against forgetting through selective replay.
- Probabilistic Policies: In deterministic Markovian settings, a feedforward network is sufficient, while mean-squared-error training over one-hot actions estimates conditional action probabilities.Different equally costly action sequences can therefore produce probabilistic policies, such as a 50% choice between two paths.
- Replay-Based Training: UDRL trains a controller by replaying every compatible pair of times, mapping observations, remaining horizons, and cumulative costs or rewards to previously executed actions.Algorithms A1 and A2 alternate environment interaction with gradient-based replay training over past behavior.
- Probabilistic Policies: Supervised learning also accommodates high-dimensional continuous actions, including probability distributions with dependent components for multi-joint robots.The paper contrasts this with traditional RL, which it says does not easily handle such actions.
- Time Horizons: Unlike discounted formulations, the approach represents observed time horizons directly and does not assume infinite horizons or distort basic rewards.The paper describes discount factors as exponentially shrinking real rewards while its method uses explicit horizon representations.
- Complexity and Retention: Replay requires O(t(t + 1)/2) time per training epoch, and discarding less-rewarding histories can reduce effort while risking forgetting commands for low rewards.The selective-replay trade-off favors recent rewarding behavior as performance improves.
- Replay-Based Training: A single 1000-step trial yields roughly half a million training examples, exposing fine-grained relationships between time, costs, rewards, and actions.Replay can capture small incremental changes as well as decisive moments such as the end of a board game.
4 Other Properties of the History as Command Inputs
UDRL augments reward-and-horizon commands with threshold, goal-state, and other history-compatible inputs, allowing one experience trace to support richer task specifications. In practice, the command set must remain finite and be chosen strategically.
- 4 Other Properties of the History as Command Inputs: Threshold commands let C generate action trajectories achieving more than a specified reward within a specified time.These commands are trained from past experiences that retrospectively match the requested condition.
- 4 Other Properties of the History as Command Inputs: Activating the morethan input and encoding the best conditional reward enables C to generalize toward goal-directed exploration.
- 4 Other Properties of the History as Command Inputs: Additional inputs can require obtaining a reward within a time limit and finally reaching a specified state, including a previously unreached goal location.
- 4 Other Properties of the History as Command Inputs: Although infinitely many computable history predicates could define commands, practical training can use only finitely many, so command selection matters.
- 4 Other Properties of the History as Command Inputs: Unlike UDRL's retrospective command discovery, POWERPLAY can actively set tasks that require new environmental interaction.
5 Probabilistic Environments
For probabilistic environments, UDRL can encode expected rewards as commands and train actions from observed reward estimates under Markov assumptions. Dynamic programming extends this treatment when randomness also affects next states, while time representations offer an alternative for partial observability.
- 5 Probabilistic Environments: UDRL estimates expected immediate rewards for state-action pairs and maps desired expected rewards to actions rather than using individual stochastic rewards.
- 5 Probabilistic Environments: Replay training encodes each future horizon and the corresponding cumulative expected reward estimated from visited state-action pairs.
- 5 Probabilistic Environments: Dynamic programming can estimate cumulative expected rewards when randomness affects both immediate rewards and resulting states, while preserving the method’s overall computational order.
- 5 Probabilistic Environments: A unique current-time representation can instead let C learn concrete reward dependence on time, avoiding misleading updates from a few lucky experiences.
6 Partially Observable Environments
In partially observable settings, recurrent controllers use history to represent the current state and can interpret increasingly complex commands. This enables deterministic high-dimensional actions in single-life settings, but makes success depend strongly on generalization across time.
- 6 Partially Observable Environments: RNN-based Algorithms B1/B2 replace feedforward state handling with history-dependent internal representations and synchronized replay training.
- 6 Partially Observable Environments: Replay over time pairs trains actions against observed cumulative costs while retaining compatible additional commands and quadratic-per-epoch complexity.
- 6.1.2 Arbitrarily Complex Commands for RNNs as General Computers: RNNs can execute essentially arbitrary computable task specifications compatible with observed traces, including reward, timing, visitation-count, and state goals.
- 6.1.3 High-Dimensional Actions: For high-dimensional actions, supervised learning can generate deterministic action vectors directly instead of relying on probabilistic assumptions.
- 6.1.3 High-Dimensional Actions: Component-by-component micro-time steps let action probabilities depend on components selected earlier, handling dependencies among high-dimensional action components.
- 6.1.4 Computational Power of RNNs: Generalization & Randomness vs. Determinism: In single-life settings, performance depends on learning when precise timestamps matter and when similar situations should generalize across time.
7 Compress Successful Behaviors Into a Compact Standard Policy Network Without Command Inputs
After command-conditioned learning discovers successful behaviors, a smaller policy network can imitate those behaviors without command inputs. This compression supports deployment of policies specialized to the successful solutions retained in training.
- 7 Compress Successful Behaviors Into a Compact Standard Policy Network Without Command Inputs: The complex command-conditioned mapping can be compressed into a smaller policy network that maps sensory history directly to actions.
- 7 Compress Successful Behaviors Into a Compact Standard Policy Network Without Command Inputs: Algorithm Compress trains recurrent policy network CC on previously observed action sequences from trials deemed successful.
- 7 Compress Successful Behaviors Into a Compact Standard Policy Network Without Command Inputs: Compression can collapse satisfactory solutions across different initial conditions into a policy that ignores reward and time commands.
8 Imitate a Robot, to Make it Learn to Imitate You!
Imitate-Imitator teaches a robot to associate human demonstrations with its own behaviors, then generalize from those learned video commands to previously unknown behaviors. The procedure repeatedly records, imitates, and trains on paired videos and behavior traces.
- Imitate-Imitator: The robot first executes known or accidental behavior, humans imitate it, and the robot learns through supervised learning to map the recorded imitation video back to that behavior.This bootstraps imitation despite differences between the human body and the robot’s joints and hands.
- Imitate-Imitator: After learning several video commands, the robot can generalize by using a video of behavior it has never previously performed as a new command.Additional human imitations can provide training data when the robot’s imitation is unsatisfactory.
- Imitate-Imitator: The training loop demonstrates behavior visually, lets the robot respond to the video, records the human imitation, and retrains the controller on each video-behavior pair.Each pair is interpreted as a sequential command for producing the corresponding behavior under its observed costs.
- Extensions: The approach extends beyond videos to spoken commands, gestures, text, and other sequential multimodal descriptions of desired behavior.These descriptions can be mapped to the robot’s corresponding behaviors through the same command-to-action framework.
- Biological connection: The paper connects this show-and-tell robotics idea to biological evolution by suggesting that parents’ imitation of babies’ babbling may teach babies to translate observed behavior into their own actions.The proposed connection is presented as a possible explanation, not as an experimentally established result.
9 Relation of Upside Down RL to Previous Work
UDRL differs from prior reinforcement-learning and replay approaches by using desired rewards and horizons as command inputs, directly mapping them to actions without requiring reward prediction. Its single-trial operation and generalization over computable task specifications distinguish it from several related systems.
- Relation to earlier systems: Unlike earlier systems that use reward signals as inputs, UDRL supplies desired rewards as command inputs and does not depend on a predictive world model for reward gradients.Earlier systems described in the paper learn mappings from observed costs or rewards to actions but typically use a reward-predicting model.
- Relation to hierarchical reinforcement learning: Compared with hierarchical reinforcement learning, UDRL does not require an evaluator that predicts costs or rewards from start-goal combinations to derive subgoals.The distinction is specifically about eliminating additional reward prediction from the learning process.
- Relation to hindsight methods: Unlike hindsight experience replay, UDRL uses real-valued horizon and cost inputs and computable predicates, allowing generalization from observed costs to desirable costs.The paper also contrasts UDRL’s recurrent handling of partial observability with HER’s controller.
- Relation to reinforcement learning: UDRL directly maps desired rewards, time horizons, and other inputs to actions, avoiding the reward-prediction stage used by traditional reinforcement learning.The method learns this mapping through gradient-based supervised learning from observed experience.
- Relation to lifelong learning: UDRL can operate during a single lifelong trial, using new observations immediately to improve behavior rather than requiring multiple independent trials.The paper contrasts this with POWERPLAY’s task-ordering and replay mechanisms.
- Limitation: The synchronization scheme does not immediately guarantee preservation of previous-task performance because synchronizations may be separated by limited time and gradient descent has basic limitations.The paper says long-run retraining is expected to help prevent forgetting, but the immediate guarantee is absent.
10 Experiments
The paper points to a separate study for concrete implementations and reports that the initial UDRL pilot produced remarkable experimental results.
- Experimental outcome: The separate experimental paper reports that an initial UDRL pilot version outperformed traditional reinforcement-learning methods on certain challenging problems.The supplied passage identifies the outcome but does not provide task names or numerical values.
11 Conclusion
The conclusion presents UDRL as a shortcut from task commands to actions, relying on neural-network generalization rather than reward prediction or explicit expected-reward maximization. It also extends the same command-learning idea to robot imitation.
- Upside Down RL: UDRL maps task specifications such as obtaining much reward within little time directly to action probabilities without predicting rewards or explicitly maximizing expected rewards.Its success is stated to depend on the generalization abilities of deep or recurrent neural networks.
- Limitations: The approach retains the usual gradient-based-learning risks, including local minima, underfitting, and overfitting.These are presented as potential drawbacks of the method’s learning strategy.
- Empirical result: Experiments in a separate paper show that the initial UDRL pilot can outperform traditional reinforcement-learning methods on certain challenging problems.The conclusion reports this as an empirical result without specifying the benchmark values here.
- Imitation learning: The related Imitate-Imitator approach trains a robot to map observations of human imitation to its own behavior, then generalize from demonstrations of new behavior.This applies the command-to-action perspective beyond reward-based task specifications.