Source-linked AI summary
Learning to Act by Predicting the Future
Alexey Dosovitskiy, Vladlen Koltun
TL;DR
Reinforcement learning still faces challenges in controlling complex three-dimensional environments from raw sensory input and learning skills for changing goals. The paper predicts future measurements from sensory input, measurements, goals, and actions using supervised learning from raw experience. In Doom-based experiments, the approach outperformed sophisticated deep RL methods on challenging tasks, generalized across environments and goals, and won a competition track in previously unseen environments.
Problem
Reinforcement learning still faces challenges in sensorimotor control from raw sensory input in complex three-dimensional environments and in flexibly pursuing dynamically specified goals.
Method
The approach uses the temporal structure between high-dimensional sensory streams and lower-dimensional measurements to predict future measurements conditioned on actions and goals, training by supervised learning from raw experience.
Results
The approach outperformed state-of-the-art deep RL models, particularly on complex tasks, generalized across environments and goals, and won Visual Doom’s Full Deathmatch track by more than 50% over the second-best submission.
Takeaways & Limitations
Dynamically specified goals can be pursued at test time even when training uses no fixed goal, with vectorial measurements providing an advantage over scalar rewards.
Takeaways & Limitations
The model is purely reactive, lacks explicit memory and test-time retention of internal representations, and was developed for discrete action spaces.
Abstract
from arXiv · showhide
We present an approach to sensorimotor control in immersive environments. Our approach utilizes a high-dimensional sensory stream and a lower-dimensional measurement stream. The cotemporal structure of these streams provides a rich supervisory signal, which enables training a sensorimotor control model by interacting with the environment. The model is trained using supervised learning techniques, but without extraneous supervision. It learns to act based on raw sensory input from a complex three-dimensional environment. The presented formulation enables learning without a fixed goal at training time, and pursuing dynamically changing goals at test time. We conduct extensive experiments in three-dimensional simulations based on the classical first-person game Doom. The results demonstrate that the presented approach outperforms sophisticated prior formulations, particularly on challenging tasks. The results also show that trained models successfully generalize across environments and goals. A model trained using the presented approach won the Full Deathmatch track of the Visual Doom AI Competition, which was held in previously unseen environments.
1 INTRODUCTION
The paper addresses sensorimotor control from raw sensory input in complex three-dimensional environments and flexible goal pursuit. It replaces scalar-reward learning with prediction from temporally coupled sensory and measurement streams, then evaluates the approach in Doom.
- Raw sensory control and flexible deployment across dynamically specified goals remain central challenges for reinforcement learning.
- The approach predicts future measurements conditioned on current sensory input, measurements, goals, and actions, reducing control to supervised learning from raw experience.The sensory stream may include visual, auditory, and tactile input, while measurements describe state-relevant quantities such as health or ammunition.
- Temporally dense, multidimensional measurements provide richer supervision than an occasional scalar reward and may stabilize and accelerate training.
- The formulation supports training without a fixed goal and pursuing dynamically specified goals at test time by conditioning predictions on the current goal.
- In Doom simulations, the model learned from raw visual input and in-game statistics without human gameplay or extraneous supervision.
- The approach outperformed state-of-the-art deep RL models on complex tasks, generalized across environments and goals, and won Visual Doom’s Full Deathmatch track by more than 50% over the second-best submission.
2 BACKGROUND
The background contrasts supervised learning with reinforcement-learning approaches to acting in environments. It also situates the paper among work on vector-valued feedback, transferable goals, three-dimensional control, and future-state prediction.
- Prior analysis suggested choosing supervised or reinforcement learning according to the environment’s feedback characteristics.Reinforcement learning may be more efficient with sparse scalar rewards, whereas supervised learning can benefit from temporally dense multidimensional feedback.
- Temporal-difference learning was argued to be preferable to supervised learning when prediction correctness is revealed many steps later, under sparse scalar rewards.
- The paper relates to Monte Carlo prediction methods, whose convergence was considered theoretically advantageous with function approximators, while temporal-difference methods were selected on practical empirical grounds.
- Related research covers vector-valued feedback, transfer across tasks, parameterized goals, and value-function sharing across states and goals.
- Deep reinforcement learning has been applied to three-dimensional environments including TORCS and navigation tasks, extending earlier Atari-game results.
- Future-state prediction spans dynamical systems, generalized value functions, and future-frame prediction, while full sensory prediction in realistic three-dimensional environments remains open.
3 MODEL
The model predicts future measurement changes from current observations, actions, and goals, then selects actions according to predicted outcomes. Its architecture combines separate sensory, measurement, and goal processing with expectation and action-specific prediction streams.
- 3 MODEL: Observations combine raw sensory input with measurements, such as images alongside health, ammunition, or score.The measurement vector is the subset of sensations the model aims to predict.
- 3 MODEL: Goals are defined by desired future-measurement differences, with the goal vector parameterizing a utility function over those differences.The goal vector has the same dimensionality as the future-measurement difference vector.
- 3 MODEL: The predictor F maps the current observation, candidate action, and goal to predicted future measurements, and the agent selects the action with the best predicted outcome.The test-time goal vector may differ from every goal encountered during training.
- 3.1 TRAINING: Training uses experience tuples containing observations, actions, goals, and future-measurement targets, optimized with a regression loss.Experiences are collected through agent-environment interaction and stored in an experience memory for sampling.
- 3.1 TRAINING: Training supports either a fixed single goal or randomly generated episode goals, using an ε-greedy policy during interaction.The exploration probability ε starts at 1 and decreases according to a fixed schedule.
- 3.2 ARCHITECTURE: The network separately processes image, measurement, and goal inputs, concatenates them into a joint representation, and predicts outcomes through expectation and action streams.The expectation stream models average future measurements, while the normalized action stream models differences among actions before their outputs are combined.
4 EXPERIMENTS
Experiments evaluate DFP in four increasingly difficult Doom scenarios against deep RL baselines, then test generalization across environments and goals. DFP performs especially strongly on complex tasks and supports generalization from varied visual environments and training goals.
- Setup: The evaluation uses four Doom scenarios of increasing difficulty, spanning health gathering, navigation, and combat in maze environments.D3 and D4 add adversaries, health and ammunition collection, and richer action demands.
- Setup: DFP, DQN, A3C, and DSR are compared using a future-predictor network configured close to DQN for a fair comparison.The experiments use raw visual input and player-visible game statistics, without human gameplay.
- Comparison to prior work: 97.5%: A3C and DFP achieve virtually identical Basic-scenario performance, while DQN reaches 89%; DFP leads Navigation by 25 percentage points during testing.In D1 and D2, DFP uses only the health measurement.
- Comparison to prior work: 6× and 2.5×: DFP outperforms A3C at test time in D3 and D4, respectively, where three measurements are available per time step.The advantage is largest in the more complex Battle and Battle 2 scenarios.
- Generalization across environments: Training on varied textures yields very good cross-environment generalization, whereas training only in simple D3 does not provide sufficient appearance invariance.The higher-capacity D4-tx-L model performs even better.
- Generalization across goals and ablations: Models trained without a fixed goal perform nearly as well as goal-specific models on the main task, while varied-goal training generalizes much better to new goals.The study also finds that predicting multiple measurements at multiple future times improves performance over scalar or single-offset variants.
5 DISCUSSION
The approach adapts supervised learning to sensorimotor control from raw sensory input and intrinsic measurements, using experience itself as supervision. Its multivariate predictions support challenging tasks and new goals, while several extensions remain open.
- Predicting all measurements at all temporal offsets yields the best results.
- The approach learns to act in complex, dynamic three-dimensional environments from raw sensory input and intrinsic measurements.
- Multivariate measurements outperform conventional scalar rewards and support pursuing goals not specified during training.
- The model is purely reactive, with no explicit memory or test-time retention of internal representations.
- Future extensions include memory, temporal abstraction, hierarchical skills, continuous actions, and learned features from rich sensory input.
A.1 NETWORK ARCHITECTURES
The appendix describes basic and large network variants and evaluates architectural modifications. Removing normalization, the expectation/action split, or input measurements reduces performance, while current measurements are helpful but not essential.
- Network variants: The basic network closely follows Mnih et al. (2015), whereas the large network doubles the width of layers starting from the third.
- Network variants: The leaky ReLU nonlinearity is applied after each nonterminal layer in all networks.Weights are initialized using the method of He et al. (2015).
- Ablations: Removing normalization, the expectation/action split, or input measurements hurts performance in the D3-tx regime.
- Ablations: The two-stream formulation is beneficial, while providing current measurements increases performance but is not crucial.
A.2 OTHER DETAILS
The experiments use minimally processed visual observations, normalized measurements, frame skipping, multiscale future predictions, and replay-based optimization. These choices define the temporal resolution, prediction horizon, and training schedule.
- The agent receives grayscale images without additional preprocessing at 84×84 pixels for the basic model and 128×128 pixels for the large model.
- Measurements are normalized by random-exploration standard deviations: ammo by 7.5, health by 30.0, and frags by 1.0.
- The agent selects an action every 4th frame and repeats it during skipped frames, making one agent step equivalent to 114 milliseconds of real time.
- Future measurements are predicted at offsets 1, 2, 4, 8, 16, and 32 steps, with the longest offset covering 3.66 seconds of real time.
- Training uses a replay memory of M = 20,000 steps, mini-batches of N = 64, and Adam optimization with an initial learning rate of 10^-4.
B BASELINES
The approach is compared with DQN, DSR, and A3C using reward definitions matched to the evaluated scenarios.
- The comparison includes DQN, DSR, and A3C, using the authors’ implementations or an independent A3C implementation.
- Health change is used as reward for scenarios D1 and D2, while D3 and D4 use a linear combination of changes in three normalized measurements.