Source-linked AI summary
Action-Conditional Video Prediction using Deep Networks in Atari Games
Junhyuk Oh, Xiaoxiao Guo, Honglak Lee, Richard Lewis, Satinder Singh
TL;DR
The paper addresses action-conditional prediction of high-dimensional video for vision-based reinforcement learning, where future frames depend on both prior frames and actions. It proposes deep encoding, action-conditional transformation, and decoding architectures, and finds visually realistic, control-useful predictions over approximately 100-step futures in some Atari games.
Problem
Action-conditional video prediction remains difficult for high-dimensional visual environments with complex object dynamics, object entry and departure, and partial observability.
Method
The paper proposes two deep architectures that encode visual histories, transform features using actions, and decode them into predicted frames.
Results
The architectures generate visually realistic and useful-for-control predictions over approximately 100-step action-conditional futures in some Atari games.
Takeaways & Limitations
Predicted frames can support control evaluation by replacing emulator frames and can guide informed exploration for DQN.
Takeaways & Limitations
The models have difficulty accurately predicting small objects and stochastic object appearances, so generated frames may not match ground truth.
Abstract
from arXiv · showhide
Motivated by vision-based reinforcement learning (RL) problems, in particular Atari games from the recent benchmark Aracade Learning Environment (ALE), we consider spatio-temporal prediction problems where future (image-)frames are dependent on control variables or actions as well as previous frames. While not composed of natural scenes, frames in Atari games are high-dimensional in size, can involve tens of objects with one or more objects being controlled by the actions directly and many other objects being influenced indirectly, can involve entry and departure of objects, and can involve deep partial observability. We propose and evaluate two deep neural network architectures that consist of encoding, action-conditional transformation, and decoding layers based on convolutional neural networks and recurrent neural networks. Experimental results show that the proposed architectures are able to generate visually-realistic frames that are also useful for control over approximately 100-step action-conditional futures in some games. To the best of our knowledge, this paper is the first to make and evaluate long-term predictions on high-dimensional video conditioned by control inputs.
1 Introduction
The paper targets action-conditional video prediction for vision-based RL, using Atari games as a high-dimensional, partially observable testbed. It proposes deep architectures that generate realistic long-horizon frames and support control-related evaluations.
- Action-conditional prediction is important for vision-based RL because future observations depend on both previous frames and agent actions.The resulting predictive model represents aspects of agent-environment dynamics relevant to model-based RL.
- The paper proposes and evaluates two deep spatio-temporal architectures that incorporate action variables.The architectures are based on encoding, action-conditional transformation, and decoding components.
- The architectures generate realistic frames over 100-step action-conditional futures without diverging in some Atari games.
- The learned representations approximately capture action similarity and distinguish directly controlled objects from indirectly influenced or uncontrolled objects.
- The paper evaluates usefulness for control by replacing emulator frames in DQN and using predictions to guide exploration.These evaluations test both prediction-based play and prediction-informed data collection.
2 Related Work
Prior work developed deep models for video prediction, Atari reinforcement learning, and action-conditional prediction. This paper extends action-conditional evaluation to higher-dimensional visual data with more complex dependencies.
- Earlier video-prediction models learned temporal correlations or multiplicative relationships from sequential visual data, often in relatively limited settings.
- Deep RL work on Atari combined Q-learning with convolutional networks, while other approaches used emulator-based predictions to improve action selection.
- Earlier action-conditional predictive models targeted attention regions or robot coordinates rather than high-dimensional game frames.
- The paper evaluates action-conditional prediction on higher-dimensional data with complex dependencies among observations.
3 Proposed Architectures and Training Method
The proposed model maps frame histories and actions to future frames through spatial-temporal encoding, multiplicative action conditioning, and decoding. Training uses multi-step prediction and a curriculum to stabilize long-horizon forecasts.
- Overall architecture: The architectures learn f: x_1:t, a_t → x_t+1 by encoding frame histories, transforming features conditionally on actions, and decoding predicted features into pixels.
- Feedforward encoding: Feedforward encoding concatenates a fixed history of frames across channels and extracts spatio-temporal features with stacked convolutions.This is an early-fusion representation of the input frames.
- Recurrent encoding: Recurrent encoding processes one frame per time step, using convolutional features as input to an LSTM that captures temporal correlations and retains deep history.
- Action-conditional transformation: Multiplicative interactions combine encoded features with action vectors so different actions can induce different feature transformations.A factorized approximation shares weights across actions through a lower-dimensional factor space when common dynamics exist.
- Decoding: Deconvolution layers map transformed feature vectors back to image pixels after reshaping the representation into a three-dimensional feature map.
- Training: Multi-step training addresses compounding prediction errors by repeatedly feeding predictions forward, while curriculum learning increases the prediction horizon after shorter-horizon training converges.The curriculum was found necessary to stabilize training.
4 Experiments
The experiments evaluate action-conditional video prediction through qualitative frame generation, pixel error, control performance, and learned representations. The proposed architectures produce realistic long-horizon predictions and outperform baselines in several evaluation settings, while feedforward and recurrent encodings show complementary strengths.
- Experimental goals: The experiments assess predicted frames qualitatively and quantitatively, their usefulness for DQN control, and the representations learned by the architectures.Evaluation includes generated videos, 100-step mean squared error, emulator-based control, informed exploration, and action-representation analysis.
- Qualitative evaluation: The proposed models generate qualitatively reasonable predictions over 30–500 steps depending on the game, while MLP quickly diverges and naFf fails to predict the controlled object.In Freeway, the proposed architectures predict the controlled object’s boundary-crossing reappearance, whereas MLP and naFf do not.
- Quantitative evaluation: The predictive models outperform both baselines on mean squared error over 100-step predictions across all domains.The advantage over naFf is relatively small except in Seaquest because the action-controlled object occupies only a small part of the image.
- Encoding comparison: Feedforward encoding predicts precise controlled-object motion better in one case, whereas recurrent encoding better captures delayed enemy movement requiring longer temporal dependencies.A 1–2 pixel recurrent translation error can produce divergent later frames, while feedforward encoding misses movement occurring after nine steps when given only the last four frames.
- Control evaluation: Replacing true frames with predictions lowers DQN scores, but repeatedly predicting only a few steps yields scores close to using real frames.For deeper predictions, the proposed architectures produce much better scores than the two baselines, likely because they better predict controlled-object movement.
- Informed exploration: Informed exploration improves DQN performance using the predictive model in three of five games, with the most significant improvement in QBert.The strategy uses predictions to guide exploration rather than relying only on random actions.
5 Conclusion
The paper presents two deep architectures for action-dependent future-frame prediction and reports visually realistic, control-useful predictions over 100-step futures in several Atari games.
- The architectures distinguish action-related from non-action-related visual factors in learned representations.The paper illustrates this distinction by separating high-variance action factors from low-variance factors.
- Two novel deep architectures predict future frames conditioned on actions over long Atari-game futures.The paper reports both qualitative and quantitative evaluation across several Atari domains.
A Network Architectures and Training Details
The appendix describes feedforward and recurrent network architectures, including convolutional components, factored transformations, and training schedules.
- Training details: Training uses LSTM weights initialized uniformly in [−0.08, 0.08] and separate initialization ranges for encoded-feature and action connections.The encoded-feature and action connections use uniform ranges [−1, 1] and [−0.1, 0.1], respectively.
- Training details: Each training phase runs for 1.5 × 10^6, 10^6, and 10^6 iterations for 1-step, 3-step, and 5-step prediction.The learning rate is multiplied by 0.9 after every 10^5 iterations.
- Network architectures: The architectures use convolutional feature maps and 2048-dimensional intermediate layers before decoding.The listed diagrams include progressively transformed feature-map sizes and repeated 2048-dimensional layers.
- Network architectures: The proposed models include feedforward and recurrent architectures illustrated with convolutional and factored layers.Element-wise multiplication is used in the factored architecture diagrams.
B Informed Exploration
The informed-exploration appendix uses a predictive model and compares exploration behavior and DQN learning curves against emulator-based and random exploration.
- Algorithm 1 describes DQN combined with the informed exploration method.The method is evaluated using predicted frames for exploration.
- The appendix uses a feedforward encoding network trained on down-sampled gray-scaled images for computational efficiency.Figure 10 illustrates this encoding network.
- Figure 11 compares Freeway and Ms Pacman trajectories over 2500 steps for two exploration strategies.Each heat map represents agent trajectories measured from the corresponding exploration method.
- Figure 12 plots DQN learning curves with standard errors for predictive-model, emulator, and random exploration.Average game score is measured from 100 game plays using an ϵ-greedy policy with ϵ = 0.05.
- Table 2 reports average game scores with standard error for DQN combined with informed exploration using emulator or predictive-model frames.The table labels the informed-exploration condition as I.E.
C Correlation between Actions
Figure 13 visualizes correlations between action-related factors using brightness to represent cosine similarity.
- Brightness in the action-correlation visualization represents cosine similarity between pairs of factors.
D Handling Different Actions
Figure 12 compares predicted frames under different action sequences, including action labels and previous-frame context.
- The displayed comparisons include a previous frame and action sequences containing fire inputs.
- Figure 12 presents predictions conditioned on different actions.
E Prediction Video
The prediction videos show that the proposed models produce action-sensitive and qualitatively reasonable long-horizon frames across several Atari games. Their accuracy varies by object, game, and prediction horizon, with some models maintaining object tracking for hundreds of steps.
- The models predict the controlled submarine and enemy movement in Seaquest, while naFf loses the submarine and MLP produces only mean-image frames.
- Over 180-step Seaquest predictions, the proposed models accurately locate the controlled object and generate realistic-looking new objects, although they do not match ground truth.
- In Space Invaders, long-horizon predictions remain realistic despite errors, whereas MLP and naFf frames remain almost identical to the last input frame.
- The feedforward network diverges at 294 steps in Freeway when a new stage begins and actions are ignored for 9 steps.
- The recurrent encoding model tracks every object over 500 Freeway steps, while the QBert recurrent model predicts the controlled object and cube colors through 68 steps.
- The proposed models predict controlled-object movement differently under different actions, unlike baselines that ignore actions or generate mean-image predictions.
- In long-horizon Ms Pacman predictions, the models track Pacman but fail to predict other objects that move almost randomly.