Recurrent Environment Simulators

Motivation

• Environment simulators support planning and action by predicting how environments respond to actions, enabling applications across psychology, neuroscience, and reinforcement learning.These applications include sensorimotor contingencies, deliberative planning, predictive state representations, and Monte Carlo planning.Source: §1 · p. 1

Requirements

• Useful simulators must make accurate, temporally and spatially coherent long-term predictions while supporting flexible action sequences, scalability, diverse environments, and high-dimensional inputs.Such properties can enable counterfactual reasoning, intuitive physical reasoning, exploration, episodic control, intrinsic motivation, and hierarchical control.Source: §1 · p. 1

Challenges

• Despite prior advances, simulator generalisation, sensitivity to architecture and training, long-horizon accuracy, and computational efficiency remain poorly understood or limited.Existing models require predicting a high-dimensional image after every action, even when only a later prediction is needed.Source: §1 · p. 1

Contributions

• Building on Oh et al. (2015), the paper develops alternative architectures and training schemes to improve simulator performance and analyze model properties.Oh et al. (2015) is described as the state-of-the-art for long-term prediction in deterministic, discrete-action environments.Source: §1 · p. 1
• The paper introduces a simulator that avoids predicting visual inputs after every action, reducing computational burden, and evaluates simulators across Atari 2600, randomly generated 3D mazes, and 3D car racing.The simulators are also shown to support model-based exploration.Source: §1 · p. 2

2 RECURRENT ENVIRONMENT SIMULATORS

• An environment simulator predicts future observations or state representations from past observations and actions, using recurrent hidden-state dynamics to model environmental evolution.The baseline encodes observations, combines hidden state with actions, decodes predicted frames, and is trained with mean squared prediction error.Source: §2 · p. 2

ACTION-DEPENDENT STATE TRANSITION

• The proposed action-dependent simulator lets actions directly influence state transitions rather than affecting them only indirectly through predicted or observed frames.Graphically, the action-to-prediction link is replaced by an action-to-state link.Source: ACTION-DEPENDENT STATE TRANSITION · p. 3
• Observation-dependent transitions restrict the interval over which predictive ability is learned, while prediction-dependent transitions expose the model to the multi-step regime used during evaluation.Training phases vary warm-up length, prediction length, and transition type because prediction-dependent objectives are more complex.Source: §2 · p. 2, ACTION-DEPENDENT STATE TRANSITION · p. 3
• Long-term accuracy is highest with only prediction-dependent transitions, whereas mixed transitions improve short-term detail when complex games cannot achieve reliable long-term predictions.More consecutive prediction-dependent transitions focus learning on global dynamics but reduce precise frame detail; consecutive-transition percentage should guide training design.Source: ACTION-DEPENDENT STATE TRANSITION · p. 3

PREDICTION-INDEPENDENT STATE TRANSITION

• A prediction-independent simulator evolves state solely from the previous state and action, eliminating frame dependence during usage.For action sequences, this avoids projecting between lower-dimensional state and higher-dimensional observation spaces at every time-step, substantially improving computational efficiency.Source: PREDICTION-INDEPENDENT STATE TRANSITION · pp. 3–4

3 PREDICTION-DEPENDENT SIMULATORS

• The evaluation predicts 100–200 future steps using test actions and separately tests interactive generalization, across Atari games, TORCS, and randomly generated 3D mazes.The first protocol measures performance under the training-data action policy, whereas human interaction probes other action policies.Source: §3 · p. 4, §3.1 · p. 4

3.1 ATARI

• Prediction-dependent transitions improve Atari long-term accuracy, but mixing them with observation-dependent transitions is preferable for overly complex games because increased long-term accuracy reduces short-term sharpness.Using only observation-dependent transitions performs poorly in most games, whereas fully prediction-dependent training can produce insufficiently good predictions in complex games.Source: §3.1 · pp. 5–7, Figure 2 · p. 6, Figure 3 · p. 7

EVALUATION THROUGH HUMAN PLAY

• Human play showed that simulators could generalize to interactive action policies in Pong and Breakout, but mixing training schemes was generally more robust than prediction-dependent-only training.Humans played Pong for 500 steps and Breakout for 350 steps with accurate dynamics and score updates, while prediction-dependent-only models were more fragile to unseen states.Source: EVALUATION THROUGH HUMAN PLAY · p. 10, Figure 7 · p. 11

EVALUATION OF STATE TRANSITIONS STRUCTURES

• Among action-dependent state transitions, higher-dimensional or convolutional structures sometimes improved performance, while local action incorporation was worse than direct global influence.Increasing state dimension from 1024 to 2816 helped independently of parameter-count increases.Source: EVALUATION OF STATE TRANSITIONS… · pp. 10–11

3.2 3D ENVIRONMENTS

• In TORCS and 3D mazes, simulators maintained temporally and spatially coherent predictions for several hundred and 200 steps, respectively, while preserving relevant dynamics and nearby spatial structure.TORCS predictions captured new background features, car motion, and instruments; maze predictions preserved wall layouts and lighting, though long corridors were less coherent.Source: §3.2 · pp. 11–12, Figure 9 · p. 12, Figure 8 · p. 12

3.3 MODEL-BASED EXPLORATION

• The model-based explorer covered 50% more maze area than random exploration after 900 steps using 100 Monte-Carlo simulations of six-action sequences.The selected paths were smoother and enabled faster movement through corridors.Source: §3.3 · p. 12
• The simulator generates salient frames highlighting wall-layout memory after a human player's 360° spin.Source: Figure 10 · p. 13
• Across 10 mazes, the evaluation compares the area visited and typical paths of a random agent with those of an agent using the model.Source: Figure 11 · p. 13
• A global exploration strategy would need better memory to avoid sending the explorer down the same corridor twice, although smooth local exploration may aid navigation.Source: §3.3 · p. 13

• By using state transitions without high-dimensional predictions, the simulator avoids mapping between 1,024-dimensional states and 100,800-dimensional observations, saving around 200 million flops per time-step.Source: §4 · p. 13
• The working structure used z_t = h_t, separate warm-up and prediction parameters, a warm-up length of τ = 10, and gradient backpropagation to time-step five to learn C.Source: §4 · p. 13
• Prediction-independent simulators are more sensitive to state-transition and training changes, with T = 15 performing much worse long term than the prediction-dependent simulator but T = 30 with truncated backpropagation alleviating the problem.Source: §4 · p. 13
• Prediction-independent simulators generally give worse long-term predictions, although they outperform prediction-dependent simulators on some difficult games.Fishing Derby illustrates a failure mode in which the simulator generates additional smaller fish over the long term.Source: §4 · pp. 13–14, Figure 12 · p. 14

Contributions

• The simulator adapts across Atari, 3D car racing, and mazes, achieving state-of-the-art Atari results and feasible live human play in all three task families.It captures complex long-term interactions with spatial and temporal coherence in high-dimensional time-series data.Source: §5 · p. 14

Training trade-offs

• Reducing reliance on past observations focuses capacity on global dynamics and improves long-term predictions, but can harm short-term performance and robustness to unseen states.Guiding noise and teaching the model to use past information through the objective function may improve long-term prediction, while more sophisticated architectures are needed to address the trade-off.Source: §5 · pp. 14–15

Architectural limitations

• Homogeneous architectures struggle with compositional environments, including independently moving ghosts in Ms Pacman and globally coherent navigation in 3D mazes.Specialised functional forms and memory structures may better represent independent entities, their interactions, and global spatial relationships.Source: §5 · p. 15

Data-distribution limitations

• Policy-induced data distributions can oversample some states, undersample policy-relevant regions, and cause models to learn action-sequence correlations instead of true event causes.Experiments suggest this problem is not yet equally pressing across environments, but training-data specificities still produce prediction faults.Source: §5 · p. 15

Future directions

• The deterministic simulator is designed for deterministic environments, so extending the approach to generative temporal models is necessary for noisy real-world state transitions.Most real-world environments involve stochastic transitions that the current approach does not model.Source: §5 · p. 15

A DATA, PREPROCESSING AND TRAINING ALGORITHM

• Each selected action was repeated for 4 time-steps, with only the 4th RGB frame recorded for analysis.Source: A DATA, PREPROCESSING AND TRAINING… · p. 17
• RGB preprocessing subtracted per-channel means computed from the initial 2048 frames and divided every pixel by 255.Source: A DATA, PREPROCESSING AND TRAINING… · p. 17
• Training used centered RMSProp with learning rate4 1e-5, epsilon 0.01, momentum 0.9, decay 0.95, and mini-batches of 16.The model was implemented in Torch using default parameter initialization.Source: A DATA, PREPROCESSING AND TRAINING… · p. 17
• The recurrent model’s initial state s1 was initialized to zero.Source: A DATA, PREPROCESSING AND TRAINING… · p. 17

B PREDICTION-DEPENDENT SIMULATORS

• Single-step simulators used a baseline state transition as the reference for prediction-dependent simulator modeling.Source: B PREDICTION-DEPENDENT SIMULATORS · p. 17
• The prediction-dependent simulator updated input, forget, and output gates, then its cell and hidden state using recurrent equations with h_t-1 and v_t dimensions 1024 and 2048.The updates combine current inputs with the previous recurrent state through sigmoid gates and tanh nonlinearities.Source: B PREDICTION-DEPENDENT SIMULATORS · p. 17

B.1 ATARI

• Training used 5,000,000 frames and testing used 1,000,000 RGB images generated by a trained DQN under an ϵ = 0.2-greedy policy.The large training set was necessary to prevent strong overfitting; with one million frames, the simulator memorized the timing of ghosts returning to chase mode.Source: §B.1 · p. 17
• The Atari simulator used a four-layer convolutional encoder producing a 2,816-dimensional latent vector and a symmetric fully connected and convolutional decoder.The convolutional layers used RReLU activations, with the decoder reversing the encoder’s filter structure.Source: §B.1 · p. 17
• Successful predictions were demonstrated at time-steps 100 and 200 for each Atari game.The examples are shown in Fig. 14.Source: §B.1 · p. 17

B.1.1 SHORT-TERM VERSUS LONG-TERM ACCURACY

• Breakout’s 100%PDT scheme has the lowest long-term error despite often omitting the ball, while max-pooling could improve ball representation.Observation-dependent transitions represent the ball more reliably but predict positions less accurately; prediction-dependent transitions predict paddle movement accurately but usually fail to represent the ball.Source: §B.1.1 · p. 18
• Across Bowling, Freeway, Fishing Derby, and Riverraid, prediction-dependent transitions generally improve long-term dynamics, although short-term accuracy and score prediction can favor observation-dependent schemes.Bowling’s prediction-dependent scheme reduces score and ball-direction errors; Freeway’s 100%PDT rarely loses the chicken; Fishing Derby has disastrous long-term 0%PDT accuracy; Riverraid’s long-term error is lower with 100%PDT and jet-loss errors disappear with BBTT(15,2).Source: §B.1.1 · pp. 18–19
• Ms Pacman and Space Invaders permit accurate prediction for only a few future steps because their dynamics are difficult, while limited exploration further harms ghost prediction in Ms Pacman.Ms Pacman’s ghosts are especially difficult to predict in frightened or later chase modes; 0%PDT fails beyond the short term in both games, and 100%PDT struggles with Space Invaders bullets.Source: §B.1.1 · pp. 18–19
• Pong shows the central trade-off: 100%PDT predicts ball direction more accurately but causes more disappearance and generally less-sharp balls and paddles.With 0%PDT, ball direction is often wrong after impacts, whereas 100%PDT improves direction prediction but increases disappearance after agent hits.Source: §B.1.1 · p. 18
• In Qbert, 0%PDT predicts accurately only over very short horizons, becoming background-only after a few frames, while more prediction-dependent transitions make the agent and moving objects less sharply represented.Source: §B.1.1 · p. 19

Different action-dependent state transitions

• The baseline uses 1024-dimensional h_t−1 and 2048-dimensional v_t vectors, about 25M parameters, prediction length T = 15, and 0%-100%PDT training.Alternatives include an 80M-parameter 2816-dimensional model, separate z_t−1 gating, removal of z_t−1 from gate updates, and modified cell updates.Source: §B.1.2 · p. 35
• No alternative transition is clearly preferable to the baseline overall, except on Fishing Derby, where 2816-dimensional hidden transitions perform better and converge earlier.The comparison covers the nonconvolutional transition variants shown in Figs. 30–34.Source: §B.1.2 · p. 35
• The convolutional alternatives reshape 2816-dimensional z_t−1 and v_t into 32 × 11 × 8 tensors and apply one or two 3×3 convolutions to gates and cell updates.The C and 2C variants use shared convolutional transformations, while CDA and 2CDA use different action-fusion parameters for gates and cell updates.Source: §B.1.2 · pp. 35–36
• The convolutional variants span roughly 8M to 48M parameters while combining convolutional gate, cell-update, and action-fusion designs.The compared families include C/2C, CDA/2CDA, 2C-based 2816-dimensional variants, and convolutional action-fusion models.Source: §B.1.2 · p. 36

B.1.3 ACTION INCORPORATION

• The action-incorporation study compares a baseline multiplicative interaction with alternatives that combine actions additively or multiplicatively with hidden states, encoded frames, or both.Experiments use prediction length T = 15 and the 0%-100%PDT training scheme across ten Atari games.Source: §B.1.3 · pp. 36–37
• The compared action-dependent variants span models with approximately 19M to 51M parameters, including separate gate-and-cell matrices and alternative multiplicative interactions.Reported sizes include around 19M, 22M, 28M, 43M, and 51M parameters.Source: §B.1.3 · pp. 36–37
• CA, which augments the input frame with action channels for a local linear interaction, is generally considerably worse than the other action-dependent structures.The augmented tensor has dimensions (nC + nA) × nH × nW, and the model has around 19M parameters.Source: §B.1.3 · p. 37

ACTION-INDEPENDENT VERSUS ACTION-DEPENDENT STATE TRANSITION

• Action-independent state transitions generally perform worse in higher-error games and cannot model moving Seaquest fish or reliably update scores in Seaquest and Fishing Derby.The comparison uses prediction length T = 15 and the 0%-100%PDT training scheme.Source: ACTION-INDEPENDENT VERSUS… · p. 37
• The action-dependent and action-independent transition comparison is visualized in Fig. 45 using continuous and dashed prediction-error curves, respectively.The figure reports averages over 10,000 sequences and tracks parameter updates in millions.Source: ACTION-INDEPENDENT VERSUS… · p. 37, Figure 45 · p. 53

B.1.4 HUMAN PLAY

• During 2000 steps of human-play Freeway, the model updates the score correctly up to (14,0), then fails to predict the warning-phase score while correctly timing flashing and game reset.The warning-phase error is attributed to dataset bias because DQN always exceeds score 20 at that point.Source: §B.1.4 · p. 54, Figure 46 · p. 54

B.2 3D CAR RACING

• Car-racing data used 10 training and one million testing 180×180 RGB images generated by an asynchronous advantage actor critic agent with three driving actions and episode-wise random ϵ between 0 and 0.5.A fourth do-nothing action was added for random action generation; smaller ϵ produced roughly 1500-frame episodes, whereas larger ϵ produced roughly 200-frame episodes.Source: §B.2 · p. 54
• The car-racing model reused Atari’s convolutional layer, filter, and kernel counts without padding.Source: §B.2 · p. 54
• Predicted 3D car-racing frames remained high-quality and temporally coherent for up to 200 actions, a result reported as common across predictions.Figure 49 compares predicted and real frames over this horizon.Source: §B.2 · p. 54
• The car slightly slowed but failed to stop when selecting no action because training lacked episodes with the accelerator released for several consecutive actions.This limitation reflects an unseen interaction case rather than a general prediction failure.Source: §B.2 · pp. 54–55

B.3 3D MAZES

• 3D-maze agents could generate informative sequences with random policies by alternating 15-step random walks and 15-step spins, encouraging coherent post-spin frame prediction.The random walk used dithering 0.7, and each episode lasted 900 frames.Source: §B.3 · p. 55
• The 3D-maze datasets contained 7,600 training and 1,100 test episodes, totaling 6,840,000 and 990,000 48×48 RGB images, respectively.Source: §B.3 · p. 55
• For 3D mazes, the encoding used three convolutional layers with 64 filters of size 6 × 6, strides of 2, and padding 0, 1, and 2, with the decoder adapted accordingly.Source: §B.3 · p. 55

Monte-Carlo simulations

• Beyond 100 Monte-Carlo simulations, performance changed little because the large number of possible actions caused diminishing returns.The passage attributes the limited benefit of additional simulations to the rapidly expanding action space.Source: §B.4 · p. 55

Action-sequence length

• 0.16 versus 0.37 average prediction error occurred after 6 versus 30 steps, and action sequences beyond d = 6 reduced performance to near-random decisions.Average minimum and maximum distances changed only from 0.23 to 0.36 and from 0.24 to 0.4, respectively, leaving deep simulations with more noise than signal.Source: §B.4 · p. 56

Exploration trajectories

• The explorer selected trajectories that were much smoother than those of the baseline agent.Examples were shown among trajectories chosen by the explorer.Source: §B.4 · p. 56

Comparison setup

• The evaluation compares different action-dependent state transitions and prediction lengths T for the prediction-independent simulator.For the state-transition comparison, Fig. 51 uses T = 15.Source: C PREDICTION-INDEPENDENT SIMULATORS · p. 56, Figure 51 · p. 59

State transitions

• The ’ht−1–iz t 2816–zt−1 = 0’ structure achieves the best performance, although it has a large number of parameters, around 95M.Its vectors ht−1 and vt have dimensionality 2816.Source: C PREDICTION-INDEPENDENT SIMULATORS · p. 57
• The ’Base–zt−1 = 0’ transition performs much worse for long-term prediction than the other transitions and than the prediction-dependent simulator’s baseline transition.The comparison uses prediction error across the Atari games shown in Fig. 51.Source: C PREDICTION-INDEPENDENT SIMULATORS · p. 57, Figure 51 · p. 59

Prediction lengths

• Using longer prediction lengths dramatically improves long-term prediction, with the best performance obtained from two subsequences of length T = 15.The prediction-length comparison uses the ’Base–zt−1 = ht−1’ structure and is shown in Figs. 52 and 53.Source: C PREDICTION-INDEPENDENT SIMULATORS · p. 58, Figure 52 · p. 60, Figure 53 · p. 61