Source-linked AI summary
Strategic Attentive Writer for Learning Macro-Actions
Alexander, Vezhnevets, Volodymyr Mnih, John Agapiou, Simon Osindero, Alex Graves, Oriol Vinyals, Koray Kavukcuoglu
TL;DR
Reinforcement-learning controllers often do not exploit structure in control sequences, motivating automatic discovery of useful temporal abstractions. STRAW learns action and commitment plans end to end, improves performance on several planning-oriented Atari games, and learns frequent n-grams in text prediction.
Problem
Reinforcement-learning agents generally do not exploit structure in control sequences, despite the potential value of automatically discovering macro-actions for exploration and learning.
Method
STRAW is a recurrent architecture that maintains an action plan and commitment plan, learning temporally abstracted macro-actions end to end from environmental rewards.
Results
STRAW achieves the highest score on 6 of 8 Atari games, exceeds the LSTM score by more than 6× on Frostbite, and learns frequent n-grams in text prediction.
Takeaways & Limitations
The architecture supports structured exploration, reduced computation during plan commitment, and temporal abstraction across reinforcement-learning and sequence-prediction tasks.
Takeaways & Limitations
STRAW assumes that one observation can reveal enough information to generate a sequence of actions, whose complexity and length may vary substantially.
Abstract
from arXiv · showhide
We present a novel deep recurrent neural network architecture that learns to build implicit plans in an end-to-end manner by purely interacting with an environment in reinforcement learning setting. The network builds an internal plan, which is continuously updated upon observation of the next input from the environment. It can also partition this internal representation into contiguous sub- sequences by learning for how long the plan can be committed to - i.e. followed without re-planing. Combining these properties, the proposed model, dubbed STRategic Attentive Writer (STRAW) can learn high-level, temporally abstracted macro- actions of varying lengths that are solely learnt from data without any prior information. These macro-actions enable both structured exploration and economic computation. We experimentally demonstrate that STRAW delivers strong improvements on several ATARI games by employing temporally extended planning strategies (e.g. Ms. Pacman and Frostbite). It is at the same time a general algorithm that can be applied on any sequence data. To that end, we also show that when trained on text prediction task, STRAW naturally predicts frequent n-grams (instead of macro-actions), demonstrating the generality of the approach.
1 Introduction
STRAW learns temporally extended macro-actions by maintaining and selectively updating multi-step action plans from environmental rewards. The architecture supports structured exploration, reduced computation during commitment, and sequence prediction beyond reinforcement learning.
- Architecture: STRAW maintains a multi-step action plan, periodically replanning from observations and committing to the plan between decision points.Replanning decisions and recurring action sequences are learned from rewards.
- Benefits: Macro-actions can make exploration more structured by extending learned action patterns across multiple steps in the state space.The paper also relates macro-actions to possible transfer and generalization when related problems share action-effect structure.
- Benefits: Commitment lets the model avoid processing observations while following a plan, allocating computation to key moments.This reduces computation during periods when the current plan is being followed.
- Evaluation: STRAW is evaluated on Atari games requiring longer-term planning and on text prediction, where it learns frequent n-grams as macro-actions.The text-prediction result demonstrates that the architecture applies beyond reinforcement-learning control.
2 Related Work
Prior temporal-abstraction methods commonly use options with explicit subgoals or pseudo-rewards, while STRAW learns macro-actions and their policy end to end from environment rewards.
- Options: The options framework defines temporally extended sub-policies with termination conditions and a policy that selects among them.Macro-actions are a simpler option instance whose sequence is decided when initiated.
- Prior limitations: Existing deep-learning approaches combine predefined subgoals with learning, but discovering subgoals remains unresolved; another jointly learned option method was tested only on a toy problem.These limitations motivate learning temporal abstractions directly from task rewards.
- STRAW: STRAW learns macro-actions and a policy over them end to end from environment rewards without explicit pseudo-rewards or hand-crafted subgoals.Its macro-actions arise implicitly from the interaction between action and commitment plans.
- Scope: The approach is demonstrated across next-character prediction and Atari games, indicating application across distinct sequence tasks.
3 The model
STRAW uses attentive recurrent plans for future actions and commitment durations, updating them from observations and executing planned actions without replanning between commitment points. A noisy variant adds multi-step randomness for structured exploration.
- The model: STRAW has an action-plan module and a commitment-plan module that respectively represent future actions and when to terminate a macro-action.The action-plan is an action-by-time matrix, while the commitment plan gives termination probabilities across time.
- The model: The action-plan stores probabilities over discrete actions and future time steps, with the current action sampled from its first time column.
- Commitment: A gate determines whether STRAW updates the plans or commits to them; macro-actions are the action sequences between successive plan updates.During commitment, the time-shift operator advances the plans and the network skips the forward pass.
- Attentive planning: Attentive reading and writing focus updates on temporally informative regions of the action and commitment plans.Attention parameters control grid position, stride, and Gaussian-filter standard deviation; stride trades coverage for effective resolution.
- Commitment: The gate also enables gradients from later write operations to reach the commitment module, while commitment invokes only a cheap time-shift operator.The paper links greater commitment to computational savings.
- Structured exploration: STRAWe samples noisy feature representations before planning, creating randomness in multi-step plan updates that commitment forces the agent to execute.The paper reports this mechanism improves scores on games such as Frostbite and Ms. Pacman.
4 Learning
STRAW’s training combines a domain-specific output loss with regularizers for latent communication and replanning, while reinforcement learning uses an actor-critic objective with an auxiliary value-plan row. Its stochastic policy samples actions from the plan, and gradients are handled for both latent variables.
- STRAW’s loss combines a domain-specific output loss with a KL communication regularizer and a commitment penalty for replanning.For supervised prediction, the output loss can be negative log likelihood; the latent prior is Gaussian with zero mean and unit standard deviation.
- In reinforcement learning, the agent observes state x_t, selects an action, receives a reward, and aims to maximize discounted return.The process continues until a terminal state and then restarts.
- STRAW forms its stochastic policy by applying SoftMax to the first column of the action plan and sampling the resulting action distribution.
- For A3C training, STRAW adds an auxiliary row to the action plan instead of using a separate value network, providing value estimates while keeping the architecture simple.
- The policy-gradient objective includes an entropy term that stimulates exploration of primitive actions, while gradients must also pass through the latent variables z_t and g_t.The re-parametrization trick is used for z_t, and a separate gradient treatment is specified for g_t.
5 Experiments
STRAW is evaluated across text prediction, 2D mazes, and Atari games to test whether learned temporal abstractions produce useful macro-actions. The experiments show structured exploration, transfer of learned abstractions, meaningful action sequences, and benefits from learned replanning.
- Experimental setup: STRAW is evaluated in supervised next-character prediction, 2D maze navigation, and Atari reinforcement learning tasks.The experiments use increasingly complex domains and compare STRAW with feed-forward and recurrent LSTM baselines.
- 2D mazes: In random mazes, STRAW tends to re-plan at corners and junctions, indicating that it learns domain-specific temporal abstractions.Re-planning frequency is normalized by visits to each cell in an 11 × 11 maze.
- 2D mazes: As maze goals move farther away, STRAWe retains performance near the optimal policy while LSTM reward declines substantially.Both methods initially learn well when goals are nearby, but only STRAWe effectively reuses abstractions learned earlier in the curriculum.
- ATARI: STRAW or STRAWe achieves the highest score on 6 of 8 Atari games and is especially strong on Frostbite, Ms. Pacman, and Amidar.On Frostbite, STRAWe achieves more than 6× the LSTM score; performance is worse than LSTM on reactive Breakout.
- ATARI: Frostbite macro-actions represent meaningful behaviors such as jumping between floes and picking up fish, while Amidar replanning responds to nearby enemies.In Amidar, the agent retreats when blocked, restores its prior plan when danger passes, and uses shorter macro-actions near enemies.
- Ablative analysis: Ablations show that learning when to commit and when to re-plan improves performance compared with replanning every step or committing for random durations.The ablation models are trained for 100 epochs, whereas the Table 1 models are trained for 500 epochs.
6 Conclusion
STRAW learns useful temporally abstracted macro-actions end-to-end and improves performance on challenging Atari domains while also supporting temporal abstraction in general sequence prediction.
- STRAW implicitly learns useful temporally abstracted macro-actions in an end-to-end manner.
- STRAW advances the state-of-the-art on several challenging Atari domains requiring temporally extended planning and exploration strategies.
- STRAW also learns temporal abstractions in general sequence prediction.