Source-linked AI summary
Deep Reinforcement Learning With Macro-Actions
Ishan P. Durugkar, Clemens Rosenbaum, Stefan Dernbach, Sridhar Mahadevan
TL;DR
The paper examines whether temporal abstraction through macro-actions can improve deep reinforcement learning, extending DQN to use macros alongside basic actions. On Atari games, macros improved convergence times and achieved better scores on six of the seven tested games.
Problem
The paper examines whether temporal abstraction through macro-actions can improve deep reinforcement learning.
Source: §1 · p. 2Method
The authors extend DQN to use macro-actions alongside basic actions and compare approaches for constructing macros.
Source: §1 · p. 2Results
Macros improved DQN convergence times and achieved better scores on six of the seven tested Atari games.
Source: §6 · p. 8, §5.4 · p. 8Takeaways & Limitations
The authors conclude that useful macros are likely to accelerate learning and encourage discovery of optimum policies.
Source: §6 · p. 8The authors describe macros as potentially too rigid for many problems and identify comparing closed-loop models such as options on Atari as future work.
Source: §6 · p. 9
Abstract
from arXiv · showhide
Deep reinforcement learning has been shown to be a powerful framework for learning policies from complex high-dimensional sensory inputs to actions in complex tasks, such as the Atari domain. In this paper, we explore output representation modeling in the form of temporal abstraction to improve convergence and reliability of deep reinforcement learning approaches. We concentrate on macro-actions, and evaluate these on different Atari 2600 games, where we show that they yield significant improvements in learning speed. Additionally, we show that they can even achieve better scores than DQN. We offer analysis and explanation for both convergence and final results, revealing a problem deep RL approaches have with sparse reward signals.
1 Introduction and Related Work
The paper investigates temporal abstraction as an improvement to deep reinforcement learning, focusing on macro-actions added to primitive actions. It contrasts this approach with other hierarchical methods and situates it among efforts to improve deep RL algorithms.
1 Introduction and Related Work
- Temporal abstraction adds hierarchical multi-step actions to primitive actions, but the number of possible abstractions grows exponentially with their length.The paper notes that finding useful abstractions has long been an important reinforcement-learning problem.Source: §1 · p. 1
- Some hierarchical models use options or closed-loop control but require external supervision, while macros are deterministic, predefined action sequences chosen as atomic actions.Rewards from states visited during a macro are still accumulated, although the agent cannot act at those intermittent states.Source: §1 · pp. 1–2
- The authors extend DQN with macro-actions, compare ways to construct them, and analyze how they interact with deep-network input representations.The approach leaves the rest of the original DQN architecture unchanged and can be paired with other improvements.Source: §1 · p. 2
2 Hierarchical Reinforcement Learning
Hierarchical reinforcement learning uses temporally extended actions, with this work focusing on open-loop macros rather than closed-loop models. The section explains the SMDP setting, how macros work, and their costs.
2 Hierarchical Reinforcement Learning
- Hierarchical RL can divide problems into subproblems, but this work focuses on open-loop action sequences; closed-loop strategies are more general once learned and difficult to learn without supervision.The paper relates closed-loop strategies to options and notes that their subgoals are difficult to define generally.Source: §2 · p. 2
2.1 Background
- An SMDP extends an MDP with a variable duration between successive states, so temporally extended actions change the value and Q-learning updates to account for duration and intervening rewards.In the Atari domain, interaction occurs at discrete 1/15-second steps.Source: §2.1 · pp. 2–3
2.2 Macros
- In deterministic Atari, a macro is a predefined action sequence whose visited states, terminal state, and cumulative reward follow from that sequence; this study does not explore composing macros from other actions.The paper gives “up, up, punch, down, down” as an example sequence in Boxing.Source: §2.2 · p. 3
- Adding macros enlarges the action space, and skipping decisions at some states could lead to a suboptimal policy.Source: §2.2 · p. 3
3 Learning Macros
The section extends DQN with open-loop macros and describes two ways to construct them: repeating atomic actions and selecting frequently observed action sequences. Macros are first-order additions to atomic actions.
3 Learning Macros
- Macro actions extend DQN’s output layer, while replay transitions store duration τ and accumulated rewards; macro replacements occur at increasingly spaced epochs.The method excludes macros containing other macros and raises exploration to ε = 0.5 after reassignment so the new macro values can be learned.Source: §3 · p. 4
3.1 Repetition of Actions
- Repetition macros repeat one atomic action for a specified duration, effectively taking larger steps when Atari’s base time step is finer than decisions require.Source: §3.1 · p. 4
3.2 Frequency
- Frequency macros select commonly occurring action subsequences, dropping candidates whose longest-common-subsequence overlap exceeds ωℓ.The method assumes frequently repeated sequences may be useful under the current policy.Source: §3.2 · p. 5
4 Experimental Results
Across the Atari evaluation, macros’ main reported advantage is faster convergence. The experiments compare macro agents trained for 50 million frames with DQN trained for 100 million frames.
4 Experimental Results
- The evaluation trained agents on eight games, comparing macro agents after 50 million frames with DQN trained for 100 million frames.Testing occurred after each one-million-frame epoch, and scores were averaged over five epochs with variance reported.Source: §4 · p. 5
- Updating frequency-based macros at epochs 6, 13, 25, and 50 tended to reduce performance and increase variance until the macros converged.The updates were spaced farther apart to give the agent more time to adapt to the changed action space.Source: §4 · p. 5
- The baseline differs from because this evaluation averages episode scores over five epochs, whereas reports averages over best scores.Source: §4 · p. 5
- Macros’ principal reported advantage is faster convergence; Table 1 reports mean scores for length-3 and length-5 macros at 50 epochs.Source: §4 · p. 5, Table 1 · p. 6
5 Analysis
The analysis evaluates macro-actions by scores, convergence speed, and score variance, and reports that useful macros outperform atomic actions.
5.1 Explanatory Hypotheses
- Macros may aid exploration through larger directed moves and speed reward propagation by carrying cumulative rewards up to ℓ steps ahead in one update.Source: §5.1 · p. 7
5.2 Convergence
- Random macros explore farther but converge worse than manual or learned macros; learned macros approach manual-macro convergence and outperform DQN, suggesting value propagation matters more than exploration alone.Figure 1 shows convergence behavior on Pong and low variance on Enduro; dashed update epochs explain performance drops as learned macros change.Source: §5 · p. 6, Figure 1 · p. 6, §5.2 · p. 7
5.3 Variance
- Score variance is consistently lower with macros, especially in Qbert, where macros also achieve higher final scores and support more stable policies.Source: §5.3 · p. 7
5.4 Scores Achieved
- Macros outperform atomic-action agents on six of seven games; Breakout is the only game at or below atomic-action performance.Source: §5.4 · p. 8
- With sparse rewards, macros spread Q-value estimates beyond approximation error, widening action gaps and supporting more reliable greedy decisions.Figure 2 compares Q-values for macro-plus-action and action-only agents.Source: Figure 2 · p. 7, §5.4 · p. 8
- Temporally extended actions also make state changes more noticeable to the network, helping it encode them sooner and devote more training to policy tuning.Source: §5.4 · p. 8
- Simple action repetition performs especially well in Ms Pacman, while it fares worse in Breakout, where extended sequences interfere with fine-grained control.In Ms Pacman, both macro actions and exploratory atomic actions can interrupt movement that continues until a wall.Source: §5.4 · p. 8
- Frequency-based macro learning has little influence overall, but in Boxing it yields strategy-like patterns such as approaching, punching, and leaving.Source: §5.4 · p. 8
6 Conclusion and Future Work
Macro-actions improved DQN convergence and scores on Atari, while their open-loop rigidity may limit use across problems. The authors identify better macro discovery—including learning actions, lengths, and quantities—as future work.
6 Conclusion and Future Work
- Macros improved DQN training efficiency and scores on six of the seven tested games.The conclusion also reports improved convergence times and scores on Atari.Source: §6 · p. 8
- Useful macros may accelerate learning and encourage discovery of optimal policies.The authors propose learning the actions, optimal lengths, and quantity of macros as a next step.Source: §6 · p. 8
- As open-loop policies, macros execute a specified action sequence without consulting the underlying state once started.The agent therefore does not make decisions as frequently during execution.Source: §6 · p. 8
- The authors describe macros as potentially too rigid for many problems and propose comparing closed-loop models such as options on Atari.They identify improved macro discovery, including learning actions, lengths, and quantities, as future work.Source: §6 · pp. 8–9