PaperlayerView exact revision on arXiv

Source-linked AI summary

Munchausen Reinforcement Learning

Nino Vieillard, Olivier Pietquin, Matthieu Geist

arXiv:2007.14430v3cs.LGstat.ML

TL;DR

The paper asks whether an RL agent’s current policy can provide another signal for bootstrapping beyond its current value estimate. It adds the scaled log-policy to immediate rewards and applies this idea to DQN and IQN. On Atari, M-DQN improves over DQN and C51, while M-IQN outperforms Rainbow; the analysis also identifies implicit KL regularization and action-gap increases.

  • Problem

    Temporal-difference RL typically bootstraps from its current value estimate, while the current policy is another estimate that could be used for bootstrapping.

    Source: §1 · p. 1

  • Method

    Munchausen RL adds the current policy’s scaled log-probability to the immediate reward in a temporal-difference scheme.

    Source: §2 · p. 2

  • Results

    M-DQN outperforms DQN on 53 of 60 Atari games and C51, while M-IQN outperforms Rainbow.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    The analysis shows that M-RL implicitly regularizes consecutive policies through KL and increases the action-gap by a quantifiable amount.

    Source: §1 · p. 2

Abstract

from arXiv · show

Bootstrapping is a core mechanism in Reinforcement Learning (RL). Most algorithms, based on temporal differences, replace the true value of a transiting state by their current estimate of this value. Yet, another estimate could be leveraged to bootstrap RL: the current policy. Our core contribution stands in a very simple idea: adding the scaled log-policy to the immediate reward. We show that slightly modifying Deep Q-Network (DQN) in that way provides an agent that is competitive with distributional methods on Atari games, without making use of distributional RL, n-step returns or prioritized replay. To demonstrate the versatility of this idea, we also use it together with an Implicit Quantile Network (IQN). The resulting agent outperforms Rainbow on Atari, installing a new State of the Art with very little modifications to the original algorithm. To add to this empirical study, we provide strong theoretical insights on what happens under the hood -- implicit Kullback-Leibler regularization and increase of the action-gap.

1 Introduction

Munchausen RL bootstraps from the current policy by adding its scaled log-policy to the immediate reward in a temporal-difference scheme. The paper applies this idea to DQN and IQN and reports empirical and theoretical contributions.

1 Introduction

  • M-RL adds the agent’s scaled log-policy to the immediate reward when using a temporal-difference scheme.The authors distinguish this from maximum-entropy RL, which subtracts the scaled log-policy from rewards.Source: §1 · p. 1
  • M-DQN is a slight modification of DQN that first introduces stochastic policies through a maximum-entropy extension.The paper also applies the same recipe to IQN to form M-IQN.Source: §1 · pp. 1–2
  • Munchausen-IQN surpasses Rainbow, establishing a new state of the art for single-agent algorithms.Munchausen-DQN also overtakes C51, the first distributional-RL agent, without using distributional RL itself.Source: §1 · p. 2
  • The analysis connects M-RL to implicit KL regularization between consecutive policies and quantifies its increase of the action-gap.The authors link the method to Conservative Value Iteration, Dynamic Policy Programming, and Advantage Learning.Source: §1 · p. 2

2 Munchausen Reinforcement Learning

Munchausen RL replaces the immediate reward in a temporal-difference update with that reward plus the policy’s scaled log-probability. The paper applies this update to DQN and IQN, reporting gains over their comparison baselines.

2 Munchausen Reinforcement Learning

  • M-RL bootstraps from the current stochastic policy by replacing r_t with r_t + α ln π(a_t|s_t) in a temporal-difference scheme.The policy is treated as the agent’s current estimate of which actions are good.Source: §2 · p. 2
  • Soft-DQN extends DQN to maximum-entropy RL, enabling stochastic policies before the Munchausen reward modification is applied.The paper derives M-DQN by modifying the regression target of Soft-DQN.Source: §2 · p. 3
  • M-DQN significantly outperforms DQN and C51 on the full set of 60 Atari games, despite being a simple DQN modification.The paper describes it as the first method not based on distributional RL to overtake C51.Source: §2 · p. 3
  • M-IQN outperforms Rainbow on both mean and median scores, including when using only 1-step returns.The compared 3-step-return versions use the same return length.Source: §2 · p. 3
  • IQN estimates a distribution whose mean gives the q-function, and M-IQN applies the Munchausen recipe after deriving soft-IQN.The tested 3-step-return M-IQN is compared with IQN and Rainbow using the same return length.Source: §2 · p. 3

3 What happens under the hood?

The analysis frames M-DQN as Munchausen Value Iteration (M-VI), then shows that its policy updates implicitly regularize consecutive policies and increase the action-gap. It also connects M-VI to existing value-iteration methods and explains why its guarantees apply in the deep-RL setting.

3 What happens under the hood?

  • The abstract scheme defines M-VI(α, τ) as a greedy step followed by an evaluation update; removing the Munchausen term gives entropy-regularized AVI, and removing the entropy term too gives classic AVI.The update error represents sampling, neural-network approximation, and network-fitting error.Source: §3 · p. 4
  • M-VI implicitly performs KL regularization between successive policies, and its sequence of policies is equivalent to Mirror Descent VI with KL scaled by ατ and entropy by (1 − α)τ.The greedy step is a softmax of the q-function, so it can be computed exactly for discrete actions even with deep networks.Source: §3 · pp. 4–5
  • The analysis applies to M-DQN because its KL regularization is implicit, avoiding greedy-step errors that arise when deep function approximation must optimize explicitly over a previous policy.For discrete actions, the M-VI greedy step is an exactly computable softmax of the q-function.Source: §3 · pp. 4–5
  • Its bounds average approximation errors across iterations rather than accumulating their discounted norms, with the error term scaled by (1 − γ)^−1 instead of AVI’s (1 − γ)^−2.This permits errors to compensate across iterations and makes the bound less negatively affected as γ approaches 1.Source: §3 · pp. 4–5
  • M-VI connects to Conservative Value Iteration, Dynamic Policy Programming, and Advantage Learning, while α = 0 recovers Soft Q-learning.With α = 1 it retrieves a variation of DPP; taking τ → 0 retrieves AL.Source: §3 · p. 5
  • M-VI increases the action-gap, with an infinite gap in the limit α = 1; the authors recommend a large α that is not too close to 1 for numerical stability.The action-gap is the difference between the optimal action’s value and suboptimal actions’ values.Source: §3 · pp. 5–6

4 Experiments

Across 60 Atari games, M-IQN is the strongest reported agent across the evaluated metrics, while M-DQN improves on DQN and C51 and is competitive with distributional methods. The experiments also compare ablations, per-game gains, and action-gap behavior under a specified single-agent ALE setting.

Munchausen agents

  • M-DQN and M-IQN use α = 0.9, τ = 0.03, and log-policy clipping at −1; both interact through ε-greedy policies despite producing stochastic policies.The log-policy term is unbounded and can cause numerical issues when the policy becomes nearly deterministic.Source: §4 · p. 6

Task and metrics

  • The reported evaluation uses the full 60-game ALE suite, 200M frames, and averages over three seeds; the authors caution that its setting differs from easier settings in earlier publications.The paper does not claim comparability with highly distributed agents trained with far more frames.Source: §4 · p. 6

Action-gap

  • On Asterix, M-DQN and AL both increase action-gaps relative to Adam DQN, but M-DQN has a more stable gap and scores about 20k versus 13k for AL and 15k for Adam DQN.The authors caution that this single-game figure is illustrative and insufficient for broad conclusions.Source: §4 · p. 7, Figure 2 · p. 7

Ablation study

  • M-DQN outperforms every tested ablation in mean and median, especially Soft-DQN((1 − α)τ), whose only stated difference from M-DQN is the Munchausen term.The ablations include Adam DQN, Soft-DQN, temperature-matched Soft-DQN, and AL; all methods outperform DQN.Source: §4 · p. 7, Figure 3 · p. 8

Comparison to the baselines

  • M-IQN is the best agent across all reported metrics, while M-DQN exceeds DQN and C51 and remains close to IQN in median score.Table 1 reports mean and median Human- and Rainbow-normalized scores at 200M frames over 60 games, averaged over three seeds.Source: Table 1 · p. 8, §4 · p. 8

Per-game improvements

  • Across a large majority of Atari games, Munchausened agents improve over their base algorithms: M-DQN improves over DQN on 53/60 games, and M-IQN improves over IQN on 40/60 games.Source: §4 · p. 8

5 Conclusion

The paper presents Munchausen RL as adding a scaled log-policy to immediate rewards, with M-DQN and M-IQN as applications. It reports Atari performance gains and theoretical results on implicit KL regularization and action-gap increase.

5 Conclusion

  • M-DQN outperforms DQN on 53 of 60 Atari games and C51, while M-IQN combines the Munchausen principle with IQN to outperform the Rainbow baseline.The paper also reports that M-VI’s action-gap increase can be infinite in the limit.Source: §5 · pp. 8–9, Figure 4 · p. 9
  • The abstract Munchausen Value Iteration scheme implicitly performs KL regularization, and the theoretical results of [32] apply to M-DQN.The paper also relates the approach to CVI, DPP, and AL.Source: §5 · pp. 8–9
  • The work argues that revisiting core reinforcement-learning components can yield efficient deep-RL algorithms.Source: §5 · p. 9

A Detailed derivation and proofs

The appendix gathers derivations and proofs for Soft-DQN and the paper’s Munchausen value-iteration results. It also derives Soft-DQN as an entropy-regularized approximate value-iteration scheme.

A Detailed derivation and proofs

  • The appendix provides proofs for the Munchausen VI–Mirror Descent VI relation, component-wise bounds, literature connections, and the quantified action-gap increase.It begins by recalling notation and useful facts about regularized MDPs.Source: A Detailed derivation and proofs · p. 13

A.1 Derivation of Soft-DQN

  • Soft-DQN follows from maximum-entropy RL for discrete actions, where policy computation needs no approximation; equivalently, it is approximate value iteration for an entropy-regularized MDP.Its target uses a softmax policy and evaluates q-values with an entropy adjustment.Source: §A.1 · p. 14

A.2 Proof of Thm. 1

The proof rewrites the M-VI update into a form involving policy entropy and a KL penalty. It then identifies the update with MD-VI and establishes that the algorithms produce the same policy sequence under matching conditions.

A.2 Proof of Thm. 1

  • The evaluation update is rewritten to include a KL penalty between consecutive policies and an entropy term.The derivation expresses the update using the KL divergence from the previous policy and the entropy of the next policy.Source: §A.2 · p. 14
  • The rewritten update is exactly the update rule of MD-VI(ατ, (1 −α)τ).This identifies the M-VI iteration with the corresponding MD-VI iteration.Source: §A.2 · p. 14
  • With the same initial policy and q′0 = q0 −τ ln π0, both algorithms produce the same policy sequence for the same errors.This correspondence allows the cited MD-VI theorems to apply to M-VI.Source: §A.2 · p. 14

A.3 Component-wise bounds for Munchausen VI

The appendix gives component-wise bounds for M-VI at α = 1 and α < 1, and connects its updates to prior dynamic-programming methods. The bounds include error compensation, while the related-work discussion derives a CVI-like update.

A.3 Component-wise bounds for Munchausen VI

  • For α = 1, the M-VI policy sequence matches MD-VI(τ, 0), allowing a component-wise bound to be transferred from the cited MD-VI result.The bound assumes ∥qk −τ ln πk∥∞ is bounded; clipping q-values can ensure this, while without it the error term has quadratic rather than linear horizon dependence.Source: §A.3 · p. 15
  • For 0 ≤ α < 1, M-VI corresponds to MD-VI with parameters ατ and (1 −α)τ, and its bound retains error compensation through a moving average.The appendix notes that this moving-average compensation is desirable.Source: §A.3 · p. 15

A.4 Details on related works

  • The M-VI update is shown to be CVI-like, and as temperature approaches zero its sum-log-exp term tends toward the hard maximum.The limiting update is written using a greedy policy in the set G(qk).Source: §A.4 · p. 16

A.5 Proof of Thm. 2

The appendix proof relates M-VI to MD-VI to obtain convergence results for α < 1 and α = 1. Additional sections detail the agents and compare interaction policies and M-IQN return lengths.

A.5 Proof of Thm. 2

  • For α < 1, M-VI follows the MD-VI policy sequence and converges to the corresponding regularized solution; for α = 1, it converges to the unregularized MDP solution under a unique-optimal-policy assumption.The α = 1 proof invokes the unique optimal policy assumption to simplify the limit argument.Source: §A.5 · pp. 16–17

B Additional experimental details and results

  • The appendix covers full agent descriptions, additional metric and learning-setting details, return-length comparisons, ablations, and complementary comparisons.It also includes comparisons with the original ALE setting and additional visualizations.Source: B Additional experimental details and… · p. 17

B.1 Detailed description of the Munchausen agents

  • M-DQN uses ε-greedy interaction, while M-IQN adds a Munchausen term to IQN's TD error; the appendix reports their settings and evaluation details.The appendix also describes ALE preprocessing and the score metrics used in comparisons.Source: §B.1 · pp. 17–19

B.2 Comparison of greedy and stochastic policies

  • The policy choice can depend on reward access: a near-uniform softmax may fail to gather rewards on Enduro, whereas softmax exploration can favor higher-Q actions when rewards are available.The proposed directed-exploration explanation for Seaquest is presented as a hypothesis.Source: §B.2 · pp. 19–20
  • Stochastic interaction is slightly better in median human-normalized score but improves fewer games, so the core results retain ε-greedy interaction.The figure comparison reports that stochastic interaction learns no reward signal initially on Enduro, but improves over greedy interaction on Seaquest.Source: §B.2 · p. 20, Figure 5 · p. 20

B.3 Comparison of 1-step and 3-steps learning in M-IQN

  • M-IQN with 3-step returns clearly outperforms its 1-step version, while 1-step M-IQN narrowly outperforms the 3-step Rainbow baseline on human-normalized scores.The appendix evaluates both M-IQN and IQN with matching 1-step returns for this comparison.Source: §B.3 · p. 20

B.4 Element of comparison with the original ALE setting

The original-ALE comparison uses a setting intended to approximate prior publications, but differs from the paper’s main setting. On Asterix, the Munchausen agents outperform their corresponding baselines, while the authors caution that the comparison covers only one game.

B.4 Element of comparison with the original ALE setting

  • The approximate original-publication comparison removes sticky actions, ends episodes on life loss, and uses published hyperparameters except for a 10,000-step target update period.It was run on Asterix, with results shown in Figure 8.Source: §B.4 · p. 21
  • M-IQN performs better than both IQN and Rainbow on Asterix, while M-DQN scores 15k versus DQN’s 19k in the more difficult setting.In the original setting, M-DQN performs much better than DQN without the apparent score fall described for DQN.Source: §B.4 · p. 21
  • Rainbow falls from roughly 350k to 18k (5% of original scores), IQN from 350k to 33k (10%), M-DQN rises from 15k to 17k (113%), and M-IQN falls from 350k to 50k (17%).The authors say this suggests M-RL might be more stable over environments.Source: §B.4 · p. 21
  • The authors limit their interpretation to one game and say no general conclusion can be drawn.They describe the considered ALE setting as more difficult and M-RL as seeming more stable.Source: §B.4 · p. 21

B.5 Additional results on the ablation study

The ablation results identify M-DQN as the strongest overall variant across the game suite, although other variants can lead on individual games. The authors also compare the variants across normalizations and discuss what the results suggest about the Munchausen term and alternative design choices.

B.5 Additional results on the ablation study

  • The authors caution that Rainbow comparisons differ from the published setup in enhancements, codebase, start protocol, and possibly ROM.The ROM difference could not be checked.Source: §B.5 · p. 21
  • Rainbow-normalized ablation results confirm the Human-normalized results, with comparable ordering and gaps despite different score values.Figure 9 provides the Rainbow-normalized scores in place of the main paper’s Human-normalized Figure 3.Source: §B.5 · pp. 21–22, Figure 9 · p. 22
  • M-DQN is consistently the best-performing agent over the full game suite in both mean and median Rainbow- and Human-normalized scores.Figure 9 reports Rainbow-normalized ablation results; Figure 10 shows Rainbow-normalized mean and median scores.Source: §B.5 · p. 22, Figure 9 · p. 22, Figure 10 · p. 23
  • M-DQN is not always best on individual games, but is often competitive with the best ablation, whose identity depends strongly on the game.Figures 11 and 12 summarize per-game improvement and learning curves, with Figure 11 reporting results after 200M frames.Source: §B.5 · pp. 21–22
  • Soft-DQN is the worst ablation in average and median despite doing better on some games, while the temperature was not specifically tuned for it.The authors say this suggests maximum-entropy RL alone might not be sufficient; they also note AL performs less well than M-DQN and Adam DQN performs surprisingly well relative to original DQN.Source: §B.5 · p. 22

B.6 Additional comparison results

The supplementary comparisons add Rainbow-normalized and per-game comparisons across baselines, alongside best-score and learning-curve summaries. These results are reported as confirming the observations in the main paper.

B.6 Additional comparison results

  • M-IQN obtains the most highest-ranking scores among the considered baselines, including the human baseline.Table 3 summarizes maximum scores during training, averaged over 100 episodes and three random seeds, and counts games with the best score.Source: §B.6 · p. 23, Table 3 · p. 28
  • The study reports Rainbow-normalized comparisons of Munchausen agents against DQN, C51, IQN, and Rainbow in Figure 1, plus per-game normalized improvements for M-DQN and M-IQN versus those baselines in Figures 13 and 14.Source: §B.6 · p. 23, Figure 13 · p. 26, Figure 14 · p. 27
  • The supplementary results include learning curves for Munchausen agents and baselines across the full Atari game set, and are described as confirming the main-paper observations.Figure 15 identifies the plotted agents; Figures 11 and 12 provide additional ablation comparisons and learning curves.Source: §B.6 · p. 23, Figure 11 · p. 24, Figure 12 · p. 25
Loading 2007.14430v3…