PaperlayerView exact revision on arXiv

Source-linked AI summary

Implicit Quantile Networks for Distributional Reinforcement Learning

Will Dabney, Georg Ostrovski, David Silver, Rémi Munos

arXiv:1806.06923v1cs.LGcs.AIstat.ML

TL;DR

Distributional reinforcement learning has largely used policies based only on mean returns, leaving open how to exploit return-distribution information for risk-sensitive control. IQN learns the full quantile function with quantile regression and an implicit distribution, achieving strong Atari-57 performance while supporting distortion-risk policies. The method substantially improves over QR-DQN and narrows the gap to Rainbow, including a 162% median human-normalized score on human starts.

  • Problem

    Recent distributional RL policies use only mean returns, leaving open whether return distributions can support broader risk-sensitive policies and when they are beneficial.

  • Method

    IQN uses quantile regression to learn the full return quantile function and an implicit distribution, with non-uniform sampling enabling distortion-risk policies.

  • Results

    IQN substantially improves over QR-DQN on Atari-57, nearly matches Rainbow, and reaches 162% median human-normalized score on human starts.

  • Takeaways & Limitations

    IQN provides a fully integrated distributional RL agent with flexible sampling, improved data efficiency, and a broad class of risk-sensitive policies.

  • Takeaways & Limitations

    The paper leaves sample-based convergence analysis for QR-based algorithms as an open theoretical question.

Abstract

from arXiv · show

In this work, we build on recent advances in distributional reinforcement learning to give a generally applicable, flexible, and state-of-the-art distributional variant of DQN. We achieve this by using quantile regression to approximate the full quantile function for the state-action return distribution. By reparameterizing a distribution over the sample space, this yields an implicitly defined return distribution and gives rise to a large class of risk-sensitive policies. We demonstrate improved performance on the 57 Atari 2600 games in the ALE, and use our algorithm's implicitly defined distributions to study the effects of risk-sensitive policies in Atari games.

1. Introduction

Distributional reinforcement learning models the full distribution of returns rather than only its mean. IQN extends quantile-regression methods to learn a continuous quantile function, enabling flexible implicit distributions and risk-sensitive policies.

  • Motivation: Distributional RL models return distributions arising from intrinsic randomness in agent–environment interactions, while the traditional value function is their mean.The randomness includes stochastic interactions, representation approximations, and chaotic physical interaction.
  • Related work: Existing distributional algorithms differ in how they parameterize return distributions and which loss or distance metric they optimize.C51 uses a categorical distribution with cross-entropy and a Cramér-minimizing projection, assuming returns lie in a known bounded range.
  • Related work: C51 outperformed previous DQN improvements on the 57-game Atari-57 benchmark, while QR-DQN further improved performance by adapting quantiles through quantile regression.QR-DQN minimizes Wasserstein distance while remaining restricted to a discrete set of quantiles.
  • Contribution: IQN extends QR-DQN from discrete quantiles to the full quantile function, forming an implicit distribution that can approximate any return distribution given sufficient network capacity.IQN is presented as a distributional generalization of DQN.
  • Contribution: IQN makes distributional approximation depend on network size and training rather than output quantile count, supports flexible sampling per update, and enables distortion-risk policies.Non-uniform base distributions expand policies to ϵ-greedy policies based on arbitrary distortion risk measures.
  • Results: IQN significantly outperforms QR-DQN and nearly matches Rainbow on Atari-57, improving over Rainbow on human-starts and the hardest games.The comparison uses IQN without other improvements to DQN.

2. Background / Related Work

The background develops value-based and distributional reinforcement learning, emphasizing Wasserstein-based quantile methods and the unresolved question of exploiting return distributions for risk-sensitive control.

  • Standard RL: In standard reinforcement learning, an MDP specifies states, actions, rewards, transitions, and discounting, while a policy maps states to action distributions.The return is a discounted sum of future rewards, and the action-value function is its expectation.
  • Standard RL: Q-learning estimates the optimal action-value function through repeated Bellman updates, with DQN using a convolutional neural network and ϵ-greedy behavior on Atari-57.The estimate is trained by minimizing squared temporal-difference error over observed transitions.
  • Distributional RL: Distributional RL replaces the scalar value function with the probability law of returns, and empirical studies report gains in sample complexity, final performance, and robustness.The scalar value remains the expectation of the modeled return distribution.
  • Distributional RL: Distributional Bellman operators have different theoretical behavior across settings: policy evaluation is a Wasserstein contraction, whereas control is not, although optimal-policy convergence can still be established.The control case requires a more involved convergence argument.
  • Metrics: Wasserstein distance measures differences through the Lp distance between inverse cumulative distribution functions, also called quantile functions.This metric trades off approximate solutions with likelihoods unlike KL divergence.
  • Related work: C51 uses a categorical distribution over fixed equidistant points and a projected KL loss, whereas QR-DQN uses quantile regression to adapt discrete quantiles and minimize Wasserstein distance.QR-DQN achieved the best Atari-57 performance among agents excluding combinations with prioritized replay and n-step updates.
  • Risk-sensitive RL: Prior distributional RL policies used only the mean return, motivating whether return-distribution information can support broader risk-sensitive policies and when those policies help.Here, risk denotes uncertainty over possible outcomes rather than parametric uncertainty in value estimates.
  • Risk-sensitive RL: Expected-utility and distortion-risk formulations provide equivalent ways to express risk sensitivity through utility transformation or probability reweighting.The paper later varies distortion risk measures by changing the sampling distribution for quantile probabilities.

3. Implicit Quantile Networks

IQN learns a continuous quantile function that implicitly represents return distributions, extending QR-DQN while enabling flexible sampling and risk-sensitive policies. Its sample-based training and architecture improve learning efficiency and long-term Atari performance.

  • Core approach: IQN maps state-actions and base-distribution samples to quantile values, thereby defining an implicit return distribution.With sufficient network capacity, the implicit distribution can approximate any return distribution.
  • Core approach: Unlike QR-DQN’s fixed quantiles, IQN approximates the full continuous quantile function with a state-action and sample embedding.The shared representation also makes IQN a type of universal value function approximator.
  • Training and policies: Independent continuous samples decorrelate sample TD errors and estimate action values as sample means from the implicit distribution.The policy can sample from non-uniform distributions to implement risk-sensitive behavior.
  • Sample-count study: N strongly affects early performance, with scores continually improving as N increases, while N′ has minimal long-term impact beyond N′ = 8.The experiment varied both sample counts over {1, 8, 32, 64} on six Atari games.
  • Sample-count study: N = N′ = 8 captures most long-term IQN gains, while N = N′ = 1 still reaches approximately 3× DQN performance.Increasing both sample counts is generally favorable, but improvements beyond eight samples are largely insignificant for long-term performance.

4. Risk-Sensitive Reinforcement Learning

IQN changes the policy’s sampling distribution to represent different distortion risk measures and induce risk-sensitive behavior. In six Atari games, risk-averse policies often outperform risk-neutral and risk-seeking policies, though effects vary by game.

  • Risk-sensitive policies: Changing the distortion risk measure changes the sampling distribution and therefore the policy while leaving the learned return representation unchanged.The resulting policies maximize distorted expectations rather than the usual sample mean.
  • Risk measures: The study evaluates cumulative prospect-theory weighting, Wang distortions, power distortions, and CVaR as alternative risk measures.CVaR changes uniform sampling from U([0, 1]) to U([0, η]).
  • Risk measures: CPW(.71) and Norm(3) reduce tail impact, whereas Wang and CVaR shift probability mass toward distribution tails.CVaR assigns zero probability above η, while Wang assigns those values small but nonzero probability.
  • Results: Risk-averse policies significantly improve performance over standard IQN on ASTERIX and ASSAULT, while CPW performs nearly like risk-neutral IQN.The Wang(1.5) risk-seeking policy performs as well as or worse than risk-neutral IQN.
  • Results: CVaR(0.1) loses performance on QBERT and SPACE INVADERS, and risk-seeking policies significantly underperform risk-neutral policies on three games.The authors leave the precise reason for improved risk-averse performance open, mentioning longer survival as one possible explanation.

5. Full Atari-57 Results

On Atari-57, IQN substantially outperforms QR-DQN and approaches Rainbow despite using only the risk-neutral distributional policy. Its gains are especially visible in human-starts and in games where agents remain below human performance.

  • Benchmark setup: IQN is evaluated against Rainbow, QR-DQN, prioritized replay DQN, and original DQN on the full Atari-57 benchmark.The reported IQN policy is risk-neutral, using epsilon-greedy action selection with respect to the mean return.
  • Training performance: IQN dramatically improves over QR-DQN during training and reaches at 100 million frames the performance QR-DQN reaches at 200 million frames.Figure 4 reports human-normalized mean and median scores during training, with traces for random seeds.
  • Human-start evaluation: 162% median human-normalized score is achieved by IQN with human-starts, compared with 153% for Rainbow.These scores are reported for the human-start evaluation condition.
  • Human-gap analysis: IQN outperforms all other agents on the human-gap metric, while the remaining Rainbow–IQN gap comes entirely from games where both are already super-human.The human-gap measures how far below human level agents perform on games where they remain below humans.

6. Discussion and Conclusions

The paper presents IQN as a flexible, integrated distributional extension of DQN that supports risk-sensitive policies and improves Atari-57 performance. It also identifies unresolved theoretical questions about convergence and contraction for quantile-based and distorted-risk methods.

  • Discussion and Conclusions: IQN generalizes quantile-regression distributional RL into a fully integrated DQN agent without prior assumptions about return-distribution parameterization.The approach learns the distribution over returns and naturally integrates risk-sensitive policies.
  • Discussion and Conclusions: IQN supports training with one sample per state-action distribution or more samples as computational limits allow, improving data efficiency with additional samples.The method also supports a broad class of risk-sensitive policies connected to distortion risk measures.
  • Discussion and Conclusions: IQN shows substantial Atari-57 gains over QR-DQN and halves the distance between QR-DQN and Rainbow.The conclusion summarizes the benchmark comparison at the paper level.
  • Open questions: Open theoretical questions concern sample-based convergence, contraction for approximate quantile functions, and convergence guarantees for distorted expectations and risk-sensitive policies.The paper presents these as areas requiring additional analysis rather than established results.
  • Future research: Future empirical work includes combining IQN with Rainbow enhancements and investigating distributional reinforcement learning in continuous control.The authors suggest a Rainbow-IQN agent could yield greater Atari-57 improvements.

Architecture and Hyperparameters

IQN extends DQN with sample embeddings and merging functions to parameterize a return distribution, with architectural variants evaluated across Atari games. The selected configuration uses multiplicative interaction, a 64-dimensional cosine embedding, and ReLU.

  • Architecture: IQN extends a DQN Q-network with a sample embedding φ(τ) and merging function m applied to convolutional features ψ(x).The resulting network outputs action-value quantiles.
  • Embedding variants: The evaluated embedding variants include learned linear, one-hidden-layer MLP, and cosine-basis functions, followed by ReLU or sigmoid.Cosine embeddings use cos(πiτ) for i = 1, . . . , n.
  • Merging function: Multiplicative merging uses ψ ⊙φ, while the residual alternative uses ψ ⊙(1 + φ) to force early interaction between representations.These alternatives were compared with simple concatenation.
  • Architecture comparisons: Figure 5 compares embedding type, embedding size, nonlinearity, and merging function using end-of-training human-normalized performance across six Atari 2600 games.White dots represent configurations, black dots the preferred configuration, and colored width a kernel density estimate.
  • Selected configuration: The final configuration uses multiplicative interaction, a linear cosine embedding with n = 64, and ReLU, yielding the highest performance consistently across multiple seeds.Most configurations also outperform the QR-DQN baseline in the reported comparisons.

Evaluation

Evaluation reports Atari performance using human-normalized and raw-score references, with training curves covering the complete Atari-57 benchmark. The human-gap metric additionally measures remaining distance to super-human performance.

  • Metrics: Human-normalized scores are computed from agent, human, and random per-game raw scores.The raw scores are undiscounted returns.
  • Metrics: The human-gap metric is gap = max(1 − score, 0), where 1 corresponds to random play and 0 to super-human performance.The quantity is clipped above at 1 when human performance is below random performance.
  • Score reporting: Figure 7 reports raw scores for a single seed across all games after 30 no-op starting actions, alongside reference values from Wang et al. (2016).
Loading 1806.06923v1…