Source-linked AI summary
Fully Parameterized Quantile Function for Distributional Reinforcement Learning
Derek Yang, Li Zhao, Zichuan Lin, Tao Qin, Jiang Bian, Tieyan Liu
TL;DR
Practical distributional RL must approximate continuous return distributions despite finite quantile representations and fixed or sampled choices on one distribution axis. FQF learns both quantile fractions and values with two jointly trained networks, and experiments show significant Atari improvements over existing distributional RL algorithms. The method also incurs slower training and leaves convergence and policy-impact questions open.
Problem
Finite quantile fractions make it unclear how to best approximate the true distribution, while prior methods fix or sample quantile fractions instead of fully learning them.
Method
FQF uses a fraction proposal network to generate quantile fractions and a quantile value network to map them to quantile values.
Results
FQF significantly outperforms existing distributional RL algorithms across 55 Atari games and sets a new record for non-distributed agents in the ALE.
Takeaways & Limitations
Learning which quantile fractions to use, in addition to their values, improves distribution approximation under limited network capacity.
Takeaways & Limitations
FQF is roughly 20% slower than IQN with the same settings, and convergence of its fraction proposal network is not guaranteed.
Abstract
from arXiv · showhide
Distributional Reinforcement Learning (RL) differs from traditional RL in that, rather than the expectation of total returns, it estimates distributions and has achieved state-of-the-art performance on Atari Games. The key challenge in practical distributional RL algorithms lies in how to parameterize estimated distributions so as to better approximate the true continuous distribution. Existing distributional RL algorithms parameterize either the probability side or the return value side of the distribution function, leaving the other side uniformly fixed as in C51, QR-DQN or randomly sampled as in IQN. In this paper, we propose fully parameterized quantile function that parameterizes both the quantile fraction axis (i.e., the x-axis) and the value axis (i.e., y-axis) for distributional RL. Our algorithm contains a fraction proposal network that generates a discrete set of quantile fractions and a quantile value network that gives corresponding quantile values. The two networks are jointly trained to find the best approximation of the true distribution. Experiments on 55 Atari Games show that our algorithm significantly outperforms existing distributional RL algorithms and creates a new record for the Atari Learning Environment for non-distributed agents.
1 Introduction
Distributional RL estimates return distributions rather than only expected returns, but practical methods must choose how to parameterize and approximate those distributions. FQF addresses this by learning both quantile fractions and their values.
- Motivation: Distributional RL models the randomness of total returns, unlike value-based RL, which focuses on expected returns.The randomness arises from both the environment and the agent’s policy.
- Motivation: The central practical challenge is parameterizing the estimated distribution well enough to approximate the true distribution.Existing approaches make different choices about which side of the distribution function to parameterize.
- Existing approaches: C51 learns probabilities at fixed return values, whereas QR-DQN learns quantile values at fixed, uniform quantile fractions.These methods leave one distribution axis fixed while learning the other.
- Existing approaches: With finite quantile fractions, IQN sampling primarily trains its implicit quantile value network and does not guarantee better full-function approximation than fixed probabilities.The limitation arises because infinitely many quantiles cannot be used in practice.
- FQF: FQF jointly parameterizes quantile fractions and corresponding values through a fraction proposal network and a quantile value network.The fraction proposal network minimizes the 1-Wasserstein distance between the approximation and the true distribution, while quantile regression trains the value network.
2 Background and Related Work
Distributional RL studies the distribution of discounted returns within a Markov decision process rather than only their expectation. Prior algorithms differ in whether they use fixed or sampled probabilities and how they estimate corresponding quantile values.
- RL background: The standard RL setting is modeled as a Markov Decision Process with state and action spaces, transitions, rewards, and a discount factor.The discount factor satisfies γ ∈(0, 1).
- RL background: A policy’s discounted return is a random variable, while the conventional action-value function is its expectation.Qπ(x, a) = E[Zπ(x, a)].
- Distributional RL: Distributional RL studies the distribution of the random return Zπ instead of representing each state-action value as a scalar.Its policy-evaluation update is expressed through a distributional Bellman operator.
- Prior evidence: Distributional RL has theoretical and empirical support, including Wasserstein contraction results and strong Atari performance from C51 and Rainbow.Rainbow combines C51 with prioritized replay, n-step updates, and a dueling architecture.
- Distributional algorithms: C51 estimates probabilities for N fixed return locations, whereas QR-DQN estimates quantile values for N fixed, uniform probabilities.QR-DQN represents the return distribution as a uniform mixture of N Dirac distributions.
- Distributional algorithms: IQN samples probabilities from a base distribution and learns an implicit quantile value network mapping sampled probabilities to quantiles.With finite fractions, the choice of how to use those fractions remains an approximation problem.
3 Our Algorithm
FQF parameterizes both quantile fractions and their corresponding values, then jointly trains the two networks to approximate the return quantile function. Its fraction proposals are optimized toward lower 1-Wasserstein error and combined with quantile regression for distributional value learning.
- Fully Parameterized Quantile Function: FQF uses a fraction proposal network to generate quantile fractions and a quantile value network to map those fractions to quantile values.For each state-action pair, fractions are generated first, followed by their corresponding quantile values.
- Fully Parameterized Quantile Function: Unlike QR-DQN and IQN, FQF parameterizes both quantile fractions and corresponding quantile values rather than fixing or sampling only the fractions.The method represents the quantile function with adjustable variables on both the probability and value axes.
- Training Fraction Proposal Network: The fraction proposal network is trained by minimizing the 1-Wasserstein error indirectly because the unbiased error cannot be computed in practice.The paper uses gradients derived from the optimality condition and treats the quantile value network as the current approximation of the unknown true quantile function.
- Fully Parameterized Quantile Function: Adjusting quantile fractions can reduce approximation error: Figure 1 associates finely adjusted fractions with minimized W1 error and random fractions with larger W1 error.The shaded area represents the 1-Wasserstein error for the two approximations.
- Training Quantile Value Network: FQF trains its quantile value network with quantile regression and a distributional Bellman update on the optimized quantile fractions.The quantile value network uses the Huber quantile regression loss, and the two networks receive joint gradient updates.
- Implementation Details: The fraction proposal network uses cumulative softmax outputs to produce ordered fractions with τ0 = 0 and τN = 1.This avoids sorting network outputs, which the paper notes would make training difficult.
4 Experiments
FQF is evaluated on 55 Atari games against IQN and other distributional RL baselines under matched training settings. It outperforms existing distributional methods, trains faster than IQN on some games, but incurs an overall training-speed cost.
- FQF is compared with IQN, QR-DQN, C51, prioritized experience replay, and Rainbow on Atari games.The experiments use IQN as the closest baseline and align FQF hyperparameters with IQN for fair comparison.
- FQF outperforms all existing distributional RL algorithms on mean and median human-normalized scores across 55 Atari games.Table 1 reports scores measured across 55 games, with averages over three seeds.
- FQF sets a new record for the number of games where a non-distributed RL agent performs better than human.
- FQF is generally much faster than IQN on games with similar performance, attributed to self-adjusting fractions.Centipede is given as an example.
- FQF is roughly 20% slower than IQN under the same settings because of its additional fraction proposal network.Increasing the number of τ samples slows FQF significantly, whereas IQN’s training speed is not sensitive to that number.
5 Discussion and Conclusions
The paper presents FQF as a complete approximation of the return distribution that learns both quantile values and the quantile fractions selecting them. It reports improved performance while leaving convergence, policy effects, and broader applicability as open questions.
- FQF parameterizes both the distribution targets and the quantile fractions that determine which targets are learned.This extends approaches that parameterize probabilities or quantile values alone.
- FQF is proposed as a more general complete approximation of the return distribution than previous distributional RL algorithms.
- Experiments show that FQF achieves significant improvement.
- The paper does not establish convergence of the fraction proposal network in deep neural networks with quantile regression and Bellman updates.
- The effects of better distribution approximation on policy and training, and the importance of quantile-fraction selection during training, remain open questions.The paper also identifies extending FQF to continuous settings as future work.
- Most games where the method fails to reach human-level performance involve complex rules requiring exploration-based policies.Montezuma Revenge and Venture are cited as examples.
Proof for proposition 1
The proof section establishes an intermediate-fraction property for the 1-Wasserstein loss when quantile functions are non-decreasing.
- The proof assumes the quantile function F−1_Z is non-decreasing.
- For two ordered quantile fractions, there exists an intermediate fraction satisfying the stated derivative condition for the 1-Wasserstein loss.The passage specifies τi−1 < τi+1 and τi between them.
Hyper-parameter sheet
The hyper-parameter sheet identifies the fraction proposal network’s learning-rate search and final setting, optimizer, and gradient-isolation procedure.
- The fraction proposal network learning rate is swept over (0, 2.5e-5) and fixed at 2.5e-9.
- RMSProp is used to train the fraction proposal network.
- Although the fraction proposal network receives the original IQN state embedding, gradients update only its new parameters, not the convolution layers.
Approximation demonstration
Figure 3 uses a toy case to demonstrate how distributional RL algorithms approximate a known quantile function and how quantile-fraction selection affects the approximation.
- Figure 3 compares algorithmic approximations of a known quantile function with a Monte Carlo-derived reference.W1 denotes the 1-Wasserstein distance between the approximated function and the Monte Carlo approximation.
Varying number of quantile fractions
Table 3 evaluates FQF and IQN across different numbers of quantile fractions, showing that FQF retains a slight performance advantage even when both methods use many fractions.
- Table 3 reports mean scores for FQF and IQN across 6 Atari games under different numbers of quantile fractions, N.Scores are measured as percentages of the human baseline and averaged over 3 seeds.
- FQF performs slightly better than IQN even at large N, when trained and randomly sampled quantile fractions are densely distributed over [0, 1].The stated advantage of trained fractions is expected to be more observable at smaller N.
Visualizing proposed quantile fraction
Figure 4 visualizes intervals between adjacent proposed quantile fractions during one run, illustrating how the fraction proposal network responds to state changes.
- The case study uses a half-trained Kungfu Master agent with N = 8 so that the return distribution is not deterministic.The learned quantile function may not always be non-decreasing, motivating future work that uses this property as prior knowledge.
- Figure 4 plots intervals between adjacent proposed quantile fractions for states at each time step in a single run.Different colors represent different adjacent intervals, such as τ2 − τ1.
- When an enemy appears behind the character, a spike in a fraction interval indicates a substantially different proposed fraction from following enemy-free states.This behavior suggests that the fraction proposal network is state dependent.