Parametric Return Density Estimation for Reinforcement Learning

• Expected-return maximization does not necessarily avoid rare large negative outcomes, motivating alternative criteria for risk evaluation.Source: §1 · p. 1
• The paper estimates return densities and evaluates risk-sensitive or robust criteria from those estimates.Source: §1 · p. 1
• The approach extends the Bellman equation to conditional return densities and develops parametric TD-learning algorithms for Gaussian, Laplace, and skewed Laplace models.Source: §1 · p. 1

2 Background of Value-based RL

• The section reviews value-based RL and motivates estimating return densities to address risk sensitivity and robustness.Source: §2 · p. 1, §2.1 · p. 2

2.1 Markov Decision Process (MDP)

• An MDP is defined by states, actions, transition probabilities, and reward densities, with immediate rewards conditioned on state, action, and successor state.Source: §2.1 · p. 2

2.3 Related work on return density

• Earlier return-distribution methods include second-moment recursion and Bayesian normal-gamma learning, while some mean-variance methods assume Gaussian returns.The proposed approach avoids numerical integration and allows choosing the return-density model; Morimura et al. (2010) propose a nonparametric approach.Source: §2.3 · p. 2

2.2 From expected return to return density

• The return is the time-discounted cumulative reward, with discount rate γ in [0, 1).Source: §2.2 · p. 2, §2.1 · p. 2

3 Distributional Bellman Equation

• The distributional Bellman equation recursively characterizes the conditional return density as the solution to an extension of the expected-return Bellman equation.Source: §3 · p. 2
• Expectation-only RL omits return information needed for risk control, and expected-return estimates can be strongly affected by rare, large noise.Source: §3 · p. 2
• Because returns equal the immediate reward plus the discounted next return, the derivation uses conditional independence of reward and next return given the successor state.Source: §3 · p. 2
• The cited distributional Bellman equation of Morimura et al. (2010) concerns the cumulative density rather than the density.Source: §3 · p. 2

4.1 Return density approximation with distributional Bellman equation

• Parametric families address the functional degrees of freedom that make the distributional Bellman solution difficult to handle, with KL divergence used to fit the estimate to its Bellman update.Source: §4 · p. 3, §4.1 · p. 3

4 Approximation of Return Density with Parametric Model

• The approach uses Gaussian, Laplace, and skewed Laplace return-density models, including symmetric and asymmetric fat-tailed forms.Source: §4.3 · p. 3, §4 · p. 3

4.3 (Natural) gradient descent approach

• Because the density family forms a manifold under KL divergence, ordinary gradients can misrepresent parameter sensitivity; natural gradients use the Fisher information metric for the steepest direction.Source: §4.3 · p. 4, §4.4 · p. 4

4.4 Stochastic natural gradient with samples

• In stochastic environments, the Bellman-updated density is estimated from observed successor-state and reward samples, enabling sample-based density updates.Source: §4.4 · p. 4

5 Using the Estimated Return Density for Risk-Sensitive RL

• The q-quantile return is used as an action-selection criterion, and the estimated return density can support criteria defined on that density.The paper identifies conditional value at risk as another criterion that can be derived analytically.Source: §5 · p. 5
• For skewed Laplace models with c > 0.5, positive temporal differences tend to influence the parameter updates more strongly than nonpositive differences.For Laplace models, the central-parameter update is bounded by the scale parameter, which the authors say can support stable learning with outlier samples.Source: §5 · p. 5
• The proposed q-Q learning and q-SARSA algorithms estimate q-Q functions using Gaussian, Laplace, or skewed Laplace return-density models.The framework is presented in Table 1.Source: §5 · p. 5, Table 1 · p. 6

6.1 Task setting and results with deterministic reward

• The task uses an 18-state, four-action cliff-walk MDP with stochastic transitions and a −10 reward for stepping toward the cliff.Reaching the goal yields 12, and other moves yield zero; the setup makes the expected-return and small-q stationary policies differ because of cliff risk.Source: §6.1 · p. 6
• The authors restrict their q-quantile policy focus to the stationary class, although an optimal policy for criteria other than expected return need not be stationary.The paper characterizes the resulting policy as suboptimal for the q-quantile criterion under that class.Source: §6.1 · p. 6
• The q-Q policy could avert the risk of falling off the cliff, while Watkins’ Q-learning policy seemed to take that risk.The comparison used a softmax policy with time-varying βt; expected and 0.1-quantile returns from the start state were plotted.Source: §6.1 · p. 6

6 Numerical Experiments

• Ordinary-gradient q-Q learning was much less stable than versions using natural gradients, so its learning-process details were omitted.The paper notes the stability comparison without reporting further results.Source: §6 · p. 6

6.2 Results with various reward densities

• Under gamma cliff penalties, skewed-Laplace q-Q learning increased small-q quantile returns most, but tended to overestimate risk; worst-case Q-hat learning performed poorly even at the 0.01 quantile.The gamma results are reported in Table 2 (A).Source: §6 · p. 7

6 Numerical Experiments

• Figure 3 shows typical learning results for Q-learning and q=0.1-Q learning with a Gaussian distribution; experiments investigate differences among parametric return-density models using policies maximizing different return objectives.Source: §6 · p. 6, Figure 3 · p. 7

6.2 Results with various reward densities

• With Student’s t cliff penalties of infinite variance, q-Q learning with the Laplace model worked most appropriately and outperformed Watkins’ Q learning on expected return in both settings.It also worked stably when maximizing median returns.Source: §6.2 · p. 7
• The experiments tested q ∈ {0.1, 0.3, 0.5} against Watkins’ Q-learning and Q-hat learning, using gamma and infinite-variance Student’s t cliff penalties with a common mean of −10.Each simulation ran for 3×10^5 time steps.Source: §6.2 · p. 7
• The results indicate that an appropriate return-density model must be chosen for the task, with model selection or combination identified as important.Source: §6.2 · p. 7

7 Conclusion

• The proposed algorithms extend TD-learning with parametric return-density models, following a Bellman-type recursive formula for the density.Source: §7 · p. 7
• Numerical experiments indicate that the approach is promising in risk-sensitive and robust RL scenarios.Source: §7 · p. 7
• Further convergence analysis and comparison with nonparametric return-density estimation are left for future work.Source: §7 · p. 7