Hyperbolic Discounting and Learning over Multiple Horizons

1 INTRODUCTION

• A discount factor's magnitude sets an effective horizon, while exponential discounting assumes a known constant environmental risk that may not hold in richer environments.The paper argues that both the magnitude and form of discounting establish implicit priors over learned solutions.Source: §1 · p. 1
• Humans and animals often discount hyperbolically, whereas RL agents commonly use exponential discounting.The figure contrasts hyperbolic curves with shallower declines at long horizons against exponential curves that decline at a constant rate.Source: §1 · p. 1, Figure 1 · p. 2
• Learning Q-values with different exponential discount factors can approximate hyperbolic and other non-exponential discounts, using deep networks with a shared representation.The approach is demonstrated in Pathworld and considered with higher-dimensional agents in the ALE.Source: §1 · p. 3
• Multi-horizon learning often improves over C51 and Rainbow in the ALE, independently of enabling alternative discounting schemes.The paper presents this as an auxiliary-task benefit.Source: §1 · p. 3

2 RELATED WORK

• Dasgupta and Maskin (2005) link hyperbolic discounting and preference reversals to uncertainty over reward timing, while this paper adopts Sozou's (1998) hazard account for simplicity.The paper notes that the reward-timing account is a softer alternative to per-time-step death through hazard.Source: §2 · p. 3
• Earlier TD approaches to hyperbolic discounting include quasi-hyperbolic solutions and HDTD, which adds two free parameters to adjust for reward-level differences.The discussion notes that simple hyperbolic modifications to TD have faced consistency issues.Source: §2 · p. 3
• Distributed exponential discount factors have modeled hyperbolic preferences in adjusting-delay assays; this paper extends the approach to other discounts and shared deep representations.Kurth-Nelson & Redish (2009) proposed µAgents that vote on actions; the paper's account also cites fMRI studies as consistent with the approximation.Source: §2 · p. 4
• Horde and γ-nets treat discount factor as a query over timescales, alongside other flexible-discounting approaches.Other examples include multiple value functions with different factors and an estimator imitating average return.Source: §2 · p. 4
• Prior multi-time-scale work includes multiple reward functions or reward components with distinct discounts, whereas this paper models the same rewards over different horizons.The related-work discussion also notes two factors used for intrinsic and extrinsic rewards and the concurrent TD(∆)-algorithm.Source: §2 · p. 4
• Auxiliary tasks have been used to support representation learning and augment low-information extrinsic rewards in reinforcement learning.The paper cites Suddarth and Kergosien (1990), Lample & Chaplot (2017), Mirowski et al. (2016), Jaderberg et al. (2016), Veeriah et al. (2018), and Sutton et al. (2011).Source: §2 · p. 4

3 BELIEF OF RISK IMPLIES A DISCOUNT FUNCTION

• A future reward is discounted by the probability of surviving to collect it, and survival is defined as the probability of surviving until time t.Hazard rate is introduced as the negative rate of change of log-survival.Source: §3 · pp. 4–5

3.1 KNOWN HAZARD IMPLIES EXPONENTIAL DISCOUNT

• A known constant hazard rate yields exponential survival and connects the RL discount factor γ to the per-time-step probability that the episode continues.The paper expresses the constant hazard as λ = −ln(γ); a single γ therefore assumes a known constant hazard.Source: §3.1 · p. 5

3.2 UNCERTAIN HAZARD IMPLIES NON-EXPONENTIAL DISCOUNT

• Weighting exponential survival rates over an uncertain hazard prior yields non-exponential discount functions; different priors can produce different functions.The figure compares hazard-rate priors with their resulting discount functions, including the delta prior associated with exponential discounting.Source: §3.2 · pp. 5–6, Figure 2 · p. 6

4 HAZARD IN MDPS

• At each episode’s start, the model samples a hazard rate and uses hazard-modified transitions, bounded rewards, and a general discount function in a non-infinite episodic MDP.The transition probability is scaled by e^-λ.Source: §4 · p. 6

4.1 EQUIVALENCE BETWEEN HAZARD AND DISCOUNTING

• A discounted MDP’s value function is equivalent to the value function under a corresponding hazard distribution and hazard-modified transitions, without discounting rewards.For exponential discounting, the corresponding per-step hazard rate is λ = −ln γ.Source: §4.1 · p. 7
• Hyperbolic discounting is also linked to a specific hazard distribution, p_k(λ).Source: §4.1 · p. 7

5 COMPUTING HYPERBOLIC Q-VALUES FROM EXPONENTIAL Q-VALUES

• Exponentially discounted Q-values can be repurposed to compute hyperbolic and other non-exponential Q-values, addressing the challenge of using TD methods beyond exponential discounting.TD methods exploit the Bellman equation, which relates values across successive states under exponential discounting.Source: §5 · p. 7

5.1 COMPUTING HYPERBOLIC Q-VALUES

• The hyperbolic discount factor can be expressed as an integral over exponential discount factors, so combining their Q-values yields hyperbolically discounted values.The integral representation considers an infinite set of γ values; the paper limits its treatment to non-infinite episodic MDPs.Source: §5.1 · p. 8

5.2 GENERALIZING TO OTHER NON-EXPONENTIAL Q-VALUES

• An exponential weighting condition provides a way to express other discount functions through weighted exponential Q-values, with hyperbolic discounting among the supported cases.The paper also describes deriving weightings from hazard priors.Source: §5.2 · p. 9

6 APPROXIMATING HYPERBOLIC Q-VALUES

• A finite number of exponentially discounted Q-values approximates the integral needed for hyperbolic Q-values, making the computation practical with standard Q-learning.The number of Q-values is controlled by a free hyperparameter.Source: §6 · p. 9, §6.1 · pp. 9–10

6.1 APPROXIMATING THE DISCOUNT FACTOR INTEGRAL

• Table 1 matches hazard priors to their corresponding discount functions and exponential weightings.It includes the Dirac delta prior as the typical exponential-discount case and also covers richer priors.Source: Table 1 · p. 10
• The approximation uses a Riemann sum and combines the Q-values as a weighted average rather than sampling a discount factor for each agent.The paper reports that emphasizing larger γ values performs well in Section 7.Source: §6.1 · p. 10

7.1 WHEN TO DISCOUNT HYPERBOLICALLY?

• Hyperbolic discounting is most relevant with an uncertain hazard and non-trivial intertemporal trade-offs; with one true hazard rate, an optimal exponential γ exists.Without non-trivial trade-offs, all discounting schemes yield the same optimal policy.Source: §7.1 · pp. 10–11

7.2 PATHWORLD DETAILS

• Pathworld trains hyperbolically discounted returns without hazard, then evaluates undiscounted returns under deployment risk across paths with longer lengths and larger terminal rewards.The agent chooses one path; deployment draws an unobserved hazard, so longer paths carry more risk.Source: §7.2 · p. 11, Figure 5 · p. 12

7.3 RESULTS IN PATHWORLD

• With a matched hazard prior, the approximation tracks theoretical path values, while γ = 0.975 can select the correct path yet still yield high error across all paths.The comparison is about the value profile, not just the best path choice.Source: Figure 6 · p. 12, §7.3 · p. 12, Table 2 · p. 12
• Hyperbolic values remain more robust than exponential discounting when the agent’s hazard prior mismatches the true coefficient or functional form.The experiments report lower errors for factor-of-two coefficient mismatches and better matching when the true hazard is uniform.Source: Figure 7 · p. 13, Table 3 · p. 13, Figure 8 · p. 13

8 ATARI 2600 EXPERIMENTS

• The evaluation builds on the Dopamine Rainbow variant, which implements distributional RL, n-step returns, and prioritized replay from Rainbow’s six improvements.The study uses three of the six improvements from the original Rainbow paper.Source: §8 · p. 14
• The agent uses a shared state representation and separate lightweight affine heads to predict Q-values for multiple discount factors.The multi-horizon model can construct policies from a particular value or a weighted combination.Source: Figure 9 · p. 14, §8 · p. 14
• Hyper-Rainbow improved over Rainbow on 14 of 19 sampled Atari games, sometimes by large margins.The evaluation used a random subset of Atari 2600 games.Source: §8 · p. 14

9 MULTI-HORIZON AUXILIARY TASK RESULTS

• On the 19-game subset, Multi-Rainbow performed nearly as well as Hyper-Rainbow, with no significant difference between hyperbolic and exponential behavior policies.This suggests the ALE improvement emerges from the multi-horizon auxiliary task.Source: Figure 10 · p. 15, §9 · p. 15

9.1 ANALYSIS AND ABLATION STUDIES

• Multi-horizon C51 improved over base C51 on 9 of 10 sampled Atari games.The experiment examines how the auxiliary task interacts with other deep-RL improvements.Source: §9.1 · p. 15
• The current prioritization scheme averaged temporal-difference errors across Q-values, and four of ten games were negatively affected.The authors report that this implementation does not generally build successfully on prioritized replay; alternative schemes show preliminary promise.Source: §9.1 · p. 15

9 MULTI-HORIZON AUXILIARY TASK RESULTS

• Across the full ALE, the multi-horizon auxiliary task empirically benefits Rainbow.The full-ALE evaluation follows the recommended protocol; Figure 11 reports improvement over Rainbow.Source: §9 · p. 15, Figure 11 · p. 16

• The work challenges the premise that RL should maximize exponentially discounted returns using a single discount factor.Source: §10 · p. 16
• Learning multiple horizons broadens the learning algorithms and enables acting according to new discounting schemes.Source: §10 · p. 16
• The method approximates hyperbolic discounting, performs better in hazardous MDP distributions, and treats multiple-horizon learning as a stand-alone auxiliary task.The authors position it as part of an algorithmic toolkit for modeling alternative discount functions.Source: §10 · p. 16

11 FUTURE WORK

• A constant hazard-rate model does not capture time-varying hazards or the policy-environment interplay that can make hazard policy-dependent.The paper points to state-action-dependent discounting as a possible formalism for more general time-preference schemes.Source: §11 · p. 16

A EQUIVALENCE OF HYPERBOLIC DISCOUNTING AND EXPONENTIAL HAZARD

• Hyperbolic discounting is equivalent to averaging exponentially discounted values under a particular hazard distribution, and discount factors can support policies robust to hazards.Source: A EQUIVALENCE OF HYPERBOLIC… · p. 22

B ALTERNATIVE DISCOUNT FUNCTIONS

• A single discount factor corresponds to a single hazard-rate assumption, while the appendix relates alternative hazard priors to alternative discount functions.The appendix considers delta, exponential, and uniform hazard priors, though the supplied passages do not specify all resulting functions.Source: §B.1 · pp. 22–23, §2 · p. 22, §B.3 · pp. 23–24

C DETERMINING THE γ INTERVAL

• The largest learned discount factor defines γmax; allowing it to approach 1 may cause learning instabilities, so the interval is bounded at a known stable value.This bound induces an approximation error.Source: C DETERMINING THE γ INTERVAL · p. 24

D ESTIMATING HYPERBOLIC COEFFICIENTS

• Lower-bound Riemann sums approximate the hyperbolic coefficients, but the Laplace-transform form is sharply peaked as γ approaches 1.The passages compare two integral forms and note that more sophisticated integral estimates are possible.Source: D ESTIMATING HYPERBOLIC COEFFICIENTS · p. 24

E PERFORMANCE OF DIFFERENT REPLAY BUFFER PRIORITIZATION SCHEME

• Prioritizing replay by the largest-γ TD errors appears better than averaging TD errors evenly across all heads.The largest γ also defines the Q-values used for action selection; the reported alternative-scheme results are preliminary.Source: E PERFORMANCE OF DIFFERENT REPLAY… · p. 24

F APPROXIMATION ERRORS

• As γmax decreases, the Riemann-sum approximation error increases; as γmax approaches 1, the approximation becomes more accurate up to small random errors.The error arises from evaluating the integral at γmax rather than 1 and increases with t.Source: F APPROXIMATION ERRORS · p. 24, Figure 15 · p. 26

G HYPERPARAMETERS

• The experiments use each base algorithm’s default hyperparameters, and the Multi-C51 and Multi-Rainbow configurations are listed separately.Source: G HYPERPARAMETERS · p. 27, Table 6 · p. 27

H AUXILIARY TASK RESULTS

• Table 7 reports Multi-Rainbow returns against the DQN, C51, and Rainbow agents.The supplied passages identify the comparison but do not provide its return values.Source: H AUXILIARY TASK RESULTS · p. 27, Table 7 · p. 28