Source-linked AI summary
Q($\gamma_H$): Entropy-Guided Horizon Mixing in Multi-Discount Q-Learning
Arwa Albelaihi, Abdulrahman Altahhan
TL;DR
A fixed discount factor imposes one planning horizon despite different demands for early exploration and long-term credit assignment. Q(γH) mixes multiple discount-specific heads at control time using entropy-derived confidence while retaining each head’s standard update. It is most effective on delayed-credit tasks, including the reported best mean AUC on MountainCar and Acrobot against oracle-tuned fixed-discount baselines.
Problem
A single fixed γ creates a tension between stabilizing early learning with a short horizon and solving tasks that require long-range credit assignment, while per-task tuning does not accommodate changing horizon needs during training.
Source: §1 · p. 1Method
Q(γH) maintains discount-specific Q-heads and mixes their values at control time using state-dependent confidence while each head retains its standard fixed-discount Q-learning update.
Source: §1 · p. 1, §6 · p. 13Results
Across eight tabular and linear benchmarks, Q(γH) is most effective on delayed-credit tasks and achieves the best mean AUC on MountainCar and Acrobot against oracle-tuned fixed-discount baselines.
Source: §6 · p. 13Takeaways & Limitations
Adaptive horizon mixing is a lightweight and interpretable alternative to selecting one discount factor, especially when delayed credit assignment makes horizon choice difficult.
Source: §6 · p. 14The deep-RL evaluation covers only three classic discrete-action control tasks and a shared-trunk multi-head architecture, establishing applicability rather than broad deep-RL generality.
Source: §5 · p. 12
Abstract
from arXiv · showhide
A fixed discount factor in Q-learning imposes a single planning horizon throughout training, despite the differing demands of early exploration and long-term credit assignment. We propose Q($\gamma_H$), a multi-horizon Q-learning framework that maintains multiple discount-specific heads and adaptively combines them at control time using an entropy-derived confidence signal. The resulting state-dependent mixing weights shift control from shorter- to longer-horizon heads as policy confidence increases, enabling adaptive horizon mixing without modifying individual Bellman updates. Across classical control and deep reinforcement-learning benchmarks, Q($\gamma_H$) shows its strongest improvements on delayed-credit tasks relative to strong fixed-discount baselines. By decoupling horizon adaptation from the underlying learning updates, Q($\gamma_H$) provides a lightweight framework for adaptive planning-horizon control in reinforcement learning, avoiding the need for manual discount-factor selection while remaining compatible with standard value-learning methods.
1 Introduction
A fixed discount factor forces one planning horizon despite the tension between stable early learning and long-range credit assignment, and task-specific tuning cannot address changing horizon needs during training. Q(γH) mixes discount-specific heads at control time using state-dependent confidence, preserving each head’s fixed-discount update; it is strongest on delayed-credit tasks.
1 Introduction
- A fixed γ creates a tension: short horizons can stabilize early learning but may impede long-range credit assignment, while no single fixed γ resolves it.Per-task tuning does not address changes in the required planning horizon during training.Source: §1 · p. 1
- Q(γH) maintains multiple discount-specific Q-heads and mixes them using state-dependent confidence, favoring shorter horizons at low confidence and longer ones at high confidence.Each head keeps its own fixed-γk Q-learning update; the mixture-weighted discount is a descriptive summary, not an equivalent Bellman discount.Source: §1 · p. 1
- Across tabular, linear, and deep RL settings, Q(γH) is strongest on delayed-credit tasks while remaining competitive across other evaluated environments.The framework preserves standard Q-learning updates while adding state-dependent horizon mixing.Source: §1 · p. 2
2 Related Work
Q(γH) draws on multi-horizon value learning, adaptive discounting, and entropy-based uncertainty signals, but combines discount-specific heads at action selection rather than changing Bellman updates. Its distinctions include state-dependent mixing and an entropy signal used for decision ambiguity.
2 Related Work
- Q(γH) mixes discount-specific heads only at action selection, preserving standard Bellman updates and avoiding reward decomposition or distributional RL machinery.It uses per-state rather than global mixing.Source: §2 · p. 2
- Parallel value prediction across horizons appears in GVFs and discount-indexed value learning [4]; related deep methods use discount-indexed Q-heads, reward-specific discounts, multi-discount PPO advantages, or time-weighted distributional-RL heads [11].Neuroscience evidence reports dopaminergic reward prediction errors across multiple discount time constants.Source: §2 · p. 2
- Prior methods adapt γ using critic loss or entropy-based uncertainty, or formalize state- and transition-dependent discounting.Other work uses discounts for termination or continuation values [23,18].Source: §2 · p. 2
- Q(γH) uses reference-policy entropy as a lightweight signal for decision ambiguity; its head diversity is temporal, not epistemic.The gate interpolates across planning horizons rather than averaging to reduce variance.Source: §2 · p. 2
3 Method: Q(𝜸𝑯)
Q(γH) trains parallel Q-heads at fixed discounts and uses entropy-derived confidence to mix their values for action selection. The gate changes control-time aggregation, not the individual heads’ Bellman updates.
3.1 Problem Setup
- Q(γH) maintains parallel Q-learning heads with different fixed discounts and mixes their values at control time using an entropy-derived confidence signal.The problem setup allows the planning horizon to vary across states and learning stages.Source: §3.1 · p. 2
3.2 Multi-Horizon Q-Heads
- Each discount-specific head has its own parameters, discount factor, and Bellman targets while sharing the experience stream and feature representation.The method defines a set Γ of discount factors and an independent Q-head for each.Source: §3.2 · p. 3
3.3 Reference Q Construction
- The reference Q-vector is built from current multi-head estimates; the default max–min aggregation balances optimistic and conservative estimates with η = 0.5.A single fixed head is also evaluated as a reference-construction baseline.Source: §3.3 · p. 3
3.4 Entropy-Based Confidence
- Normalized Shannon entropy measures reference-policy ambiguity: entropy is maximal when actions appear equally valuable and approaches zero when one action is clearly preferred.Confidence is inversely related to normalized entropy.Source: §3.4 · p. 4
3.5 Adaptive Horizon Mixing
- Soft proximity weights map confidence to the horizon positions, so low confidence favors shorter horizons and high confidence favors longer ones.The mixture-weighted discount is only a descriptive summary of horizon emphasis, not an equivalent Bellman discount.Source: §3.5 · p. 4
3.6 Learning Rule
- Actions are selected ε-greedily from the aggregated values, then every head receives its own standard fixed-γk Q-learning update.The mixing weights affect action selection only; the algorithm applies each head’s own TD error and update.Source: §3.6 · pp. 4–5
3.7 Complexity and Practical Considerations
- Overhead scales linearly with K: memory and dense linear evaluation and updates cost O(K|A|d), while reference, entropy, and weighting add O(K|A|).Tabular and linear variants need no replay buffer, target network, or neural component; the deep extension builds on standard DQN or DDQN machinery.Source: §3.7 · p. 5
4 Experiments
The evaluation covers eight tabular/linear benchmarks and three deep-RL classic-control tasks; results favor Q(γH) most on delayed-credit problems, with mechanism and ablation analyses examining the gate.
4.2 Mechanism Analysis
- 0.960→0.800 normalized entropy accompanies mean γ rising 0.782→0.816 on MountainCar; across states, normalized entropy spans 0.042–1.000.Across states, γ=0.70 and γ=0.99 head weights span 0.031–0.540 and 0.027–0.479, respectively.Source: §4.2 · p. 7, Figure 1 · p. 9
4.1 Main Results
- Q(γH) achieves the best mean AUC on MountainCar and Acrobot, reducing AUC by 6.0% and 2.1%, respectively, versus the next-best fixed-γ baselines.Across remaining environments, it remains competitive and often improves early learning, though fixed-γ or on-policy baselines can do better when one horizon suffices or risk-sensitive behavior is required.Source: §4.1 · p. 6, Table 4 · p. 8
- Deep Q(γH)-DDQN achieves the best AUC on all three environments and exceeds DDQN by 8.5% on Acrobot-v1 and 7.0% on MountainCar-v0.Its margins over Scheduled-γ are 48.0% on Acrobot (p<0.0001) and 7.5% on MountainCar (p=0.0007).Source: §4.1 · pp. 6–7, Table 5 · p. 9
4.3 Head Disagreement as Uncertainty Proxy
- Disagreement rises as entropy falls and mean γ rises; disagreement–entropy correlations are r=−0.883 on MountainCar and r=−0.969 on Acrobot.The authors interpret these patterns as consistent with increasingly decisive reference policies as horizon-specific value estimates separate.Source: §4.3 · p. 9, Table 6 · p. 11
4.4 Ablation Study
- EarlyMean 397.63→366.36, TailMean 141.83→136.45, and AUC 216.50→203.58 with entropy-guided versus parameter-matched uniform mixing.The comparison holds five discounts, representation, training procedure, and seeds fixed.Source: §4.4 · p. 10
- MaxMin linear improves AUC by 2.3% over the best fixed-head variant; entropy, variance, and disagreement gates span 203.58–206.41 AUC.Entropy is retained as the default because it is a bounded measure of action ambiguity.Source: §4.4 · pp. 10–11, Table 7 · p. 11, Table 8 · p. 12
4.5 Sensitivity Analysis
- Sharpness β has the greatest sensitivity: too-small values spread weight diffusely and slow early learning, while τ∈[0.25, 0.50] and η∈[0.5, 0.9] are robust.The defaults are β=3.0, τ=0.25, and η=0.5.Source: §4.5 · p. 11
4.4 Ablation Study
- The K=5 and K=3 configurations differ only slightly in MountainCar performance; K=5 remains the default for finer horizon granularity.K=3 performs marginally better in the separate 20-seed analysis, while both configurations increase mean γ over training.Source: §4.4 · p. 11, Table 9 · p. 12
5 Discussion and Limitations
Q(γH) is most useful when the planning horizon is difficult to choose in advance, especially on delayed-credit tasks. Its deep-RL evidence is promising but limited in scope, and its entropy gate is sensitive to reward and Q-value scale.
5 Discussion and Limitations
- The strongest gains occur on MountainCar and Acrobot, where short horizons aid stable early learning but longer horizons support delayed credit assignment.As confidence increases, control shifts from shorter- to longer-horizon heads, gradually changing horizon emphasis within a run.Source: §5 · p. 12
- Horizon mixing offers limited benefit when a fixed discount already works well, while SARSA remains competitive in cliff environments through safer on-policy behavior.Horizon mixing does not remove this on-policy advantage.Source: §5 · p. 12
- Deep-RL evaluation covers only three classic discrete-action tasks and a shared-trunk multi-head architecture, so it establishes applicability rather than broad generality.The paper identifies larger benchmarks, separate-head architectures, continuous-action actor–critic variants, and theoretical analysis as future work.Source: §5 · p. 12
- The tabular and linear comparison uses oracle-selected fixed discounts, while Q(γH) adapts its horizon mixture online; deep DDQN uses γ = 0.99, the strongest fixed discount considered.Tabular and linear experiments isolate horizon mixing without replay buffers, target networks, or neural-optimization effects.Source: §5 · p. 12
- Mechanism analyses track confidence, the mixture-weighted discount summary, and head disagreement together, while state-wise analysis finds the learned mixture varies across MountainCar states.The mixture-weighted discount is a descriptive summary of relative emphasis on shorter- and longer-horizon heads.Source: §5 · pp. 12–13
- The entropy gate is sensitive to reward and Q-value scale under a fixed Softmax temperature, and large scale changes can materially affect performance.Scale-adaptive temperatures or normalized confidence signals are identified as future directions.Source: §5 · p. 13
6 Conclusion
Q(γH) maintains discount-specific Q-heads and mixes them at control time using state-dependent confidence, while retaining fixed-discount updates for each head. Across the evaluated benchmarks, it is most effective on delayed-credit tasks, and its deep-RL extension shows the largest gains on MountainCar-v0 and Acrobot-v1.
6 Conclusion
- Q(γH) mixes discount-specific Q-heads at control time using state-dependent confidence while each head retains a standard fixed-γk Q-learning update.The mixture-weighted discount is a descriptive summary of horizon emphasis, not an equivalent Bellman discount.Source: §6 · p. 13
- Across eight tabular and linear benchmarks, Q(γH) is most effective on delayed-credit tasks and achieves the best mean AUC on MountainCar and Acrobot against oracle-tuned fixed-discount baselines.Mechanism and ablation analyses show state-dependent mixing and improved learning efficiency from the entropy-guided gate; sensitivity analysis identifies reward/Q-value scale as a limitation.Source: §6 · p. 13
- In the three evaluated classic-control deep-RL tasks, the largest gains occur on MountainCar-v0 and Acrobot-v1, with little improvement on CartPole-v1.The paper describes these results as establishing applicability rather than broad deep-RL generality and points to larger benchmarks and continuous-action settings for future work.Source: §6 · pp. 13–14
- The results suggest adaptive horizon mixing is a lightweight, interpretable alternative to manual selection of a single discount factor, especially when delayed credit makes horizon choice difficult.Future work includes alternative multi-head architectures and scale-robust confidence mechanisms.Source: §6 · p. 14