Source-linked AI summary
Reward Inflation: A Healthy Stimulus for Reinforcement Learning
Ganghun Lee, Minji Kim, Minsu Lee, Byoung-Tak Zhang
TL;DR
Reward magnitudes are usually held fixed during training, leaving their temporal scale underexplored. The paper introduces gradual reward inflation and adaptive Fed, reporting broad benefits across ALE and MuJoCo tasks, with Fed further improving results.
Problem
The temporal dynamics of reward scale during RL training remain underexplored despite reward optimization’s central role.
Source: §1 · p. 1Method
The paper gradually and exponentially scales rewards during training and introduces Fed, which adapts the inflation rate using moving-average policy-gradient norms.
Source: §1 · pp. 1–2Results
Moderate reward inflation consistently improves performance across broad ALE and MuJoCo task sets, with reported gains of 11.53% over baseline on ALE and 11.48% over the non-inflation baseline on MuJoCo; Fed further raises them to 12.62% and 16.48%.
Source: §1 · p. 2, §6 · p. 10Takeaways & Limitations
The results support temporal reward dynamics as a lightweight direction for improving RL, while Fed offers dynamic inflation control that enhances robustness and adaptability.
Source: §6 · p. 10Experiments cover only moderate training horizons; longer runs may destabilize optimization, optimal schedules may be task-dependent, and Fed uses multiple hyperparameters and a heuristic rule.
Source: §5.5 · pp. 9–10
Abstract
from arXiv · showhide
Reward serves as the primary learning signal in reinforcement learning (RL). However, while reward magnitudes are typically held fixed throughout training, their temporal modulation remains underexplored. In this paper, we propose reward inflation, a gradual scaling of rewards over the course of training, and show that it can act as a healthy stimulus for RL. Theoretically, reward inflation induces an implicit recency weighting that upweights recent transitions during policy updates, enabling faster adaptation. We further show that, by sustaining gradient signals as the policy saturates, reward inflation suppresses the emergence of dormant neurons and helps preserve plasticity. Empirical results on ALE games and MuJoCo tasks corroborate these findings, showing that an appropriate level of reward inflation benefits a broad range of tasks. Finally, we introduce Fed, an adaptive variant that adjusts the inflation level on the fly, and find that it often improves upon fixed inflation.
1 Introduction
The paper addresses underexplored temporal changes in reward scale by proposing gradual reward inflation and an adaptive variant, Fed. It links inflation to recency weighting and preserved plasticity, and reports broad benefits on ALE and MuJoCo tasks.
1 Introduction
- The paper identifies temporal reward-scale dynamics as an underexplored aspect of reward optimization, alongside established reward design and augmentation approaches.Examples of prior approaches include reward normalization, reward shaping, intrinsic motivation, curriculum rewards, and human- or LLM-augmented rewards.Source: §1 · p. 1
- Reward inflation gradually and exponentially scales rewards during training, while Fed adaptively adjusts the inflation rate using moving-average policy-gradient norms.Previously stored replay-buffer rewards remain unchanged.Source: §1 · p. 1
- Reward inflation induces implicit recency weighting in TD updates, assigning greater effective weight to recent replayed transitions.The paper also argues that gradually increasing gradient norms delays network saturation and helps preserve plasticity by suppressing dormant neurons.Source: §1 · pp. 1–2
- 11.53% improvement over the baseline on ALE and 11.48% improvement over the non-inflation baseline on MuJoCo; Fed increased these gains to 12.62% and 16.48%, respectively.The paper reports that these gains are more consistent than those of reward normalization, prioritized experience replay, learning rate inflation, and recency-weighted sampling.Source: §1 · p. 2
2 Related works
The related work positions reward inflation alongside reward design, replay weighting, and plasticity interventions, distinguishing it by scaling reward magnitudes over training rather than changing reward structure or sampling.
Reward shaping and normalization
- Reward inflation gradually scales reward magnitude over training while preserving reward semantics, unlike per-state reward reshaping.The related work also contrasts it with reward and value normalization methods.Source: §2 · p. 2
Implicit weighting of replayed experience
- Unlike replay-prioritization and recency-sampling methods, reward inflation increases recent transitions’ influence on TD updates by amplifying rewards rather than changing replay or sampling.The passage notes theoretical support for desirable properties of recency-based TD estimators [39].Source: §2 · p. 2
3 Method
The method defines a gradually changing reward scale and an adaptive controller, Fed, for adjusting inflation during training.
3 Method
- The method section presents reward-inflation notation and formulation, then introduces Fed as an adaptive inflation-control mechanism.Source: §3 · p. 2
3.1 Preliminaries
- An RL problem is formulated as an MDP with state and action spaces, transitions, rewards, and a discount factor; the objective is to find a policy maximizing expected return.Source: §3.1 · p. 3
3.2 Reward Inflation
- Reward inflation progressively scales rewards using an inflation rate and cumulative inflation level; positive rates grow the scale, zero recovers standard RL, and negative rates contract it.Inflation is applied before rewards enter the replay buffer, so stored rewards remain fixed; rates at or below -1 are excluded because rewards vanish or invert.Source: §3.2 · p. 3
3.3 Fed: Adaptive Inflation Control
- Fed adjusts the inflation rate in response to recent gradient-norm dynamics, using short- and long-term exponential moving averages whose ratio signals overheating or downturn.Its update rule includes proportional and derivative responses and a soft update to prevent abrupt changes.Source: §3.3 · p. 3
4 Theoretical Analysis of Reward Inflation
The analysis argues that positive reward inflation preserves the optimal policy at each fixed step while changing learning dynamics: it reweights updates toward recent transitions and sustains gradient signals associated with plasticity.
4 Theoretical Analysis of Reward Inflation
- At each fixed training step, positive reward scaling preserves the optimal policy, although its gradual evolution makes the effective MDP mildly non-stationary.The reward structure is unchanged; only its magnitude changes.Source: §4 · p. 4
- For positive inflation, later-collected transitions contribute larger reward-driven update components; deflation reverses this ordering.The bootstrapping components cancel when comparing updates across collection times, leaving the reward-driven components to determine the difference.Source: §4 · p. 5
- With exponential inflation, relative transition weights depend on the time gap, remaining stationary through training; linear or polynomial profiles flatten over time.Transitions Δ steps apart retain relative weight (1 + ρ̃)^Δ under exponential inflation.Source: §4 · p. 5
- The bootstrapping component cumulatively scales Q-values by the replay buffer’s average inflation level, which tracks recent inflation in a finite FIFO buffer.That average drifts upward under positive inflation and downward under deflation.Source: §4 · p. 5
- Under positive inflation, gradient norms grow with the buffer-average scale, sustaining effective learning rates as policies saturate and helping preserve plasticity.The authors connect amplified gradients to delayed saturation and suppression of dormant neurons.Source: §4 · p. 6
5 Experiment
Experiments across ALE and MuJoCo compare reward inflation and Fed with baseline and related methods, then examine mechanisms, robustness, and limitations. Moderate inflation improves average benchmark scores, while results vary with inflation rate and task.
5.1 Experimental Setup
- Experiments use DQN on 30 ALE games and SAC on eight MuJoCo tasks, averaging eight and ten seeds, respectively.Training ran for 10M steps on ALE and 3M on MuJoCo.Source: §5.1 · p. 6
5.2 Performance Evaluation
- ALE reward inflation improved average HNS 11.53% over baseline and Fed 12.59%; on MuJoCo, 2% inflation improved normalized score 11.48% and Fed 16.48%.The ALE values are HNS gains; the MuJoCo values are normalized-score gains.Source: §5.2.1 · p. 7, §5.2.2 · p. 7, Table 2 · p. 7
5.2.2 MuJoCo
- Across MuJoCo tasks, 1–5% inflation showed slight gains, but −2% deflation and 10% inflation degraded normalized score by 24.66% and 23.08%, respectively.The best inflation rate appears task-dependent; related methods showed inconsistent or weaker gains than moderate inflation and Fed.Source: §5.2.2 · pp. 7–8
5.3 Empirical Analysis of Reward Inflation
- On HalfCheetah-v4, nominal rewards rose faster than real rewards under 2% inflation, while inflation amplified the TD-error gap between recent and older transitions.Gradient-norm trajectories also differed across inflation, baseline, and deflation conditions.Source: §5.3 · p. 8, Figure 1 · p. 8
- On HalfCheetah-v4, dormant-neuron proportion fell as inflation rose; positive inflation increased gradient norms, while deflation accelerated gradient decay and dormancy.The lowest dormant-neuron rate did not mean the best performance: 10% inflation had the fewest dormant neurons but the worst overall result.Source: §5.3 · pp. 8–9, Figure 2 · p. 8
- Fed lowered inflation early on fast-learning HalfCheetah-v4 but raised it during reward-scarce early training on Swimmer-v4, then converged toward its 2% target.Its adjustments responded to the short-long gradient norm ratio.Source: Figure 3 · p. 8, §5.3 · p. 9
5.4 Additional Results
- Fed was reasonably robust to moderate hyperparameter changes, and 2% inflation maintained comparable or slightly better performance than baseline under added noise.The noise tests included observation, reward, and action noise on four MuJoCo tasks.Source: §5.4 · p. 9
5.5 Limitations
- Experiments cover moderate training horizons; at sufficiently long horizons, the growing reward scale may destabilize optimization, and task-dependent optima may preclude a universal schedule.Fed also relies on multiple hyperparameters and a heuristic control rule.Source: §5.5 · pp. 9–10
6 Conclusion
The paper presents reward inflation as a lightweight temporal reward-scaling scheme and reports theoretical mechanisms and broad empirical benefits, with Fed further improving training through dynamic control.
6 Conclusion
- Moderate reward inflation consistently improves performance across a broad range of ALE games and MuJoCo tasks.The conclusion reports that Fed further enhances robustness and adaptability through dynamic inflation control.Source: §6 · p. 10
- Reward inflation induces implicit recency weighting and suppresses dormant neurons, mechanisms that can benefit training.Source: §6 · p. 10
- Fed adaptively controls inflation dynamically and further enhances robustness and adaptability.Source: §6 · p. 10
- The paper frames temporal reward dynamics as a lightweight direction for further investigation in RL.Source: §6 · p. 10
A.1 RL Hyperparameters
The appendix identifies the SAC and ALE DQN configurations and notes selected MuJoCo training choices, including longer training and warm-up and task-specific discount adjustments.
A.1 RL Hyperparameters
- SAC experiments use the hyperparameters listed in Table 3, which mostly follow CleanRL’s implementation.The appendix also specifies that the ALE DQN agents use the CleanRL ALE setting.Source: §A.1 · p. 14, Table 3 · p. 14
- MuJoCo runs use 3M timesteps to examine long-term effects and 100K warm-up steps for stability.The authors report that 5K warm-up steps occasionally caused severely poor performance for both baseline and proposed agents.Source: §A.1 · p. 14
- The discount factor is adjusted for Hopper-v4 and Ant-v4 to reduce performance volatility, with specifications listed in Table 4.The baseline with those hyperparameters outperformed its default γ = 0.99 setting on each task.Source: §A.1 · p. 14, Table 4 · p. 14
A.2 Implementation of Recency-Weighted Sampling (RWS)
Recency-weighted sampling (RWS) increases the sampling frequency of recent replay transitions through a geometric age-based probability, unlike reward inflation, which keeps sampling uniform and changes reward magnitudes.
A.2 Implementation of Recency-Weighted Sampling (RWS)
- RWS draws replay samples with an age-based probability that gives more recent transitions higher sampling probability.Sample age a ranges from 1 to buffer size N, and decay rate r satisfies 0 < r < 1.Source: §A.2 · p. 14
- For older sample age a2 than a1, the probability ratio p(a2)/p(a1) = r^(a2−a1), imposing geometric decay over sample age.The setting r = 0.981/100000 aligns with the stated notion of 2% reward inflation.Source: §A.2 · p. 14
- RWS increases exposure to recent transitions by biasing sampling, whereas reward inflation preserves uniform sampling and raises rewards for recently collected transitions.Source: §A.2 · p. 14
A.3 Resources
Experiments used multiple Linux servers with GPUs and CPUs; a reported run took about eight hours on average.
A.3 Resources
- Experiments ran on multiple Linux servers equipped with NVIDIA GPUs and AMD CPUs.Reported hardware included RTX 4090, A6000, A5000, and TITAN RTX GPUs, paired with several AMD EPYC and Ryzen Threadripper CPUs.Source: §A.3 · p. 15
- A single ALE run of 10M steps or MuJoCo run of 3M steps took approximately 8 hours on average.Source: §A.3 · p. 15
- The reported resource estimate covers both ALE and MuJoCo experimental runs.Source: §A.3 · p. 15
B Additional Results
Additional results examine Fed hyperparameters, task-wise comparisons, more uncertain conditions, and dormant-neuron ratios across tasks and activation functions.
Fed hyperparameters
- Larger K, λ1, and λ2 and smaller τρ drive more aggressive inflation-rate fluctuations; β settings amplify gradient-norm variation, while target ρ* does not affect volatility.The caption also notes that smaller β values and larger gaps between βs and βl increase volatility.Source: Figure 5 · p. 16
- Fed learning curves compare five hyperparameter settings across four tasks, with default values marked and episode rewards normalized.Source: Figure 6 · p. 16
Atari comparisons
- Atari task-wise bootstrap 95% CIs compare baseline with 2% inflation and Fed.Source: Figure 7 · p. 16
MuJoCo comparisons
- MuJoCo task-wise bootstrap 95% CIs compare baseline with the best-performing inflation rate from Table 1 and with Fed.Source: Figure 8 · p. 17
Dormant neurons
- Additional dormant-neuron ratio results span tasks, with HalfCheetah-v4 corresponding to the main paper’s Figure 2(a).Source: Figure 9 · p. 17
- With LeakyReLU, neurons are counted as dormant when their negative activation ratio exceeds 99% across inputs.Source: Figure 10 · p. 18
- With SiLU, neurons are counted as dormant when activation variance across inputs falls below 0.0001.Source: Figure 11 · p. 18
Uncertain conditions
- Figure 12 contrasts 0% and 2% inflation under more uncertain conditions.Source: Figure 12 · p. 19