PaperlayerView exact revision on arXiv

Source-linked AI summary

AdaGamma: State-Dependent Discounting for Temporal Adaptation in Reinforcement Learning

Yaomin Wang, Jianting Pan, Ran Tian, Xiaoyang Li, Yu Zhang, Hengle Qin, Tianshu YU

arXiv:2605.06149v1cs.LGcs.AI

TL;DR

Deep RL usually fixes the discount factor despite its effects on planning and bootstrapping, while naive learned state-dependent discounting can collapse TD errors. AdaGamma learns state-dependent discounts with return-consistency regularization, analyzes the induced Bellman operator, and integrates the method into SAC and PPO. It reports consistent continuous-control gains and statistically significant improvements in a JD Logistics online A/B test.

  • Problem

    Most deep RL methods use one fixed discount across states, while naive learned state-dependent discounting can become unstable and collapse TD errors.

  • Method

    AdaGamma learns a state-dependent discount function and uses a return-consistency objective to regularize the induced backup structure.

  • Results

    AdaGamma achieves consistent gains on continuous-control benchmarks and statistically significant improvements over standard SAC in an online JD Logistics A/B test.

  • Takeaways & Limitations

    State-dependent discounting can be effective in deep RL when paired with return-consistency regularization that prevents degenerate target manipulation.

  • Takeaways & Limitations

    The theory provides operator-level results rather than a full convergence guarantee for deep RL with function approximation.

Abstract

from arXiv · show

The discount factor in reinforcement learning controls both the effective planning horizon and the strength of bootstrapping, yet most deep RL methods use a single fixed value across all states. While state-dependent discounting is conceptually appealing, naive deep actor--critic implementations can become unstable and degenerate toward TD-error collapse. We propose AdaGamma, a practical deep actor--critic method for state-dependent discounting that learns a state-dependent discount function together with a return-consistency objective to regularize the induced backup structure. On the theory side, we analyze the Bellman operator induced by state-dependent discounting and establish its basic well-posedness properties under suitable conditions. Empirically, AdaGamma integrates into both SAC and PPO, yielding consistent improvements on continuous-control benchmarks, and achieves statistically significant gains in an online A/B test on the JD Logistics platform. These results suggest that state-dependent discounting can be made effective in deep RL when coupled with a return-consistency objective that prevents degenerate target manipulation.

1 Introduction

AdaGamma addresses the instability of learned state-dependent discounting by combining adaptive bootstrapping with return-consistency regularization. The paper supports the approach with operator-level theory, SAC/PPO integrations, benchmark evaluations, and JD Logistics validation.

  • Motivation: State-dependent discounting is motivated by heterogeneous temporal structure, where different states may require different levels of target propagation.A fixed discount factor imposes the same planning and bootstrapping tradeoff across states with different uncertainty, controllability, and credit-assignment needs.
  • Empirical validation: AdaGamma improves continuous-control results and achieves statistically significant gains over standard SAC in a four-week JD Logistics online A/B test.The evidence covers both standard benchmarks and a live real-world deployment.
  • Method: AdaGamma learns state-dependent discounting with a return-consistency objective that prevents degenerate TD-target collapse.The objective constrains the induced backup structure rather than allowing the discount module to manipulate targets freely.
  • Theory: Theoretical analysis establishes basic well-posedness properties for the Bellman operator induced by state-dependent discounting under suitable conditions.The analysis supports the operator-level behavior of adaptive discounting rather than claiming full end-to-end convergence for deep RL with function approximation.
  • Algorithmic integration: AdaGamma integrates into SAC and PPO through corresponding modifications to return estimation and optimization.These integrations make the method applicable across two actor–critic algorithms with different estimation procedures.

2 Related Work

Prior work motivates flexible discounting but generally does not provide a reliable learned state-conditioned discount module for deep actor–critic bootstrapping. AdaGamma distinguishes itself by targeting the resulting target-manipulation failure mode and validating a stabilized implementation in SAC, PPO, and deployment.

  • Discounting beyond a fixed global factor: Prior research shows that global discount choices affect optimization stability, variance, planning horizon, regularization, policy quality, robustness, and data efficiency.These findings motivate moving beyond one global discount but do not address reliable state-dependent implementation in modern deep actor–critic systems.
  • State-dependent and adaptive discounting: Existing adaptive-discounting studies use prescribed rules, schedules, or algorithm-specific signals rather than a learned state-conditioned module in standard deep actor–critic targets.The cited literature includes state-, time-, and training-stage-dependent discounting, uncertainty or advantage signals, and generalized return criteria.
  • Distinction from prior work: AdaGamma focuses on the practical instability caused by naively training a neural discount module through bootstrapped TD objectives.The paper frames state-dependent discounting as an implementation problem involving unconstrained target manipulation, not as a new modeling idea.
  • Distinction from prior work: A return-consistency objective regularizes the backup structure, making state-dependent discounting stable enough to integrate into SAC and PPO.The method is further validated in a real-world JD online deployment.

3 Preliminaries

The preliminaries define the MDP setting and explain the discount factor's dual role in objectives and Bellman bootstrapping. They also summarize SAC's soft value formulation and PPO's GAE-based advantage estimation.

  • Notation: The paper studies an infinite-horizon MDP with continuous states and actions, bounded rewards, and trajectory-induced state and state-action marginals.The transition density and reward bounds define the underlying formal setting.
  • The Discount Factor’s Dual Role: The discount factor controls both the discounted objective and the damping of future bootstrapped estimates in Bellman updates.Lower values favor short-term rewards and reduce variance at the cost of bias, whereas higher values support longer-horizon planning while amplifying value-estimation errors.
  • Soft Actor-Critic (SAC): SAC maximizes an entropy-augmented objective and uses a soft Q-function satisfying a Bellman equation.These preliminaries establish the soft actor–critic framework into which AdaGamma is integrated.
  • PPO and Generalized Advantage Estimation: PPO computes advantage estimates with generalized advantage estimation, using a recursion over TD residuals weighted by γλ.The displayed formulation describes GAE as a geometrically weighted sum of TD residuals.

4 AdaGamma: Method and Theory

AdaGamma implements state-dependent discounting through a bounded gamma network, algorithm-specific SAC and PPO adapters, and objectives that prevent TD-target collapse. Supporting operator-level results establish well-posedness and policy-improvement properties under explicit assumptions, while an error bound quantifies the cost of replacing state-dependent discounting with a fixed value.

  • 4 AdaGamma: Method and Theory: AdaGamma combines a state-dependent gamma network, SAC/PPO adapters, and a return-consistency training objective to regularize adaptive bootstrapping.The framework also includes regularization terms and cross-validated training to discourage degenerate discount choices.
  • 4.1 Gamma Network Architecture: The gamma network maps states through a sigmoid-rescaled two-layer MLP, keeping γϕ(s) within [γmin, γmax] ⊂ [0, 1).The experiments use γmin = 0.900 and γmax = 0.999; the network has hidden dimension 256 and no shared parameters with policy or value networks.
  • 4.2 The Common Interface: Bootstrapped Value Targets: AdaGamma replaces the scalar discount only in bootstrapped value targets, while SAC retains its policy and entropy updates and PPO propagates state-dependent discounts through GAE.For PPO, the resulting advantage expansion uses products of state-dependent discounts, so low-γ states suppress later residuals and high-γ stretches support longer-range credit assignment.
  • 4.3.1 The Collapse Problem: Naively minimizing TD error lets γϕ collapse toward γmin, producing near-constant values and small errors without learning informative state-dependent structure.This is the central failure mode the return-consistency objective is designed to prevent.
  • 4.3.2 Return-Consistency Objective: Return consistency trains the one-step bootstrap to match an n-step return, so γϕ adapts propagation to local predictive reliability rather than minimizing TD error through horizon collapse.The loss uses stop-gradient targets; γ → 0 generally cannot minimize it because the immediate reward will not match the multi-step return.
  • 4.3.3 Cross-Validated Variant: The cross-validated variant updates the value function on one batch split and trains γϕ on another, preventing γ → 0 from automatically reducing the training loss.The full objective additionally anchors discounts near a target, encourages smoothness, and discourages boundary values.
  • 4.4.1 Soft Policy Evaluation: Under finite actions and sup_s γ(s) < 1, the adaptive soft Bellman operator is a contraction, yielding a unique soft Q-value under a fixed policy.The proof uses a β-contraction in the sup-norm and the Banach fixed-point theorem.
  • 4.4.2 Soft Policy Improvement: Under γ(s) ∈ [0, 1) and β = sup_s γ(s) < 1, repeated adaptive soft policy evaluation and improvement converge to a policy whose Q-values dominate those of policies in Π.The operator-level analysis does not provide an end-to-end convergence guarantee for deep RL with function approximation or PPO-specific estimators.

5 Numerical Experiments

AdaGamma is evaluated across continuous-control benchmarks, online deployment, learned-discount analyses, and ablations. It consistently outperforms fixed and alternative adaptive discounts while benefiting from state dependence and training-stable return consistency.

  • 5.2 Performance in SafetyPointGoal1-v0: AdaGamma gives the strongest overall performance for both SAC and PPO on SafetyPointGoal1-v0, beyond the improvement from uncertainty-based adaptive discounting.The result supports combining state-dependent adaptation with training stability rather than merely varying the discount.
  • 5.3 High-Dimensional Control: Humanoid and Ant: AdaGamma attains the best test performance for both SAC and PPO on Humanoid-v4 and Ant-v4, with the largest gains for SAC on Humanoid-v4.These results indicate that the method scales to high-dimensional continuous-control settings.
  • 5.4 Online deployment on JD Logistics platform.: AdaGamma consistently outperforms standard SAC over four weeks in JD Logistics with statistically significant gains in recommendation-induced order volume uplift.The controlled test used 10% of production traffic, split evenly between AdaGamma-enhanced SAC and standard SAC.
  • 5.5 Analysis of Learned γϕ(s): The learned discount is analyzed on Ant-v4 through statewise heatmaps and empirical distributions over rollout states rather than uniformly across the full state space.The visualization uses torso height and planar velocity projections under a fixed trained policy.
  • 5.5 Analysis of Learned γϕ(s): Cross-algorithm discount means are closely aligned on several tasks, while SAC learns a shorter average horizon than PPO on Ant-v4.The alignment suggests task-level temporal structure, whereas the Ant-v4 difference indicates interaction between discount adaptation and policy optimization.
  • 5.6 Ablation Studies: AdaGamma’s gains persist against matched fixed discounts, showing that state dependence matters beyond choosing a better global γ.Expanded fixed-γ comparisons also leave AdaGamma superior on SafetyPointGoal1-v0, Humanoid-v4, and Ant-v4.
  • 5.6 Ablation Studies: The gamma network’s performance saturates around 256 hidden units, while n = 5 gives the best overall reward–cost tradeoff and longer horizons reduce stability, particularly for PPO.The architecture result comes from Ant-v4; the return-horizon result comes from SafetyPointGoal1-v0.
  • 5.6 Ablation Studies: Experiments also cover classic control tasks and extensions of AdaGamma to DDPG and TRPO.These additional results are reported in the appendices.

6 Conclusion

The paper concludes that AdaGamma provides a unified practical approach to state-dependent discounting while supporting operator-level theory under restricted assumptions. Related work motivates adaptive temporal weighting but leaves the deep actor–critic implementation challenge addressed here.

  • 6 Conclusion: AdaGamma integrates with SAC and PPO through bootstrapped value targets, while return consistency prevents the degenerate behavior of naive discount learning.The conclusion presents this as a practical unified implementation of state-dependent discounting.
  • 6 Conclusion: Under tabular assumptions, the paper establishes convergence properties for soft policy iteration with a fixed state-dependent discount and characterizes the resulting value discrepancy.This is an operator-level result rather than an end-to-end convergence guarantee for deep function approximation.
  • 6 Conclusion: Empirically, AdaGamma improves safety reward–cost tradeoffs, scales to high-dimensional control, remains competitive on classic benchmarks, and shows stable discount patterns across algorithms.A four-week JD Logistics A/B test further supports practical use in real-world sequential decision making.
  • A Additional Related Work Discussion: Prior work studies fixed discounting as regularization, dynamic or non-exponential schedules, state-dependent objectives, and uncertainty- or policy-feedback-based adaptation.These lines motivate flexible discounting but do not directly solve practical state-conditioned learning in deep actor–critic systems.
  • A Additional Related Work Discussion: Longer effective horizons can be statistically fragile under noisy data or poor mixing, motivating adaptation of bootstrap strength across states.The related analyses connect horizon length with estimation difficulty, sample complexity, and offline-RL trade-offs.
  • A Additional Related Work Discussion: State-dependent discounting is conceptually related to temporal abstraction because low-γ regions shorten credit-assignment horizons while high-γ regions preserve longer-range dependencies.The paper presents this relationship as complementary to, rather than a replacement for, options and termination methods.

B.1 Proof of Lemma 1

Under a uniform upper bound β = sup_s γ(s) < 1, the adaptive-discount soft Bellman operator is well posed and supports policy evaluation, improvement, and iteration. The analysis also bounds deviation from fixed discounting and characterizes adaptive GAE weighting.

  • B.1 Proof of Lemma 1: Under β = sup_s γ(s) < 1, repeated adaptive-discount soft policy evaluation converges to a unique soft Q-value function.The proof invokes contraction and the Banach fixed-point theorem.
  • B.2 Proof of Lemma 2 / B.3 Proof of Theorem 1: With γ(s) ∈ [0, 1) and β < 1, soft policy improvement and evaluation converge through repeated application to a policy dominating every policy in soft Q-value.Bounded rewards, bounded entropy terms, and geometric convergence provide the required upper bound.
  • B.4 Proof of Theorem 2: The adaptive-versus-fixed soft Q-function discrepancy scales linearly with max_s |γ(s) − γ| under bounded rewards, positive policy probabilities, and β < 1.The bound also depends on the factors (1 − β)^-1, (1 − γ)^-1, and R + α log(1/ϵ).
  • B.5 Proof of Proposition 1: State-dependent discounting weights GAE advantages through products of state-dependent discounts across future steps.
  • C Details of Uncertainty-rule adaptive-γ baseline: The uncertainty-rule baseline shortens the effective horizon when critic disagreement is high and extends it when critics agree.SAC uses twin-Q disagreement, while PPO adds an auxiliary value network only for discount computation.
  • D.1 Cross-algorithm consistency: Table 2 reports mean learned γϕ(s) separately for SAC-AdaGamma and PPO-AdaGamma.

D.2 Training Objective Comparison

The empirical analysis compares learned-discount variants, architecture capacity, return horizons, fixed-discount controls, and SAC/PPO integrations. AdaGamma is reported as strongest overall, with state dependence and a five-step consistency target supported as important design choices.

  • D.3 Network architecture: On Ant-v4, SAC-AdaGamma performance is presented as a reward mean ± std curve across gamma-network hidden dimensions 8, 16, 32, 64, 128, 256, and 512.
  • D.4 SAC/PPO with AdaGamma under different return horizon n: Table 4 reports mean reward and cost, including standard deviations, for SAC-AdaGamma and PPO-AdaGamma under different return horizons n ∈ {5, 10, 20}, enabling comparison across these horizons.
  • D.5 Comparison to fixed-γ: The study evaluates SAC and PPO using fixed γ values around the learned mean and through a fixed-γ grid search, as reported in Tables 5 and 6.
  • E The algorithm framework under AdaGamma: AdaGamma combines a gamma network, algorithm-specific SAC/PPO adapters, and a return-consistency training objective.The framework inserts state-dependent discounts into bootstrapped value estimation and provides separate algorithm variants.
  • E.1 SAC-AdaGamma: The SAC implementation uses adaptive targets and return-consistency gamma updates, while policy and temperature updates remain unchanged.
  • E.3 Reward Curves: The experiments report reward curves for SAC and PPO with AdaGamma and baselines on SafetyPointGoal1-v0, Humanoid-v4, and Ant-v4.

F Generalization to TRPO and DDPG

The paper extends AdaGamma beyond SAC and PPO by evaluating integrated DDPG and TRPO agents on matched safety and locomotion benchmarks. The supplied passages define the transfer protocol but do not report comparative outcomes.

  • F Generalization to TRPO and DDPG: DDPG and TRPO are evaluated with AdaGamma on SafetyPointGoal1-v0, Humanoid-v4, and Ant-v4.The study keeps environments, seeds, horizons, evaluation, and discount baselines aligned with the main experiments.
  • F Generalization to TRPO and DDPG: The benchmark suite covers safety-constrained reward–cost tradeoffs, long-horizon coordination, and contact-rich dynamics.

G Classic Control Results

Classic-control experiments show that AdaGamma often matches or improves fixed-discount baselines, while snapshots provide qualitative views of trained Humanoid and Ant agents. Performance remains below task optima and variable on several tasks.

  • G Classic Control Results: AdaGamma improves or matches the base algorithm on most classic-control tasks for both SAC and PPO.
  • G Classic Control Results: On CartPole-v1, both SAC-AdaGamma and PPO-AdaGamma reach the maximum score of 500 and reduce residual variance versus fixed-discount baselines.
  • G Classic Control Results: On Pendulum-v1, MountainCarContinuous-v0, and Acrobot-v1, performance remains below the task optimum and shows greater variability, despite often being competitive or modestly better.
  • H Snapshots of Humanoid-v4 and Ant-v4: The paper presents snapshots of trained SAC and PPO AdaGamma agents on Humanoid-v4 and Ant-v4.

J Limitations

AdaGamma’s advantages are most evident when effective planning horizons vary across states, while gains may be smaller on temporally homogeneous tasks. The supplied materials also reference benchmark results, snapshots, and task-specific hyperparameters without reporting their values here.

  • J Limitations: AdaGamma is most advantageous in environments with substantial state-dependent variation in effective planning horizon.Adaptive discounting is intended to capture heterogeneous temporal structure more effectively than a single global discount.
  • J Limitations: Gains may be more modest on temporally homogeneous tasks already well served by a fixed discount factor.
  • J Limitations: The referenced evaluation materials include SAC and PPO test results, Ant-v4 performance, task-specific hyperparameters, and snapshots on Ant-v4 and Humanoid-v4.
Loading 2605.06149v1…