Deep Weighted Bellman Residual Minimization for $Q^*$ Estimation

1. Introduction

• Off-policy evaluation faces distribution shift, while deep Q-learning methods also encounter Q-value overestimation and substantial trajectory-sample requirements.MABO uses importance weighting for shift correction but requires repeated minimax equation sets, which the paper describes as difficult to implement.Source: §1 · p. 2
• The proposed framework combines expert demonstrations with behavioral data to model distribution shifts in offline RL without restrictive assumptions on the underlying data distributions.The theoretical analysis also removes the completeness assumption and establishes convergence rates for the density-ratio and deep estimators.Source: §1 · p. 3

1.1. Related Work

• Prior work addresses distribution shift with density-ratio weighting and minimax estimation, while expert demonstrations are used in imitation and inverse reinforcement learning.The reviewed minimax methods can involve complex iterative computations and high computational demands.Source: §1.1 · pp. 4–5

1.2. Outlines

• The paper proceeds from preliminaries to the proposed method, theoretical analysis, numerical experiments, and conclusion.Proofs are deferred to the appendix.Source: §1.2 · p. 5

2.1. Notations

• The paper uses asymptotic notation, vector and function norms, and definitions of Hölder classes and covering numbers.The supplied passages state the O and Ω conventions and identify the Hölder and covering-number definitions.Source: §2.1 · pp. 5–6, §2.3 · pp. 7–8

2.2. Markov Decision Process

• The analysis assumes continuous state–action spaces on [0, 1]^d, i.i.d. data, and separate behavior-policy and expert-policy datasets.The MDP is discounted, and the optimal action-value function satisfies the optimal Bellman equation.Source: §2.2 · p. 6

2.3. Deep Neural Networks

• The function class uses ReLU deep neural networks, with depth, width, size, and weight bounds specified for the analysis.The network has D hidden layers; its width is the maximum hidden-layer dimension and its size counts nonzero parameters.Source: §2.3 · p. 7

3. Methodology

• The objective estimates Q* by minimizing a weighted Bellman-residual loss over a ReLU neural-network function class.The weight is the density ratio between state–action distributions induced by the optimal and behavior policies.Source: §3 · pp. 8–9
• Because the optimal policy is unknown, the method uses an expert policy as an intermediate distribution to approximate the optimal-to-behavior density ratio.The expert policy is assumed to approximate the optimal policy with bounded error, and its ratio to the behavior policy serves as a surrogate weight.Source: §3 · p. 8
• The empirical Q* estimator replaces the surrogate density ratio with an estimated weight in the Bellman-residual minimization.The target is formed from the reward plus the discounted maximum next-state action value.Source: §3 · pp. 8–9
• The density-ratio estimator uses robust density-ratio techniques based on Basu’s power divergence, avoiding nonparametric density estimation and bandwidth selection.Its empirical risk minimization uses state–action samples from both policies, with their union independent of the Bellman-residual batch.Source: §3 · p. 9

4. Theoretical Analysis

• The guarantees rely on additional density-ratio conditions and a Hölder-continuous optimal action-value function; the stated statistical analysis assumes i.i.d. data.The authors note that dependent data can be handled using a mixing assumption to establish a corresponding statistical error bound.Source: §4 · pp. 10–11
• Ignoring logarithmic orders and other terms, the excess-risk convergence rate matches that of MABO [57].The theorem's bound combines density-ratio estimation, weighted minimization, and expert-policy approximation error.Source: §4 · pp. 10–11
• The analysis dispenses with the completeness assumption commonly used in deep reinforcement-learning convergence results.That assumption requires T*g to belong to H for every g in the DNN function class G; expert demonstrations bridge the behavior and target policies.Source: §4 · p. 11
• The proof decomposes excess risk into four error components, bounding density-ratio and expert-policy terms, statistical error, and approximation error.The statistical term is bounded using Rademacher complexity, while the approximation term is bounded under Assumption 4.3.Source: §4 · pp. 11–13

5.1. Linear Environment

• The linear experiments use a two-dimensional continuous state and action space, three groups with distinct system configurations, and an LQR numerical solution to approximate the optimal policy.The experiments record cumulative rewards over training and evaluate methods after each epoch and in additional post-training epochs.Source: §5 · p. 14, §5.1 · pp. 14–15, Table 1 · p. 14
• In the linear environment, all methods’ training rewards rise and then stabilize, while the proposed method consistently outperforms other algorithms in evaluation.Its evaluation rewards also show smaller fluctuations than some baselines across evaluation epochs.Source: §5.1 · p. 15, Figure 1 · p. 16, Figure 2 · p. 16

5. Numerical Experiments

• The method leads DQN, DDPG, and MABO in average reward across three linear experiments and has the highest mean reward in MountainCarContinuous.The linear results approach the theoretical maximum; the MountainCarContinuous comparison also reports the method's highest mean reward among the three baselines.Source: §5.1 · p. 15, Table 2 · p. 15, §5.2 · p. 18

5.2. OpenAI Gym Environment

• MountainCarContinuous experiments run for 500 training epochs, evaluate every 20 epochs, and conduct 100 additional evaluation epochs after training.The reported theoretical maximum reward is approximately 100, used as the Figure 4 reference asymptote.Source: §5.2 · p. 17, Figure 4 · p. 18
• In MountainCarContinuous, the proposed method’s training rewards stabilize after initial fluctuations and rise as training progresses; its evaluation rewards outperform DQN, DDPG, and MABO.The experiment uses 20K expert demonstration entries and approximately 100K behavioral entries, including randomly sampled actions and expert demonstrations.Source: §5.2 · pp. 17–18, Figure 4 · p. 18

• The framework estimates density-ratio weights from expert demonstrations and behavioral data for weighted Bellman residual minimization in deep Q* estimation.Source: §6 · p. 18
• The analysis establishes a sharp convergence rate for density-ratio estimation and a convergence rate for the excess risk of the resulting Q* estimator.Source: §6 · pp. 18–19
• Experiments report significant improvements in numerical performance and policy generalization, with future work planned for broader expert settings and more complex environments such as robotics control.Source: §6 · p. 19

• Kang handled methodology, investigation, and conceptualization; Zhong handled methodology, visualization, software, and data curation.Source: CRediT authorship contribution statement · p. 19
• Yang provided supervision, project administration, and funding acquisition.Source: CRediT authorship contribution statement · p. 19
• Yuan contributed writing review and editing.Source: CRediT authorship contribution statement · p. 19

• The statistical-error argument uses an independent ghost sample and covering numbers for the weight and Q-function classes.Source: Appendix A · pp. 21, 22, 26
• Theorem 4.1’s excess-risk analysis decomposes error into density-ratio, statistical, and approximation components.Source: Appendix A · p. 29
• The proof controls density-ratio error using the difference between the estimated and target weights, together with an expert-policy approximation term.Source: Appendix A · pp. 29–30
• The appendix states that combining the error bounds yields a dominant term simplifying the result in Eq. (2).Source: Appendix A · pp. 30–31

• A ReLU DNN approximation lemma bounds the depth needed to approximate a function in the stated smoothness class.Source: Appendix B · p. 31
• A second lemma gives a bound for ReLU DNNs with specified width, depth, size, and bounded parameters.Source: Appendix B · p. 31
• The appendix states a concentration result for independent two-valued variables and their average.Source: Appendix B · p. 31