The Mirage of Action-Dependent Baselines in Reinforcement Learning

1. Introduction

• On-policy Monte-Carlo policy-gradient estimates have high variance, requiring large batches for stable optimization and limiting sample efficiency.Actor-critic and λ-weighted return methods reduce variance using function approximators, but introduce bias that can cause instability and hyperparameter sensitivity.Source: §1 · p. 1
• Recent proposals extend state-dependent baselines to depend on both state and action, with reported variance reductions and sample-efficiency improvements.The proposals are attractive because baselines are intended to reduce variance without biasing policy-gradient estimates.Source: §1 · p. 1
• On synthetic LQG and benchmark continuous-control tasks, learned state-action-dependent baselines do not significantly reduce variance over learned state-dependent baselines.The variance from value-function or state-dependent-baseline approximation is much larger than the reduction from adding action dependence.Source: §1 · p. 2
• The authors report that prior code for Q-prop, Stein control variates, and LAX diverges from the papers’ unbiased methods through subtle implementation decisions that trade variance for bias.They empirically evaluate variants to show these decisions are crucial for the previously reported gains.Source: §1 · p. 2
• The proposed horizon-aware value-function parameterization improves performance over the state-action-dependent baseline without biasing the underlying method.It addresses the poor value-function approximation identified as a larger source of variance.Source: §1 · p. 2

2. Background

• Policy gradients differentiate expected return with respect to policy parameters; the value function averages state-action values, and the advantage is their difference.The paper notes that its practical use of undiscounted state-visitation frequencies (γ = 1) yields a biased estimator for ∇J(θ) and more closely aligns with average reward.Source: §2 · p. 2

2.1. Advantage Function Estimation

• The k-step advantage estimator uses a value-function estimate; the infinite-horizon form gives an unbiased gradient estimator for any estimate, while finite-k forms are biased unless the value estimate is exact.GAE combines such estimators, so its hyperparameters balance bias and variance when the value function is imperfect.Source: §2.1 · pp. 2–3

2.2. Baselines for Variance Reduction

• A state-dependent baseline is a control variate of the form φ(s)∇log π(a|s), which reduces variance without bias because the expected policy score is zero.State-action-dependent baselines use φ(s,a), but their additional expectation term must be evaluated analytically or estimated with low variance.Source: §2.2 · p. 3
• For continuous actions, prior methods either use a linear Taylor expansion of a learned advantage function or estimate the baseline expectation with the reparameterization trick when the policy permits it.For small discrete action sets, the expectation can instead be computed analytically.Source: §2.2 · p. 3

3. Policy Gradient Variance Decomposition

• Only the action-sampling variance term depends on the baseline, and its variance-minimizing choice is the estimated advantage E[return | s, a].The decomposition interprets trajectory, action, and state terms as variance from sampling one rollout, one action per state, and a limited set of states, respectively.Source: §3 · p. 3
• An optimal action-dependent baseline is most useful when the state-baseline action variance is large relative to trajectory and state variance; learning φ(s, a) further limits its potential benefit.The decomposition says an optimal baseline removes the action term, while the practical learned baseline must also approximate φ(s, a).Source: §3 · p. 4

3.1. Variance in LQG Systems

• In the LQG point-mass task, the state-value baseline sharply reduces action variance, but total variance remains largely unchanged because trajectory variance is already comparatively large.The analysis uses LQG because its variance terms are nearly analytically measurable.Source: §3.1 · p. 4

3.2. Empirical Variance Measurements

• For benchmark continuous-control measurements, the authors use repeated samples and unbiased estimators of the variance terms, tracking them during TRPO training on Humanoid and HalfCheetah.They also repeat the measurements with horizon-aware value functions.Source: §3.2 · pp. 4–5
• With discounted returns, even an optimal action-dependent baseline does not improve over a state-dependent baseline; with GAE, learned baselines of both types have similar, higher variance than oracle baselines.For GAE, the optimal action-dependent baseline could reduce overall variance relative to a state-dependent baseline, but the learned function approximators produce similar variance and the authors report no expected learning advantage on these tasks.Source: §3.2 · p. 5
• Closing the value-function and state-baseline approximation gap would reduce variance more than using an optimal action-dependent baseline, while the state-sampling term is relatively small.The authors therefore identify improving V(s) and φ(s) approximation, and focusing on trajectory and action variance, as more fruitful targets.Source: §3.2 · p. 5

4. Unveiling the Mirage

• Prior gains from state-action-dependent baselines arise from bias–variance trade-offs in implementations, not unbiased variance reduction.The authors say the implementation decisions explain empirical success; the continuous-control conclusion about Grathwohl et al. is limited to those experiments.Source: §4 · pp. 5–6

4.1. Advantage Normalization

• Asymmetric advantage normalization leaves the bias-correction term unnormalized, introducing bias; unbiased Q-Prop does not outperform TRPO.The implementation constrains learning-signal variance to 1 by adding bias when necessary, and the authors find this trade-off crucial to Q-Prop’s observed gains.Source: §4.1 · p. 6

4.2. Poorly Fit Value Functions

• A poorly fit value function can make a state-action-dependent baseline appear beneficial, although a state-dependent baseline achieves comparable variance reduction.This can occur when the value approximator is insufficiently trained; the cited Liu et al. result is attributed to a poorly fit value function.Source: §4.2 · pp. 6–7

4.3. Sample-Reuse in Baseline Fitting

• Fitting a baseline on the current batch before computing the policy gradient introduces bias, and flexible approximators can overfit that batch.The bias depends on implicit or explicit regularization; in the tested variants, unbiased state-dependent and state-action-dependent baselines matched, while biased results varied by task.Source: §4.3 · p. 7

5. Horizon-Aware Value Functions

• For fixed-horizon tasks, a stationary value function ignores that expected return necessarily falls near episode end.The standard continuous-control benchmarks use a fixed horizon, while most value-function parameterizations are stationary.Source: §5 · p. 7
• The proposal combines a predicted average reward over future states with a state-dependent offset, multiplying the rate prediction by discounted time remaining.The parameterization outputs two values, ˆr(s) and ˆV′(s), and combines them with discounted time left.Source: §5 · p. 7
• The horizon-aware parameterization performs favorably compared with including time as a value-function input.The comparison is reported in Appendix Figures 6 and 7.Source: §5 · p. 7
• Across environments, the horizon-aware value function outperforms TRPO, TRPO with state-dependent baselines, and TRPO with state-action-dependent baselines.The authors state that this practical improvement does not introduce bias.Source: §5 · p. 7

6. Related Work

• State-action-dependent baselines extend doubly robust estimation to policy gradients and, for discrete actions, yield the gradient of a doubly robust estimator.The section also extends analysis of the optimal state-dependent baseline to state-action-dependent baselines and analyzes GAE variance.Source: §6 · pp. 7–8
• Some policy-gradient methods mix high-variance unbiased on-policy estimates with lower-variance biased off-policy or model-based estimates to improve performance.The authors note related work on minimizing mean squared error for mixtures of doubly robust and model-based estimates in off-policy evaluation.Source: §6 · p. 8

7. Discussion

• In common continuous-control benchmarks, learned state-action-dependent baselines cannot achieve significant variance reduction, while improving value-function or state-dependent baseline accuracy could yield larger gains.The discussion also reports that a horizon-aware value function improves performance compared with the state-action-dependent baseline without biasing the method.Source: §7 · p. 8
• Prior empirical gains are attributed to implementation decisions that introduce bias in exchange for variance reduction.The authors identify pitfalls in earlier state-action-dependent baseline work and plan to study the bias-variance trade-off further.Source: §7 · p. 8
• The horizon-aware value-function parameterization consistently improves over TRPO, whereas the unbiased state-action-dependent baseline does not outperform it.The modification makes the parameterization aware of the finite time horizon.Source: §7 · p. 8
• The relative contributions to policy-gradient variance are problem-specific; in the studied continuous-control benchmarks, the condition for a learned action-dependent baseline to help was not met.The discussion gives discrete tasks with critical decisions as a possible setting where action-related variance could be larger.Source: §7 · p. 8

8.1. Q-Prop Experiments

• The Q-Prop experiments use the authors’ implementation and default settings on CartPole-v0, HalfCheetah-v1, and Humanoid-v1; the unbiased variant divides the bias-correction term by ˆσ.The reported batch size is 5000 steps, and the conservative Q-Prop variant is used.Source: §8.1 · p. 11

8.2. Stein Control Variate Experiments

• The Stein experiments compare five algorithms on HalfCheetah-v1 and Humanoid-v1, including biased and unbiased state-dependent and state-action-dependent baselines.One biased variant estimates the correction gradient with an extra policy sample rather than importance weighting; unbiased-baseline updates follow the policy update for the batch.Source: §8.2 · p. 11

8.3. Backpropagating through the Void

• The LAX experiments use the authors’ implementation with a change to measure policy-gradient estimator variance and a correction for an advantage-estimator bug.The original code measured the variance of a gradient estimator used by neither method.Source: §8.3 · p. 11

8.4. Horizon-Aware Value Function Experiments

• The horizon-aware experiments test four algorithms on three continuous-control tasks, using a two-head value network trained on Monte Carlo returns.The policy and value networks are two-layer MLPs with 64 units per hidden layer; separately trained baselines use Adam to fit GAE estimates, and the action-dependent baseline uses reparameterization.Source: §8.4 · p. 11

9. Variance computations in Linear-Quadratic-Gaussian (LQG) systems

• For finite-horizon open-loop LQG, the appendix derives state and state-action marginals and integrates quadratic costs to obtain Q, V, and advantage functions.The finite horizon makes Q non-stationary.Source: §9 · p. 11, §9.1 · pp. 12–13

9.2. Computing the LQG analytic gradients

• The LQG gradients with respect to Gaussian policy parameters are tractable analytically, and the experiments analyze gradients with respect to the mean parameters.The derivation omits the time index because it applies at every time step.Source: §9.2 · p. 13

9.3. Estimating the LQG variances

• The variance procedure computes Σs and uses a sample-based estimate for Σa to avoid complex fourth-order integration; the Q- and A-based expressions distinguish state- and action-dependent baseline reductions.The passage identifies V(s) as the state-dependent baseline and Q(s,a) as the optimal state-action-dependent baseline.Source: §9.3 · p. 13
• For Στ, the authors avoid computing the full Gaussian because it is computationally expensive, using a sufficiently low-variance alternative with a sample estimate over s′, a′, and τ′.The samples are drawn from the state, policy, and conditional trajectory distributions, respectively.Source: §9.3 · p. 13

9.4. LQG Experiments

• The LQG evaluation uses a 2D point-mass system with a 100-step horizon and 20,000 samples for Monte Carlo variance estimates.The stated setup includes dt = 0.05, m = 1.0, q = 1.0, and r = 0.01.Source: §9.4 · p. 14
• The experiment optimizes the policy mean parameters with analytic gradients and reports monotonically improving cost.Figure 5 adds visualizations of policies at different learning stages.Source: §9.4 · p. 14

• Each variance term is estimated with an unbiased single-sample estimator that can be repeated to reduce measurement variance.The procedure first draws a policy batch, then selects a state and samples an action from the policy.Source: §10 · p. 14
• For the first term, the procedure draws two trajectories conditional on the same state and action.Source: §10 · p. 14
• The second term is estimated for both zero baseline and state-dependent advantage baseline cases, with the latter using two policy actions and their respective trajectories.The passage identifies the state-dependent control variate as Vπ(s).Source: §10 · p. 14
• Because the unbiased estimate of E[ĝ]² is small relative to the other terms, the authors use an upper bound for the third term.Source: §10 · p. 14

• A convex combination controlled by λ trades variance against bias: λ = 1 is unbiased, λ = 0 is lower-variance but biased, and dynamically managing this trade-off is future work.The variance contribution is reduced by λ^2.Source: §11 · p. 14
• The finite-horizon T = 100 LQG point-mass evaluation visualizes variance terms alongside on-policy trajectories, in a challenging open-loop setting with unoptimized policy variances.Source: Figure 5 · p. 15
• Figures 6 and 7 compare standard, horizon-aware, and time-input value-function parameterizations using episode reward and estimated gradient variance.For the variance comparison, estimates follow the standard-parameterization trajectory and are averaged across batch timesteps.Source: Figure 6 · p. 15, Figure 7 · p. 16
• Figure 8 evaluates biased Stein control-variate variants, unbiased state-dependent-baseline variants, and PPO; Figure 13 compares LAX with a state-dependent baseline on InvertedPendulum-v1.Source: Figure 8 · p. 16, Figure 13 · p. 19
• Figures 9–12 evaluate gradient-estimator variance terms across HalfCheetah, Humanoid, discounted returns, GAE, and different baselines; learned labels indicate function-approximated φ.Source: Figure 9 · p. 17, Figure 10 · p. 17, §11 · p. 17
• State-dependent and state-action-dependent baselines reduce variance similarly, while the larger gap from Value reflects poor value-function fitting.Source: Figure 12 · p. 18