Addressing Function Approximation Error in Actor-Critic Methods

1. Introduction

• Noisy value estimates and temporal-difference updates accumulate error, allowing poor states to appear valuable and causing suboptimal policy updates or divergent behavior.Function approximation makes estimator noise unavoidable, while each update uses an estimate of a subsequent state.Source: §1 · p. 1
• Because Double DQN (Van Hasselt et al., 2016) remains too similar to the current policy in actor-critic training, TD3 uses two independently trained critics and their minimum to favor underestimation.The paper establishes overestimation in deterministic policy gradients and notes that even unbiased high-variance estimates can cause local overestimation.Source: §1 · p. 1
• TD3 reduces variance with target networks, delayed policy updates until value estimates converge, and SARSA-style regularization that bootstraps similar action estimates.The paper presents target networks as reducing accumulated error and policy delays as addressing coupling between value and policy updates.Source: §1 · p. 1
• TD3, formed by modifying DDPG (Lillicrap et al., 2015), outperforms the state of the art by a wide margin across seven OpenAI Gym continuous-control domains.The authors report using many random seeds, fair evaluation metrics, ablations, and releasing code and learning curves.Source: §1 · pp. 1–2

2. Related Work

• Function-approximation error has been studied for its effects on reinforcement-learning bias and variance, while this work focuses on overestimation bias and high variance build-up.Source: §2 · p. 2
• Prior Q-learning remedies include Double Q-learning’s independent estimators for unbiased values, variance reduction, protection from early high-variance estimates, and corrective terms.These approaches are associated with Van Hasselt (2010) and Van Hasselt et al. (2016), Anschel et al. (2017), Fox et al. (2016), and Lee et al. (2013), respectively.Source: §2 · p. 2
• Value-estimate variance has also informed risk-aversion and exploration, but these lines did not connect it to overestimation bias.The cited work includes Mannor & Tsitsiklis (2011) on risk-aversion and O’Donoghue et al. (2017) on exploration.Source: §2 · p. 2
• Temporal-difference error accumulation has been addressed by minimizing per-step errors or mixing off-policy and Monte-Carlo returns; multi-step methods trade accumulated bias against policy- and environment-induced variance.Multi-step approaches include importance sampling (Munos et al., 2016), distributed methods (Mnih et al., 2016; Espeholt et al., 2018), and approximate bounds (He et al., 2016); reducing the discount factor also reduces each error’s contribution.Source: §2 · p. 2
• This work highlights target networks for reducing per-update error and regularizes variance by averaging value estimates; smoothed value functions also reduced variance and improved performance in concurrent work.The concurrent result is attributed to Nachum et al. (2018).Source: §2 · p. 2
• The method builds on DPG and DDPG; orthogonal DDPG improvements include distributed methods, multi-step returns, prioritized replay, and distributional methods.DPG and DDPG are attributed to Silver et al. (2014) and Lillicrap et al. (2015); the listed improvements include work by Popov et al. (2017), Schaul et al. (2016), Horgan et al. (2018), Bellemare et al. (2017), and Barth-Maron et al. (2018).Source: §2 · p. 2

• An agent selects actions in each state, receives rewards and new states, and learns from the discounted sum of rewards.The discount factor γ determines the priority of short-term rewards.Source: §3 · p. 2
• The objective is to optimize a parameterized policy for expected return, with continuous-control actors updated using the deterministic policy gradient.The passage attributes this algorithm to Silver et al. (2014).Source: §3 · p. 2
• The critic, or value function, Qπ(s, a), estimates expected return after taking action a in state s and then following policy π.Source: §3 · p. 2
• Temporal-difference Q-learning uses the Bellman equation to relate a state-action value to its reward and the expected value of the subsequent state-action pair.The update is Qπ(s, a) = r + γE[Qπ(s′, a′)], with a′ sampled from π(s′).Source: §3 · p. 2
• For large state spaces, differentiable function approximators estimate values, while target networks provide a frozen reference updated periodically or by a proportion τ each step.Updates can use random mini-batches sampled from an experience replay buffer.Source: §3 · pp. 2–3

4.1. Overestimation Bias in Actor-Critic

• DDPG exhibits clear value overestimation against estimated true value, while Clipped Double Q-learning greatly reduces critic overestimation (Lillicrap et al., 2015).The comparison measures value estimates over time on MuJoCo environments.Source: Figure 1 · p. 3, §4.1 · pp. 3–4
• If the critic is at least accurate in expectation at πtrue and actor step α is below min(ϵ1, ϵ2), the expected actor update overestimates true value.The analysis assumes a deterministic policy gradient and defines the comparison against a hypothetical update using the true value function.Source: §4.1 · p. 3
• Overestimation can accumulate across updates and create a feedback loop in which a suboptimal critic reinforces highly rated suboptimal actions.The resulting inaccurate value estimates can also lead directly to poor policy updates.Source: §4.1 · p. 3

4.2. Clipped Double Q-Learning for Actor-Critic

• Actor-critic Double DQN suffers overestimation similar to DDPG, while Double Q-learning reduces but does not eliminate it.The authors report that this reduction is experimentally insufficient; the clipped method can also favor states with lower-variance value estimates, producing safer policy updates and more stable targets.Source: §4.2 · p. 4
• Clipped Double Q-learning uses the minimum of two critic estimates as the target, limiting added overestimation; any underestimation is less likely to be propagated through policy updates.The method can replace the critic in any actor-critic method; its critics are not fully independent because they share replay data and use opposite critics in their targets.Source: §4.2 · p. 4

5.1. Accumulating Error

• Function approximation leaves residual TD-error because the Bellman equation is never exactly satisfied, allowing errors to accumulate into overestimation and suboptimal policy updates.With a large discount factor, variance can grow rapidly, and mini-batch error reduction does not guarantee small errors outside the batch.Source: §5.1 · p. 5

5.2. Target Networks and Delayed Policy Updates

• With learned policies, fast-updating target networks diverge despite similar convergence under fixed policies, motivating policy updates only after d critic updates.Less frequent policy updates use lower-variance value estimates and improve performance while requiring fewer policy updates.Source: Figure 3 · p. 5, §5.2 · pp. 5–6

5.3. Target Policy Smoothing Regularization

• Target policy smoothing adds clipped noise to the target action and averages over mini-batches, training values for a noisy policy independently of exploration noise.This approach is reminiscent of Expected SARSA and enforces similar actions having similar values.Source: §5.3 · p. 6

6. Experiments

• TD3 maintains a pair of critics trained toward the minimum target value and updates its single actor with respect to Qθ1 every d iterations.The algorithm builds on DDPG and applies modifications intended to increase stability and performance under function approximation error.Source: §6 · p. 6

6.1. Evaluation

• TD3 matched or outperformed all other algorithms in final performance and learning speed across all tasks.Results were reported in Table 1 and learning curves in Figure 5.Source: Table 1 · p. 7, Figure 5 · p. 7, §6.1 · p. 8
• Evaluation used the original MuJoCo continuous-control tasks through OpenAI Gym, with no environment or reward modifications, for 1 million time steps per task.Evaluations every 5,000 steps averaged reward over 10 episodes without exploration noise; results covered 10 random seeds.Source: §6.1 · pp. 7–8

6.2. Ablation Studies

• The full algorithm outperformed every other tested combination in most tasks; delayed policy updates generally improved performance despite halving actor iterations and reduced training time.The ablations removed Clipped Double Q-learning, delayed policy updates, or target policy smoothing, and also tested architecture, hyper-parameter, and exploration modifications.Source: §6.2 · p. 8, Table 2 · p. 8
• DQ-AC and DDQN-AC yielded insignificant improvements over TD3-CDQ except in Ant-v1, while the full method outperformed both, suggesting value in reducing overestimation.Both variants also used delayed policy updates, target policy smoothing, and the study’s architecture and hyper-parameters.Source: §6.2 · p. 8

7. Conclusion

• Overestimation bias is problematic in actor-critic methods, common discrete-action deep Q-learning solutions are ineffective there, and a Double Q-learning variant limits possible overestimation.The paper reports that mitigating overestimation can greatly improve modern algorithms’ performance.Source: §7 · p. 8
• The paper examines how target networks limit errors from imprecise function approximation and stochastic optimization, and introduces SARSA-style regularization that bootstraps from similar state-action pairs.The work links noise and overestimation to the accumulation of temporal-difference learning errors.Source: §7 · p. 8
• TD3 improves DDPG’s learning speed and performance on challenging continuous-control tasks and exceeds numerous state-of-the-art algorithms.Its modifications are simple to implement and can be added to other actor-critic algorithms.Source: §7 · p. 8

A. Proof of Convergence of Clipped Double Q-Learning

• In the function-approximation setting, both value estimates may be updated at each iteration to address approximation error; the proof also extends to random updates.The passage contrasts this with finite-MDP Double Q-learning, where either estimate is updated randomly.Source: A. Proof of Convergence of Clipped… · p. 11
• The theorem also requires learning rates with divergent cumulative updates and finite reward variance.The learning rates are stated to lie in [0, 1]; the cited passage’s remaining condition is incomplete.Source: A. Proof of Convergence of Clipped… · pp. 11–12
• Clipped Double Q-learning converges with probability 1 to the optimal value function in a finite MDP under the theorem’s stated sampling and tabular-value assumptions.The assumptions include infinite sampling of each state-action pair, infinitely many updates to both critics, and γ ∈ [0, 1).Source: A. Proof of Convergence of Clipped… · pp. 11–12

B. Overestimation Bias in Deterministic Policy Gradients

• Under the stated expectation condition, deterministic policy gradients still produce overestimation when gradients are unnormalized.The argument concludes that the approximate policy’s estimated value is at least the true value under the true policy, which is at least the true value under the approximate policy.Source: B. Overestimation Bias in… · p. 12

C. DDPG Network and Hyper-parameter Comparison

• Table 3 presents a complete comparison of the paper’s DDPG hyperparameter choices with the OpenAI baselines implementation (Dhariwal et al., 2017).The supplied passage identifies the comparison’s scope but does not provide the table’s cell values.Source: Table 3 · p. 13
• The listed network configurations use first-layer dimensions of state dim by 400 or state dim plus action dim by 400, followed by ReLU and an action dim plus 400 by 300 layer.The passages do not identify which configuration corresponds to which network component.Source: C. DDPG Network and Hyper-parameter… · p. 13

D. Additional Implementation Details

• For failure-state terminations before the maximum horizon, DDPG and TD3 set the target’s next-state value to zero; horizon endings retain the bootstrapped value.The target is r for terminal s′ when t < max horizon, and otherwise r + γQθ′(s′, πφ′(s′)).Source: D. Additional Implementation Details · p. 14
• Target-policy smoothing noise is clipped to the range of possible actions to avoid errors from evaluating impossible actions.Source: D. Additional Implementation Details · p. 14

E. Soft Actor-Critic Implementation Details

• The Soft Actor-Critic implementation uses the authors’ code and paper hyperparameters, with a Gaussian-mixture policy.The policy uses four Gaussian distributions except on Reacher-v1, where numerical instability in the provided implementation led to using one.Source: E. Soft Actor-Critic Implementation… · p. 14
• Reward scaling multiplies rewards by 3 for Walker2d-v1 and Ant-v1, and by 1 for all other environments.Source: E. Soft Actor-Critic Implementation… · p. 14
• For fair comparison, Soft Actor-Critic was trained for one iteration per time step rather than the authors’ four, with fewer total time steps also used.The authors suggest these differences explain discrepancies on some environments.Source: E. Soft Actor-Critic Implementation… · p. 14
• The comparison is against a prior Soft Actor-Critic version; its newer variant adds Clipped Double Q-learning to the value update and is competitive with TD3 on most tasks.Source: E. Soft Actor-Critic Implementation… · p. 14

F. Additional Learning Curves

• Figure 6 compares TD3 ablations that subtract delayed policy updates, target policy smoothing, or Clipped Double Q-learning from DDPG (AHE).Source: Figure 6 · p. 15, F. Additional Learning Curves · p. 15
• Figure 7 compares AHE with additions of delayed policy updates, target policy smoothing, or Clipped Double Q-learning.Source: Figure 7 · p. 15, F. Additional Learning Curves · p. 15
• Figure 8 compares TD3 with DQ-AC and DDQN-AC, variants that also use delayed policy updates and target policy smoothing.Source: Figure 8 · p. 15