Source-linked AI summary
Better Exploration with Optimistic Actor-Critic
Kamil Ciosek, Quan Vuong, Robert Loftin, Katja Hofmann
TL;DR
Poor sample efficiency limits the wider adoption of actor-critic methods, partly because existing algorithms underexplore uncertain actions and waste samples directionally. OAC uses upper-confidence-bound exploration while retaining lower-bound critic and actor updates, achieving state-of-the-art sample efficiency in continuous control. The paper reports a Humanoid result of 5033 ± 147 for OAC versus 4586 ± 117 for SAC and discusses stability and scope constraints.
Problem
Poor sample efficiency limits wider adoption of actor-critic methods, while lower-bound-driven and directionally uninformed exploration impede efficient exploration.
Method
OAC uses a separate exploration policy that maximizes an approximate upper confidence bound while retaining lower-bound critic and target-policy updates, with a KL constraint for stability.
Results
OAC achieves state-of-the-art sample efficiency on the Humanoid benchmark; at training end, performance was 5033 ± 147 for OAC versus 4586 ± 117 for SAC.
Takeaways & Limitations
The results suggest optimism can improve policy-gradient sample efficiency with almost no additional computational overhead.
Takeaways & Limitations
Off-policy exploration with function approximation can bias updates through changed state occupancy, motivating a KL constraint between exploration and target policies.
Abstract
from arXiv · showhide
Actor-critic methods, a type of model-free Reinforcement Learning, have been successfully applied to challenging tasks in continuous control, often achieving state-of-the art performance. However, wide-scale adoption of these methods in real-world domains is made difficult by their poor sample efficiency. We address this problem both theoretically and empirically. On the theoretical side, we identify two phenomena preventing efficient exploration in existing state-of-the-art algorithms such as Soft Actor Critic. First, combining a greedy actor update with a pessimistic estimate of the critic leads to the avoidance of actions that the agent does not know about, a phenomenon we call pessimistic underexploration. Second, current algorithms are directionally uninformed, sampling actions with equal probability in opposite directions from the current mean. This is wasteful, since we typically need actions taken along certain directions much more than others. To address both of these phenomena, we introduce a new algorithm, Optimistic Actor Critic, which approximates a lower and upper confidence bound on the state-action value function. This allows us to apply the principle of optimism in the face of uncertainty to perform directed exploration using the upper bound while still using the lower bound to avoid overestimation. We evaluate OAC in several challenging continuous control tasks, achieving state-of the art sample efficiency.
1 Introduction
Actor-critic methods are costly to deploy because they require many environment interactions, while existing exploration can be pessimistic and directionally wasteful. OAC addresses these issues with optimism-based exploration while retaining lower-bound updates for stability and overestimation control.
- Millions of environment interactions are often needed for reasonably performant policies, making actor-critic deployment costly when samples are expensive.
- SAC and TD3 maximize an approximate lower confidence bound, which can discourage actions that might improve an inaccurate critic estimate.
- Gaussian policies sample equally in opposing directions, wasting samples in regions likely already explored by recent policies.
- OAC maximizes an upper confidence bound for exploration, using bootstrap-based epistemic uncertainty while retaining lower-bound updates to mitigate overestimation.
- OAC achieves state-of-the-art sample efficiency on the Humanoid benchmark across challenging continuous-control tasks and remains stable in ablations.
2 Preliminaries
The paper frames actor-critic learning through policies, value functions, and reparameterized gradients in a Markov decision process. Modern methods use lower-confidence critics and target networks for stability, while off-policy exploration requires a KL constraint to limit update bias.
- The reinforcement-learning problem is modeled as a Markov decision process with states, actions, rewards, transitions, initial-state distribution, and discount factor.
- A policy generates trajectories, and its action-value function measures expected discounted returns after a state-action pair.
- Actor-critic methods learn a target policy with an actor and a critic, typically adding entropy to regularize learning and encourage exploration.
- Modern critics approximate a lower confidence bound using two differently initialized networks, while slowly updated target networks improve stability.
- Off-policy exploration can bias function-approximation updates through changes in state occupancy, so the method constrains exploration and target policies with KL divergence.
3 Existing Exploration Strategy is Inefficient
The paper identifies two inefficiencies in lower-bound-driven exploration: inaccurate critics can collapse policy variance around spurious maxima, and symmetric Gaussian sampling wastes actions in already explored directions.
- Pessimistic underexploration: Greedily maximizing an inaccurate lower-bound critic can concentrate the policy near a spurious maximum and cause underexploration.
- Pessimistic underexploration: When the lower-bound maximum is at the current policy mean, finding the true critic’s different maximum requires sampling farther from that mean.
- Pessimistic underexploration: An upper bound can encourage exploration of uncertain action regions and help correct the critic, unlike a lower bound that suppresses such exploration.
- Directional uninformedness: Gaussian policies sample opposite directions from the mean equally, even though recent incremental updates make previously visited regions more likely to be explored already.
4 Better Exploration with Optimism
OAC constructs an optimistic exploration policy from an upper confidence bound while retaining lower-bound updates for stability and overestimation control. Its KL-constrained, Gaussian exploration policy targets informative and directionally asymmetric actions, and experiments compare it with established continuous-control methods.
- Optimistic exploration: OAC uses a separate Gaussian exploration policy whose covariance matches the target policy, while its mean is shifted toward actions favored by the upper-bound gradient.Actions are sampled from this exploration policy and stored in the replay buffer for subsequent critic and actor updates.
- Upper confidence bound: OAC derives an approximate upper confidence bound by estimating epistemic uncertainty, forming an upper bound, and linearizing it for tractable optimization.The uncertainty estimate uses critic bootstraps, and the optimism level is controlled by βUB.
- Optimistic exploration: The exploration policy maximizes the linearized upper bound while constraining KL divergence from the target policy to preserve stable off-policy learning.Because both policies are Gaussian, the resulting optimization has a closed-form solution.
- Exploration behavior and evaluation: Figure 2 shows OAC sampling away from the spurious lower-bound maximum and asymmetrically around the target-policy mean, addressing pessimistic underexploration and directional uninformedness.Figure 3 compares OAC with SAC, TD3, and DDPG across five MuJoCo environments using environment steps and total undiscounted return, with one-standard-deviation shading.
- Stability and overestimation: OAC retains lower-bound critic and actor updates, restricting the optimistic upper bound to exploration rather than allowing it to directly cause catastrophic overestimation.The actor can use a more conservative lower bound parameterized by βLB, with the critic recovered at βLB = −1.
- Related work: OAC differs from related uncertainty-based methods by using bootstrap uncertainty from Qπ for exploration, while remaining computationally comparable to SAC in per-iteration wall-clock time.The paper also distinguishes OAC from Bayesian actor-critic, discrete-action bootstrap, and computationally expensive model-based approaches.
5 Experiments
The experiments evaluate OAC against established continuous-control methods, test its uncertainty estimate and stability, and examine sensitivity to exploration constraints. OAC achieves higher Humanoid performance than SAC, while remaining similarly stable and effective across a broad range of δ settings.
- Ablations and stability: The experiments test whether bootstrapped uncertainty improves OAC performance and whether optimistic exploration affects learning stability.They also assess competitiveness with state-of-the-art algorithms.
- Benchmark evaluation: OAC is evaluated on MuJoCo continuous-control benchmarks against SAC, TD3, and tuned DDPG.The experiments use three exploration-related hyperparameters: βUB, βLB, and δ.
- Benchmark evaluation: OAC outperforms SAC on Humanoid, with final performance of 5033 ± 147 versus 4586 ± 117 using 90% confidence intervals.Hyperparameters were not tuned on Humanoid.
- Ablations and stability: A broad range of δ settings produces good OAC performance after 1 million environment steps on Ant-v2.The tested quantity is 2δ in the exploration policy.
- Ablations and stability: OAC and SAC are similarly stable across individual learning runs despite OAC using a shifted exploration policy.Figure 5 reports 90% confidence intervals.
6 Conclusions
The paper concludes that OAC improves exploration efficiency in continuous-control policy-gradient methods by addressing pessimistic underexploration and directional uninformedness.
- Conclusion: OAC uses an approximate confidence bound on the state-action value function for exploration and achieves state-of-the-art sample efficiency in continuous-control tasks.The authors report almost no additional computational overhead.
A Proof of Proposition 1
The appendix derives the Gaussian exploration policy for OAC by imposing a divergence constraint and maximizing an approximate upper confidence bound, then relates the result to deterministic exploration and critic bounds.
- Gaussian exploration policy: Proposition 1 states that the exploration policy has Gaussian form πE = N(µE, ΣE).The derivation begins from the KL distance between Gaussian distributions.
- Gaussian exploration policy: The covariance optimization reduces to setting the exploration covariance equal to the target covariance, ΣE = ΣT.The reduction follows from non-negativity of the KL divergence.
- Gaussian exploration policy: With equal covariances, the remaining problem maximizes the approximate upper confidence bound subject to a constraint on the policy mean.The derivation introduces a Lagrangian and differentiates it to obtain the solution.
- Deterministic exploration: The deterministic OAC variant uses a Dirac-delta exploration policy and a Wasserstein divergence because KL becomes singular for deterministic policies.Its exploration policy has the form πE = δ(µE).
- Confidence-bound interpretation: The appendix interprets TD3 and SAC critic minimization as an approximate lower confidence bound and shows that βUB = 1 corresponds to taking the maximum of two critics.These relations connect the critic constructions to confidence-bound interpretations.
D Experimental setup and hyper-parameters
The evaluation uses controlled, reproducible implementations and separately specified hyperparameters for OAC and baseline algorithms. Supplementary figures examine critic bounds, policy-gradient behavior, underexploration, and OAC ablations.
- Evaluation protocol: Learning curves report total undiscounted return smoothed over the preceding 100 data points.
- Evaluation protocol: Experiments run in Docker containers on CPU-only Azure Standard_D8s_v3 machines using an implementation based on official SAC softlearning code.
- Evaluation protocol: Environment seeds are fixed for reproducibility, with disjoint training and evaluation seed sets.
- Hyperparameters: OAC hyperparameters were tuned by Bayesian optimization on four MuJoCo environments, using average performance over the first 250,000 steps; Humanoid-v2 was not tuned.
- Baselines: Baseline results use official or previously defined implementations of SAC, TD3, and DDPG, with HalfCheetah-v2 showing high SAC performance variance.
- Supplementary analyses: Additional analyses visualize critic lower bounds, policy variance, upper-bound ablations, and the relationship between lower or upper bounds and exploration.
J Plot of sample efficiency
The appendix explicitly plots sample efficiency by measuring how many environment steps are required to reach a specified performance level.
- Figure 11 reports the number of steps required to achieve a given level of performance.
K Additional ablations
The additional ablations examine exploration-policy displacement and the effect of varying the upper-bound coefficient.
- Figure 12a measures the norm of the difference between target-policy and exploration-policy means, averaged over one minibatch.
- Figure 12b compares learning curves across four βUB values, including the final sweet-spot value of 4.36.