Source-linked AI summary
Diversity-Driven Exploration Strategy for Deep Reinforcement Learning
Zhang-Wei Hong, Tzu-Yun Shann, Shih-Yang Su, Yi-Hsiang Chang, Chun-Yi Lee
TL;DR
Reinforcement-learning agents struggle to explore effectively in large state spaces and environments with deceptive or sparse rewards. The paper adds a policy-distance term to existing losses, encouraging behavior different from recent policies while retaining reward optimization, and reports superior or comparable benchmark performance with adaptive scaling.
Problem
Efficient exploration remains challenging in large state spaces and environments with deceptive local optima or sparse rewards.
Method
The method adds a distance measure between the current policy and recent prior policies to arbitrary DRL loss functions, with adaptive scaling for the distance term.
Results
The method achieves superior or comparable performance to baselines across benchmark settings in mean scores and learning time, while handling deceptive and sparse rewards.
Takeaways & Limitations
Diversity-driven exploration can be combined with current off- and on-policy RL algorithms to encourage efficient exploration of large state spaces.
Abstract
from arXiv · showhide
Efficient exploration remains a challenging research problem in reinforcement learning, especially when an environment contains large state spaces, deceptive local optima, or sparse rewards. To tackle this problem, we present a diversity-driven approach for exploration, which can be easily combined with both off- and on-policy reinforcement learning algorithms. We show that by simply adding a distance measure to the loss function, the proposed methodology significantly enhances an agent's exploratory behaviors, and thus preventing the policy from being trapped in local optima. We further propose an adaptive scaling method for stabilizing the learning process. Our experimental results in Atari 2600 show that our method outperforms baseline approaches in several tasks in terms of mean scores and exploration efficiency.
1 Introduction
Exploration is difficult in large, deceptive, or sparse-reward environments, where common heuristics and random perturbations may be ineffective. The paper introduces a diversity-driven strategy that modifies DRL losses to encourage novel policies and evaluates it across benchmark settings.
- Motivation: Exploration is especially challenging with large state spaces, deceptive rewards, and sparse rewards because agents may become trapped in local optima or receive few training signals.The introduction illustrates this problem with sub-optimal behavior in HalfCheetah and limitations of heuristic exploration methods.
- Contribution: The paper proposes a diversity-driven exploration strategy that encourages agents to attempt policies different from prior policies.It is designed to apply to most off- and on-policy DRL algorithms.
- Contribution: A distance measure between the current policy and recent prior policies is added to the loss function to promote exploration.The method combines this distance measure with reward optimization rather than ignoring rewards.
- Evaluation: The method is reported to produce better exploratory behavior in deceptive and sparse-reward gridworlds, where baselines can become trapped or fail to find feasible policies.The evaluation also includes Atari 2600 and MuJoCo benchmark environments.
- Evaluation: The paper proposes adaptive scaling methods for the distance measure and reports superior or comparable performance to baselines in mean scores and learning time.It also presents ablations of the adaptive scaling strategy and comparisons across three environments.
2 Background
The background introduces reinforcement learning as interaction between an agent and environment to maximize discounted rewards. It then distinguishes off-policy methods such as DQN and DDPG from on-policy A2C and describes their exploration mechanisms.
- 2.1 Reinforcement Learning: Reinforcement learning trains an agent to interact with an environment by observing states, selecting actions through a policy, and receiving rewards.The objective is to maximize discounted accumulated rewards over the task horizon.
- 2.2 Off-Policy Methods: Off-policy methods separate behavior and target policies, allowing learning from arbitrary-policy samples or experience replay.DQN and DDPG are presented as representative off-policy algorithms.
- 2.2 Off-Policy Methods: DQN approximates the optimal Q-function with a neural network trained on experience-replay samples using a loss based on target-network values.Its exploration mechanism is epsilon-greedy, and the target network is updated periodically.
- 2.2 Off-Policy Methods: DDPG is an actor-critic method for continuous actions whose actor maximizes critic-estimated Q-values and whose exploration adds noise to the deterministic policy.The noise may be normally distributed or generated by an Ornstein-Uhlenbeck process.
- 2.3 On-Policy Methods: On-policy methods update value functions from samples generated by the current policy, and A2C trains parallel agents while using entropy to encourage exploration.Entropy also helps prevent premature convergence to sub-optimal policies.
3 Diversity-Driven Exploration Strategy
Diversity-driven exploration modifies an existing DRL loss with a distance term that encourages policies to diverge from recent policies, while adaptive scaling stabilizes learning.
- 3 Diversity-Driven Exploration Strategy: The loss adds a distance-based term between the current policy and recent policies, encouraging exploration of different behaviors and novel states.The distance can use KL-divergence, L2-norm, or MSE, and the method can support greedy or stochastic policies.
- 3.1 Implementation on Off-Policy Methods: The approach can be incorporated into off-policy methods by storing prior values or actions in the replay buffer and measuring policy or action differences.Div-DQN uses softmax policy distributions and KL-divergence, while Div-DDPG compares current actions with prior actions using MSE.
- 3.2 Implementation on On-Policy Methods: On-policy integration maintains the most recent policies to compute the distance term, with n = 5 generally sufficient for satisfactory performance.Div-A2C uses KL-divergence between the current and prior policies.
- 3.3 Adaptive Scaling Strategy: Distance-based scaling adaptively changes α according to whether the distance measure is below or above a threshold δ.Different δ values are used for different methods in the experiments.
- 3.3 Adaptive Scaling Strategy: Performance-based scaling addresses high variance from using only recent policies by adjusting α according to past-policy performance for on-policy algorithms.The proactive strategy favors high-performing prior policies and avoids poor ones, whereas the reactive strategy only avoids underperforming policies.
- 3.3 Adaptive Scaling Strategy: Clipping the distance measure D between −c and c is used to reduce training instability when D becomes extraordinarily high.The constant c is predefined.
4 Experiments
The experiments evaluate the proposed methodology across three environment types and include a supplementary ablation analysis.
- 4 Experiments: The study reports results in three different environments after describing the experimental setup and evaluation environments.An ablative analysis is provided in the supplementary material.
4.1 Experimental Setup
The evaluation uses gridworlds, Atari 2600, and MuJoCo to test exploration across sparse or deceptive rewards, discrete control, and continuous robotic control.
- 4.1 Experimental Setup: Gridworld experiments use 2D maps with sparse- and deceptive-reward settings, where agents start at the top-left and seek a reward of 1 at the bottom-right.The agent observes absolute coordinates and chooses among four movement actions.
- 4.1 Experimental Setup: Atari 2600 experiments select eight games with varying exploration difficulty and provide agents with stacked 84 × 84 × 4 grayscale images.The tasks use the Arcade Learning Environment for discrete control.
- 4.1 Experimental Setup: MuJoCo experiments evaluate robotic continuous-control tasks in which agents map physical-state vectors to action-value vectors.The proposed and baseline methods are compared across selected robotic control environments.
- 4.1 Experimental Setup: Baseline methods include vanilla and noisy or curiosity-driven DQN/A2C for discrete control, plus vanilla and parameter-noise DDPG for continuous control.The baseline set varies by environment type.
4.2 Exploration in Huge Gridworld
In large gridworlds, Div-DQN outperforms baseline methods under deceptive and sparse rewards while exploring more broadly and systematically.
- Deceptive reward: Div-DQN outperforms vanilla and Noisy-DQN in deceptive-reward gridworlds, escaping deceptive areas and discovering the optimal reward.Baseline methods remain trapped near deceptive rewards and never visit the optimal bottom-right location, whereas Div-DQN explores all four sides.
- Sparse reward: In sparse-reward gridworlds, Div-DQN is the only method that explores uniformly and systematically rather than repeatedly visiting previously explored states.DQN covers only a small state-space region, while Noisy-DQN explores more broadly but wastes substantial time revisiting states.
- Overall finding: The results indicate that diversity-driven exploration offers advantages for large gridworlds with deceptive or sparse rewards.The conclusion covers both reward settings evaluated in the gridworld experiments.
4.3 Performance Comparison in Atari 2600
The Atari experiments compare diversity-driven DQN and A2C variants with baseline methods using in-training learning curves. Across the evaluated games, the proposed methods are superior or comparable, with faster learning in easy-exploration tasks and improved scores and efficiency in most hard-exploration games except Montezuma’s Revenge.
- Overall comparison: The proposed methods achieve superior or comparable performance to baseline methods across all evaluated Atari games.Figure 3 covers DQN variants and Figure 4 covers A2C variants; learning curves report in-training median scores with interquartile ranges.
- Hard Exploration Games: In Freeway, diversity-driven agents quickly discover the only reward across the road while other methods remain at the starting position.The associated curves show Div-DQN and Div-A2C learning considerably faster and better than baseline methods.
- Easy Exploration Games: Diversity-driven DQN and A2C learn significantly faster than baselines in Enduro and BankHeist and show superior performance for most of training.These results concern the easy-exploration games shown in Figures 3 and 4.
4.4 Performance Comparison in MuJoCo Environments
Across continuous-control settings with deceptive rewards, large state spaces, and sparse rewards, Div-DDPG learns faster and achieves stronger outcomes than baseline methods. The approach also escapes suboptimal policies and remains effective when rewards are difficult to obtain.
- Environments with Deceptive Rewards: Div-DDPG learns faster and achieves higher average rewards than baselines in deceptive-reward environments, escaping local optima to find better strategies.In Humanoid it later walks forward longer without falling, while in HalfCheetah it learns to balance and move forward swiftly.
- Environments with Large State Spaces: Div-DDPG learns significantly faster than baseline methods in Pusher, Thrower, and Strike, despite the challenge of learning feasible robotic-arm policies.
- Environments with Sparse Rewards: In SparseReacher, all methods succeed, but Div-DDPG learns faster and achieves higher average rewards.
- Environments with Sparse Rewards: With the SparseHalfCheetah distance threshold set to 15.0, Div-DDPG is the only method that acquires a stable policy.Vanilla DDPG and parameter-noise DDPG rarely exceed the threshold and receive no reward most of the time.
- Overall Findings: Across these experiments, the method efficiently explores continuous-control environments and achieves promising results under deceptive, large-state-space, and sparse-reward conditions.
5 Related Work
The related-work comparison positions diversity-driven exploration as distinct from entropy-based and maximum-entropy approaches. Its central novelty is maximizing a distance measure between current and prior policies through a new loss term.
- Entropy Regularization for RL: Entropy-regularization methods address premature policy convergence through entropy constraints on old and new state-action distributions or prior and current policies.
- Maximum Entropy Principle for RL: Maximum-entropy methods jointly optimize expected entropy and rewards, retaining policy stochasticity while pursuing optimal actions under uncertain dynamics.
- Comparison with Prior Work: The method differs fundamentally from prior work by enhancing exploration efficiency with a novel loss term.
- Comparison with Prior Work: To the authors’ knowledge, this work is the first to encourage exploration by maximizing distance between current and prior policies.
6 Conclusion
The paper presents a diversity-driven exploration strategy that adds a distance term to existing RL losses, and reports superior performance across varied benchmark settings. It also finds benefits in sparse, deceptive, and large-state-space environments and validates adaptive scaling methods.
- Conclusion: The proposed strategy combines effectively with current RL algorithms and promotes behaviors different from an agent’s previous ones through an added distance-measure loss term.
- Conclusion: Experiments show superior performance in most benchmark settings, including environments with sparse or deceptive rewards and large state spaces.
- Conclusion: Analysis of the adaptive scaling methods validates that they improve overall performance.