Source-linked AI summary
Effective Diversity in Population Based Reinforcement Learning
Jack Parker-Holder, Aldo Pacchiano, Krzysztof Choromanski, Stephen Roberts
TL;DR
Reinforcement-learning exploration is limited by agents’ dependence on environmental experience, while population diversity methods can cycle, rely on handcrafted embeddings, and disrupt fruitful reward optimization. DvD jointly optimizes the population using determinant-based behavioral volume, task-agnostic action embeddings, and online adaptation of the reward-diversity trade-off. Across challenging tasks, its evolutionary and gradient-based implementations find diverse high-quality solutions while maintaining performance when only one good solution is needed.
Problem
Reinforcement-learning agents learn from acquired environmental experience, but existing population-diversity methods can cycle, use handcrafted behavioral representations, and make reward-diversity trade-offs difficult to set.
Method
DvD jointly updates population members, measures population diversity as determinant-based behavioral volume using action-based embeddings, and adapts the diversity objective with Thompson sampling.
Results
Across challenging tasks, DvD finds diverse, high-quality solutions and maintains strong performance in settings requiring one good solution; DvD-TD3 reached a median best-agent score of 6091 on Humanoid-v2.
Takeaways & Limitations
DvD provides evolutionary and off-policy RL implementations that improve exploration without reducing performance when diversity is not required.
Takeaways & Limitations
DvD increases mean wall-clock time, and performance depends on choosing a suitable population size M.
Abstract
from arXiv · showhide
Exploration is a key problem in reinforcement learning, since agents can only learn from data they acquire in the environment. With that in mind, maintaining a population of agents is an attractive method, as it allows data be collected with a diverse set of behaviors. This behavioral diversity is often boosted via multi-objective loss functions. However, those approaches typically leverage mean field updates based on pairwise distances, which makes them susceptible to cycling behaviors and increased redundancy. In addition, explicitly boosting diversity often has a detrimental impact on optimizing already fruitful behaviors for rewards. As such, the reward-diversity trade off typically relies on heuristics. Finally, such methods require behavioral representations, often handcrafted and domain specific. In this paper, we introduce an approach to optimize all members of a population simultaneously. Rather than using pairwise distance, we measure the volume of the entire population in a behavioral manifold, defined by task-agnostic behavioral embeddings. In addition, our algorithm Diversity via Determinants (DvD), adapts the degree of diversity during training using online learning techniques. We introduce both evolutionary and gradient-based instantiations of DvD and show they effectively improve exploration without reducing performance when better exploration is not required.
1 Introduction
Exploration motivates population-based reinforcement learning because agents learn from environmental experience, while existing diversity methods face cycling, handcrafted behavior spaces, and reward-diversity trade-offs. DvD addresses these issues by measuring population-wide behavioral volume, adapting diversity online, and providing evolutionary and gradient-based implementations.
- Exploration is crucial because reinforcement-learning agents typically learn only from experience acquired in the environment.
- Population-based training gathers diverse experiences within similar wall-clock training time, motivating its use in reinforcement learning.
- Mean-field pairwise optimization can create cycling, preventing individual agents from exploiting promising behaviors.
- Quality Diversity methods commonly depend on predefined behavioral characterizations, such as final locomotion coordinates, which may not generalize across environments.
- DvD measures the entire population’s behavioral diversity as the volume of an inter-agent kernel matrix while retaining local reward optimization.
- DvD adapts the reward-diversity objective with Thompson sampling and includes evolutionary DvD-ES and off-policy DvD-TD3 implementations.
2 Preliminaries
The preliminaries describe policy representations, reward optimization, population diversity objectives, and their limitations. They motivate task-agnostic action-based behavioral embeddings and simultaneous population updates as the basis for DvD.
- A policy maps states to actions and is typically represented by neural-network parameters, while model-free RL uses on-policy gradients or off-policy value learning.
- On-policy reinforcement learning optimizes policy parameters to maximize total expected or discounted reward over a finite rollout horizon.
- Evolution Strategies optimize policy performance as a black-box function and naturally maintain populations that can support deep exploration.
- Novelty-search formulations augment reward objectives with behavioral novelty, but action-based behavior embeddings have received limited attention in policy-gradient methods.
- Reward-diversity formulations can boost exploration but may rely on heuristic embeddings, move high-performing policies away from improved reward, and produce population cycles.
- DvD addresses these issues by updating all agents simultaneously.
3 Diversity via Determinants
DvD represents policies with task-agnostic action-based behavioral embeddings and measures population diversity using the determinant of an inter-agent similarity matrix. The resulting objective jointly optimizes individual rewards and population-wide diversity, with theoretical analysis showing that determinants favor distinct high-performing solutions and avoid redundancy that pairwise distances can miss.
- 3.1 Task Agnostic Behavioral Embeddings: Behavioral embeddings represent each policy by the actions it takes across a finite set of states.For deterministic policies, identical embeddings imply identical behavior on every state, although parameter vectors may differ.
- 3.2 Joint Population Update: Population diversity is defined as the determinant of a positive-semidefinite kernel matrix computed from policy embeddings.The determinant represents the volume spanned by kernel feature maps, encouraging the population to fill behavioral space.
- 3.2 Joint Population Update: The DvD objective combines local individual rewards with a global population-diversity term while directly optimizing the full parameter population.The trade-off coefficient λ_t weights the two objectives.
- 3.2 Joint Population Update: In a finite tabular MDP with at least M distinct optimal policies, some positive λ_t makes the objective maximizable only when the population contains M distinct optimal solutions.The determinant’s first-order approximation relates to mean pairwise L2 distance, but for larger populations its first-order term is zero and higher-order terms determine the determinant.
- 3.2 Joint Population Update: Unlike pairwise distance, determinants remain low for clustered or low-rank populations even when average pairwise distances are high.This targets redundant policies whose embeddings can be represented using fewer canonical policies.
- 3.3 An Illustrative Example: Tabular MDP: For the illustrative five-agent MDP, the 5 × 5 kernel determinant is positive only when the population contains exactly the five listed optimal policies, whereas pairwise distance prefers a redundant alternative.The redundant alternative duplicates one optimal policy while achieving a higher pairwise-distance value.
4 DvD Algorithm
DvD approximates or differentiates population-wide diversity while collecting reward signals locally, and it adaptively selects the reward-diversity trade-off with Thompson sampling. Practical implementations use sampled states for embeddings and include evolutionary-strategy and gradient-based updates.
- 3.1 Task Agnostic Behavioral Embeddings: Action-based embeddings must use sampled states when the state space is intractably or infinitely large.Random state sampling corresponds to frequency weights, while DPP-driven or probabilistic state selection are alternative choices.
- 4 DvD Algorithm: DvD treats the reward-diversity coefficient λ_t as an adaptive decision rather than a fixed user-specified priority.The objective is framed as a multi-armed bandit problem that can favor reward or diversity at different optimization stages.
- 4 DvD Algorithm: Thompson sampling selects among trade-off arms by sampling candidate mean rewards from posterior models and updating the selected arm after observing its reward.The practical Bernoulli signal indicates whether the next policy improves the observed reward.
- 4 DvD Algorithm: The Bernoulli bandit model uses two arms, λ = 0 and λ = 0.5, with beta priors and posteriors under a simplifying stationarity assumption.The chosen arm receives reward 1 when the next observed reward exceeds the current reward.
- 4 DvD Algorithm: In the evolutionary-strategy implementation, perturbations are evaluated with local rewards and global diversity, then combined into a blackbox objective for the population parameters.Gaussian perturbations are partitioned separately for workers computing local rewards and for groups contributing to diversity.
- 4 DvD Algorithm: DvD-ES uses a smoothed gradient update, while DvD-TD3 uses analytic diversity gradients and multiple policies sharing a replay buffer.DvD-TD3 computes embeddings from samples in the existing optimization batch.
5 Related Work
Related work spans neuroevolution, quality-diversity optimization, novelty-based reinforcement learning, unsupervised diverse-behavior learning, and behavioral-space meta-learning. These approaches differ in their use of perturbation-based optimization, behavioral representations, reward signals, and task-specific or pretrained embeddings.
- Neuroevolution: Neuroevolution optimizes policy rewards through perturbations, gradient estimates, or selection of high-performing perturbations, with simplicity and scalability supporting its use in reinforcement learning.The paper places DvD within this broader family of biologically motivated optimization methods.
- Behavioral Diversity: Quality-diversity and novelty methods commonly optimize reward jointly with behavioral difference measured against other policies.One cited approach uses mean pairwise distance as the diversity measure.
- Diverse Behaviors and Meta-Learning: Other reinforcement-learning methods learn diverse behaviors without rewards or focus on behavioral-space meta-learning, but some require pretraining on similar tasks.The cited unsupervised methods define behavior through visited states.
6 Experiments
The experiments test DvD across multimodal and single-mode environments, kernel choices, adaptive weighting, and Humanoid control. DvD learns diverse solutions while preserving performance when diversity is unnecessary, and improves Humanoid results.
- Multi-Modal Environments: DvD learns both forward and backward behaviors in Cheetah and Ant, whereas vanilla ES struggles with the multimodal objectives.For Cheetah, vanilla ES quickly learns the simpler backward task but performs poorly on forward; in Ant, it fails to learn.
- Single Mode Environments: Across four single-mode environments, DvD incurs only a minimal performance drop versus reward-focused vanilla ES, while NSR-ES fails to train.DvD also outperforms ES on Walker2d, whose survival bonus induces a deceptive local optimum at 1000.
- Single Mode Environments: DvD avoids the cycling seen in novelty search, where successful behaviors are initially found and later abandoned while pursuing novelty.This behavior is demonstrated in the single-mode experiments through NSR-ES performance that deteriorates after initially improving.
- Choice of kernel: Performance remains strong across almost all tested DPP kernels and is similar to the squared exponential kernel used in the main experiments.The kernel comparison uses median rewards from five seeds, with algorithms otherwise held identical.
- Do we need to adapt?: The adaptive mechanism boosts performance across all five fixed-λ experiments compared with using λ = 0.5.The fixed-weight variant still outperforms NSR-ES on the joint diversity score, but adaptation provides an additional improvement.
- Teaching a Humanoid to Run with DvD-TD3: DvD-TD3 achieves stronger sample efficiency and final performance on Humanoid-v2, with median best-agent rewards of 6091 versus 5654 for E-TD3 and 5727 for TD3.The differences versus both baselines are statistically significant at p < 0.05; DvD also has higher mean wall-clock time.
7 Conclusion and Future Work
DvD promotes diverse, high-quality solutions while maintaining strong performance when tasks have one good solution. The paper identifies adapting population size and learning embeddings or kernels as future directions.
- DvD addresses cycling through joint population updates based on determinants of policy embeddings.
- DvD adapts the reward-diversity trade off online to provide flexible diversity during training.
- Across challenging tasks, DvD finds diverse, high-quality solutions while maintaining strong performance in one-good-solution settings.
- Future work includes learning the optimal population size and learning embeddings or kernels that identify important action-space dimensions.
Broader Impact
The paper presents increased diversity as potentially beneficial for populations and ensembles through performance, generalization, robustness, and fairness-related effects. It also argues that task-agnostic behavioral embeddings could support novelty search across tasks without domain knowledge.
- Increased diversity may improve sample efficiency and asymptotic performance for populations of agents or ensembles of models.
- Diverse ensemble members may improve generalization by reducing overfitting to the same features.
- Diversity may improve robustness and fairness by encouraging predictions based on broader characteristics.
- Behavioral embeddings are presented as task-agnostic representations that work across tasks without domain knowledge for novelty search.
8 Additional Experiment Details
Additional experiments test DvD’s sensitivity to behavioral-embedding choices, kernel choice, and population size, alongside fixed configurations for the reported algorithms. Embedding choices and kernels generally have little performance impact, whereas an oversized population can hurt performance.
- Embedding sensitivity: The ablations vary the number and selection mechanism of states used to construct behavioral embeddings.
- Embedding sensitivity: State-selection choices have minimal impact: Point escapes its local maximum and Swimmer reaches rewards above 300 in all cases.
- Kernel choice: Kernel choice produces strong and usually similar performance to the squared exponential kernel used in the main experiments.
- Population size: With two modes, increasing population size from M = 3 to M = 5 was harmful for performance.
- Configurations: The reported experiments used fixed behavioral-embedding and algorithm configurations, with environment-specific neural-network and training settings.
9 Theoretical Results
The theoretical results characterize determinant-based diversity relative to pairwise distances and establish conditions under which the objective favors distinct optimal policies. For populations larger than three, determinant diversity depends on higher-order terms rather than first-order pairwise-distance behavior.
- Objective and gradients: Determinant diversity is defined from a positive-semidefinite kernel matrix, and gradients can be computed through the objective for deterministic behavioral embeddings.
- Optimality result: Under 0 < λt < ∆ and at least M distinct solutions, the objective’s maximizer corresponds to M distinct optimal policies.
- Determinants versus distances: For M ≤ 3, the first-order approximation of determinant diversity is proportional to the sum of pairwise embedding distances; for M > 3, it is zero.
- Determinants versus distances: For three policies with a squared exponential kernel, the determinant is approximately the mean pairwise l2 distance when embeddings are close.
- Determinants versus distances: When M > 3, determinant diversity varies through higher-order terms and is not equivalent to a pairwise-distance score.
10 Extended Background
The background reviews population-based exploration, Gaussian-smoothed Evolution Strategies, and novelty-search updates. It highlights novelty search’s reliance on pairwise embeddings and its difficulty rigorously measuring whole-population diversity.
- Evolution Strategies smooth blackbox objectives with Gaussian perturbations, enabling stochastic gradient estimates and gradient-based policy updates.The smoothing level is controlled by σ, and Monte Carlo estimators use perturbed-policy rewards.
- Novelty-search reinforcement learning augments optimization with embedding-based novelty, computed as mean Euclidean distance from the other population members.The approach sequentially selects individual agents, evaluates perturbed policies, normalizes reward and novelty, and updates the selected policy.
- The novelty weight λ controls the reward-diversity trade-off: λ = 0 recovers standard Evolution Strategies, whereas λ = 1 optimizes only diversity.This fixed-population template updates agents using normalized reward and novelty signals.
- Mean pairwise Euclidean distance can overestimate diversity when M policies collapse into fewer than M conjugacy classes, lacking a rigorous whole-population evaluation.