PaperlayerView exact revision on arXiv

Source-linked AI summary

Repulsive Population Policy Gradients for Behaviorally Diverse Reinforcement Learning

Saeed Hedayatian, Harsh Shah, Celine Mneimneh, Stefanos Nikolaidis

arXiv:2610.36425v1cs.LGcs.AI

TL;DR

QD-RL seeks populations of high-performing policies that differ in their behaviors, while archive-based approaches face scaling costs as policies and behavior spaces grow. RePPO adapts PPO to optimize a compact population using population-dependent shaped rewards. It attains a strong quality-diversity balance on continuous control and extends to pixels and language-model fine-tuning.

  • Problem

    QD-RL methods predominantly rely on evolutionary operators, and archive-based costs grow with policy size and behavior-space complexity, motivating direct optimization of quality and diversity over compact populations.

    Source: §1 · p. 1

  • Method

    RePPO adapts PPO to jointly train a compact policy population using a population-dependent shaped reward derived from the Soft QD objective for additive Markovian behavior descriptors.

    Source: §1 · p. 2

  • Results

    RePPO achieves a better quality-diversity balance than established QD-RL methods on continuous control and, on pixels and LLM fine-tuning, produces substantially more diverse populations at nearly the quality of independent training; this is the first QD-RL demonstration for LLM fine-tuning.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    Directly optimizing compact populations offers a scalable approach to QD-RL across increasingly complex domains.

    Source: §6 · p. 8

    RePPO targets compact populations rather than exhaustive behavior-space coverage, requires additive descriptors, and its hand-designed LLM descriptors yielded differences mostly in presentation rather than strategy.

    Source: §6 · p. 8

Abstract

from arXiv · show

The goal of Quality-Diversity Reinforcement Learning (QD-RL) is to find a population of high-performing policies that solve a decision-making problem in diverse ways. Despite recent progress, QD-RL methods predominantly rely on evolutionary operators--including black-box optimizers, mutations, and crossovers--to facilitate exploration and induce diversity. Scaling these methods becomes increasingly costly as policies and behavior spaces grow larger. Building on the Soft QD view of quality-diversity optimization, we introduce \textbf{Re}pulsive \textbf{P}opulation \textbf{P}olicy \textbf{O}ptimization (\textbf{RePPO}), which adapts PPO to jointly optimize a compact population of diverse, high-quality policies using a population-dependent, dynamically shaped reward. On standard continuous-control tasks, RePPO achieves a strong quality-diversity tradeoff compared with state-of-the-art QD-RL methods. Ablations further characterize this tradeoff by examining how each component affects individual quality and collective diversity. To assess RePPO's scalability, we extend our evaluation to learning from pixels and fine-tuning large language models. In pixel-based tasks, where large policies make most traditional QD-RL methods infeasible, RePPO maintains quality on par with standard PPO while achieving substantially higher diversity. Its successful application to language model fine-tuning further demonstrates, for the first time, that QD-RL methods can scale to this domain. Together, these results show that directly optimizing compact populations with RePPO provides a scalable approach to QD-RL across increasingly complex domains.

1. Introduction

RePPO adapts PPO to optimize a compact population of high-quality, behaviorally diverse policies using population-shaped rewards. The paper motivates this approach by the scaling costs of archive-based QD-RL and evaluates it across continuous control, pixels, and language-model fine-tuning.

1. Introduction

  • QD optimization seeks high task performance and diversity along user-defined behavior descriptors, while archive costs grow with descriptor-space resolution and dimension.The introduction motivates retaining distinct strategies so alternatives for achieving high reward remain explicit.Source: §1 · p. 1
  • A population-dependent shaped reward makes each policy’s gradient match its gradient of the Soft QD lower bound for additive Markovian behavior descriptors.The derivation connects a continuous QD objective to standard policy optimization in RL.Source: §1 · p. 2
  • RePPO extends PPO to jointly train a compact policy population without behavior-space discretization or evolutionary operators.The method includes controls for stable population training and examines their effects on quality and diversity.Source: §1 · p. 2
  • RePPO achieves competitive quality-diversity performance on continuous-control benchmarks and scales to pixel-based policies and language-model fine-tuning, the first QD-RL demonstration in that domain.The evaluation also examines the balance between individual quality and collective diversity through component ablations.Source: §1 · p. 2

2. Related Work

Related work covers archive-based QD objectives and search, QD methods for reinforcement learning, and approaches to diverse policies and behavior representations. RePPO is positioned as direct population optimization using interactions derived from a continuous QD objective.

2.1. Quality-Diversity objectives and search

  • Archive-based QD methods partition behavior space and retain high-quality solutions, while Soft QD offers a continuous population objective optimized through quality-weighted interactions.Search advances include adaptive evolutionary methods and gradient-guided exploration.Source: §2.1 · p. 2

2.2. Quality-Diversity in reinforcement learning

  • QD-RL combines evolutionary search with deep RL updates, while conditioned-policy methods represent diverse behaviors through shared skill-conditioned policies.RePPO instead optimizes a finite population with interactions determined by a continuous QD objective.Source: §2.2 · pp. 2–3

2.3. Diverse policies and behavior representations

  • Other population-diversity objectives use occupancy, determinants, or trajectory-distribution differences, while skill discovery and descriptor learning address how behavioral alternatives are represented.RePPO focuses on population optimization given descriptors, with its policy-gradient construction applying to descriptors that admit an additive representation.Source: §2.3 · p. 3

3. Background

The background defines a Soft QD objective for a population and describes how additive behavior descriptors and policy gradients make its optimization possible in RL. Directional changes in behavior can be estimated from sampled experience without differentiating through the environment.

3.1. The Soft QD population objective

  • The Soft QD lower bound sums individual qualities and penalizes behavioral overlap between pairs, with stronger penalties when both policies have high quality.The interaction scale ℓ determines how far the interactions extend in behavior space.Source: §3.1 · p. 3

3.2. RL with additive behavior descriptors

  • Additive Markovian behavior descriptors represent each behavior coordinate through the expected cumulative value of a per-step signal.The RL setup represents policy quality and behavior through expected cumulative signals.Source: §3.2 · p. 3

3.3. Policy gradients for quality and behavior

  • The policy-gradient theorem provides gradients for task return and scalar behavior signals, allowing directional descriptor changes to be estimated from sampled experience.For a direction v, the corresponding scalar reward is v⊤m(s, a), and the estimate does not require differentiating through the environment.Source: §3.3 · pp. 3–4

4. Repulsive Population Policy Optimization

RePPO jointly trains a compact policy population by shaping PPO updates according to population quality and behavior. It turns Soft QD population interactions into behavior-signal weights that guide policies toward behavioral separation while retaining task learning.

4.1. Population gradients as shaped rewards

  • For additive Markovian behavior descriptors, each policy’s Soft QD gradient is a policy gradient under a population-shaped reward, estimable from sampled experience without differentiating through the environment.The reward combines task performance with a behavior component; its population-derived coefficients are recomputed as the population evolves.Source: §4.1 · p. 4
  • RePPO fixes the task coefficient at one because negative quality coefficients can reverse task learning and cause instability leading to degenerate behaviors.Population summaries and diversity weights are refreshed after rollouts, then held fixed during PPO epochs on that rollout data.Source: §4.1 · p. 4, §4.3 · p. 5

4.2. Constructing diversity weights from the population

  • RePPO constructs each policy’s diversity weights from population qualities and behaviors, computing repulsion in logit-transformed descriptor space among the K nearest policies.The repulsion direction is converted to weights on original behavior signals through the logit Jacobian, then norm-capped to control diversity pressure.Source: §4.2 · pp. 4–5

4.3. PPO updates with population-shaped advantages

  • Separate task and behavior critics estimate advantages for their respective signals; RePPO combines standardized channels using a unit task weight and population-derived behavior weights in the clipped PPO actor update.Setting the behavior weights to zero recovers the task-only PPO actor update.Source: §4.3 · p. 5

5. Experiments

Experiments compare RePPO with established QD-RL methods and independent PPO across continuous control, pixel-based Atari, and LLM fine-tuning. RePPO balances quality and diversity in these settings, with distinct strategies and writing styles across its populations.

5. Experiments

  • The evaluation spans four Brax locomotion tasks, four Atari games, and two LLMs fine-tuned on MATH and evaluated on three test sets.Comparisons use Ind-PPO throughout, archive-based methods in continuous control, RefQD in Atari, and base-model and prompting baselines for LLMs; quality and diversity are measured by mean fitness and Vendi Score.Source: §5 · pp. 5–6

5.1. Continuous control

  • RePPO attains the highest qVS on all four continuous-control tasks, combining PPO-like quality with diversity rivaling archive-based search.On Humanoid, it is about twice as diverse as any archive at mean fitness close to that of the fittest archive policy.Source: §5.1 · p. 6, Figure 1 · p. 7
  • Repulsion lowers population mean fitness but leaves the best policies largely intact; the best policy stays close to Ind-PPO’s and beats nearly all archive-based baselines.Less common contact patterns often produce slower gaits, most visibly on HalfCheetah.Source: §5.1 · p. 7

5.2. Pixel-based policies in Atari

  • In Atari, RePPO retains independent PPO’s quality while achieving substantially higher diversity and qVS than both baselines.Its mean scores are close to or above Ind-PPO’s in every game; RefQD covers behavior space broadly but scores far below PPO-trained policies.Source: §5.2 · p. 7, Figure 1 · p. 7
  • In Seaquest, RePPO policies spread across depths; in four of five seeds, the population’s best policy operates just below the surface and outscores every Ind-PPO policy.Ind-PPO and competent RefQD policies patrol near the sea floor.Source: §5.2 · p. 7, Figure 2 · p. 7

5.3. Fine-tuning large language models

  • RePPO’s LLM members acquire distinct, consistent writing styles while nearly matching independent training’s accuracy.Per-member accuracy drops by about one point, while the population solves as many problems as Ind-PPO’s.Source: §5.3 · p. 8
  • LLM styles differ across members and base models: Qwen3 members vary in code, markdown, and prose, while Granite members vary in verification and notation.A simple classifier identifies RePPO authors more reliably than authors from any other population.Source: §5.3 · p. 8

6. Conclusion and Outlook

RePPO directly trains a compact policy population, using additive Markovian descriptors to turn Soft QD population gradients into PPO updates under shaped rewards. Its scope is useful populations rather than exhaustive behavior-space coverage, while population-shaped rewards may also apply where one optimum is insufficient.

6. Conclusion and Outlook

  • RePPO turns each policy’s Soft QD population gradient into an ordinary policy gradient under a shaped reward, optimized with PPO.The derivation uses additive Markovian behavior descriptors; the reported cost scales with population size rather than archive resolution.Source: §6 · p. 8
  • RePPO targets compact populations of useful policies, not exhaustive behavior-space coverage, where archive-based methods remain better suited.It requires additive descriptors and diversifies what they measure; in the LLM experiments, hand-designed descriptors yielded mostly presentation differences rather than strategy differences.Source: §6 · p. 8
  • Population-shaped rewards may be useful when one optimum is insufficient, including changing conditions and generative models with adapters expressing different preferences or styles.In these settings, diversity can be encoded as a reward induced by population geometry.Source: §6 · p. 8

A. Derivations

The derivation connects the Soft QD population objective to policy gradients for task return and behavior signals. RePPO then converts this result into a PPO update with transformed-coordinate interactions and practical modifications.

A.1. Proof of Proposition 4.1

  • A directional descriptor gradient is an ordinary policy gradient under the per-step signal v^T m, allowing behavioral changes to be estimated from sampled experience without differentiating through the environment.The identity applies the policy-gradient theorem to a general per-step signal, with linearity following from linearity of expectation.Source: §A.1 · p. 13
  • Each policy’s Soft QD gradient decomposes into a task-return gradient and a descriptor-direction gradient, expressible as a policy gradient under a scalar shaped reward.The proof holds the current population coefficients fixed in the gradient expression; their parameter dependence is already accounted for in the derived terms.Source: §A.1 · p. 13

A.2. From the objective to the RePPO update

  • RePPO measures interactions in logit-transformed descriptor coordinates and converts behavior weights back through the logit Jacobian, which increases weights near descriptor bounds.Normalized coordinates are clipped before evaluating the logit and its Jacobian for numerical stability.Source: §A.2 · p. 13
  • The practical update smooths episode statistics, shifts qualities positive, restricts interactions to nearest neighbors, fixes the task coefficient, and caps repulsion.Channel standardization makes diversity weights control relative channel influence independently of numerical scale.Source: §A.2 · p. 14

B. Evaluation protocol

The protocol evaluates retained policies using task fitness and behavior descriptors, then compares RePPO with PPO and archive-based baselines. It specifies evaluation counts, task descriptors, and the principal baseline implementations.

B. Evaluation protocol

  • Policies are reevaluated after training, with 256 deterministic episodes per continuous-control policy, 32 Atari games per policy, and 8 test samples per LLM member.Fitness is mean undiscounted return and descriptors are mean episode descriptors; all reported numbers use these reevaluations.Source: B. Evaluation protocol · p. 14

C.1. Tasks and descriptors

  • Continuous-control descriptors record foot-contact fractions, while task fitness is summed episode reward; the benchmark uses Ant, HalfCheetah, Humanoid, and Walker2D.The contact signals used during training are signed indicators, 2c_t − 1.Source: §C.1 · p. 14

C.2. RePPO and Ind-PPO

  • Ind-PPO matches RePPO’s implementation, population size, and hyperparameters but sets diversity weights to zero; RePPO’s continuous-control hyperparameters are listed in Table 2.The learning rate decays with environment steps and has a 5% floor.Source: Table 2 · p. 15

C.3. Baselines

  • Continuous-control archive baselines include PPGA, PGA-ME, DCRL-ME, and Sep-CMA-MAE, using their reported settings with budgets matched to the comparison where applicable.PPGA is ported to the current Brax version and uses 200 outer iterations; DCRL-ME and PGA-ME use QDax implementations.Source: §C.3 · p. 15

C.4. Additional results

RePPO ranks first in Vendi Score on two continuous-control tasks and second on the other two, despite archive-based methods having larger populations. Additional analyses show how quality filtering affects diversity measurements and how retained behaviors differ across methods.

C.4. Additional results

  • RePPO has the highest Vendi Score on two tasks and the second highest on the other two, despite archive-based methods’ larger populations.Table 4 reports final populations as mean ± standard error over seeds; metrics other than best fitness concern retained policies.Source: §C.4 · p. 16, Table 4 · p. 16
  • On Ant, RePPO attains the highest qVS above the 25% fitness threshold, while archive diversity falls as the threshold rises because many distinct policies barely move.Figure 4 varies the minimum required fitness as a fraction of the task-best policy and plots Vendi Score and qVS for complete populations.Source: §C.4 · p. 16, Figure 4 · p. 17
  • Archive methods cover behavior space more uniformly, but much of that coverage is low-fitness; RePPO is less uniform and concentrates policies in regions supporting fast locomotion.Independent PPO instead concentrates policies in a small, high-quality region.Source: §C.4 · pp. 16–17
  • RePPO populations include gaits that differ in which feet carry the body and how often they touch the ground across HalfCheetah, Walker2D, and Humanoid.Figures 7–9 show gaits from one population per task.Source: §C.4 · p. 17, Figure 7 · p. 19, Figure 8 · p. 19

D. Ablations

The Humanoid ablations show that task improvement requires a stable task coefficient, while symmetric repulsion and tuned interaction strength and population size balance quality with diversity. Atari implementation details specify the behavior signals, policy architecture, and RefQD comparison setup.

D.1. Structural choices

  • With the exact task coefficient, every seed collapsed and the best policy reached fitness 282 ± 230; fixing Wk = 1 preserves a consistent task-improvement signal.When high-quality policies overlap early, the exact coefficient can become negative and reward lower returns.Source: §D.1 · p. 21
  • Asymmetric repulsion left best-policy fitness similar (8409 ± 201 versus 8336 ± 68) but reduced qVS from 39212 to 34086; symmetric interactions spread the population more evenly.In the asymmetric variant, policies are repelled only by higher-quality neighbors.Source: §D.1 · p. 21

D.2. Interaction strength and scale

  • Raising Cmax from 0.1 to 1 increased Vendi Score from 3.92 to 6.33 and spread from 0.60 to 0.91, while larger caps could reduce fitness; bandwidth changes interaction scale and population geometry.At ℓ2 = 0.5, spread saturated near 1.0 and Vendi Score peaked at 6.84, while qVS peaked at ℓ2 = 0.1; Figure 10 plots parameter effects and Figure 11 shows final populations.Source: §D.2 · p. 21, Figure 10 · p. 22, Figure 11 · p. 22

D.3. Local interactions

  • For K ∈ {4, 8, 16}, Vendi Score stayed between 6.33 and 6.50 and qVS remained within 4% of default; K = 8 retains most interaction at lower cost.Using the entire population (K = 31) produced slightly fitter but less diverse populations; Figure 12 (left) shows the comparison.Source: §D.3 · p. 22, Figure 12 · p. 23

D.4. Population size at a fixed budget

  • At fixed environment-step budget, Vendi Score rose from 3.76 at N = 8 to 6.97 at N = 64, but mean fitness dropped at N = 64; N = 32 balances these effects.Best fitness remained above 7200 at every population size, and Figure 12 (right) shows the population-size comparison.Source: §D.4 · p. 22, Figure 12 · p. 23

E.1. Games and descriptors

  • Atari policies use two-dimensional behavior signals averaged over agent steps; positional descriptors are clipped to [−1, 1], while Ms. Pac-Man’s descriptor records which maze half is cleared first.The four games are Boxing, Asterix, Seaquest, and Ms. Pac-Man; Figure 13 illustrates them and their behavior dimensions.Source: §E.1 · p. 23, Table 6 · p. 24, Figure 13 · p. 24

E.2. RePPO and Ind-PPO

  • Each Atari policy is a separate 1.69 M-parameter Nature CNN with actor, task-critic, and two-output behavior-critic heads; behavior channels run through full games and use rewards scaled by 1 − γ.RePPO and Ind-PPO hyperparameters are listed in Table 7.Source: §E.2 · p. 24, Table 7 · p. 24

E.3. RefQD

  • The Atari Ref-DQN-ME comparison uses a 1000-cell CVT archive, two decision parts per cell, 100 offspring per generation, and 300 DQN training steps per generation.Its 80 M-step budget is rounded up to 86.4 M to complete each re-evaluation cycle; Seaquest uses a larger replay buffer for longer episodes.Source: §E.3 · pp. 24–25

E.4. Additional results

RePPO’s Atari populations add behavioral variation along dimensions where Independent PPO policies often remain concentrated. The reported comparisons include spatial patterns across games and, for Ms. Pac-Man, the order in which maze regions are cleared.

E.4. Additional results

  • RePPO policies spread across ring heights and closer-range fighting in Boxing, and add outer-lane coverage in Asterix and top-half-first clearing in Ms. Pac-Man.In Asterix and Ms. Pac-Man, Ind-PPO already varies along one dimension, which the games’ symmetry makes equally rewarding; Table 8 reports final populations over five seeds.Source: §E.4 · p. 25, Table 8 · p. 25
  • Across the first seed’s eight policies per game, RePPO adds right-side and edge fighting in Boxing and varied depths and horizontal positions in Seaquest, while its best Seaquest policy stays just below the surface.Figure 14 shows player positions for Boxing and Seaquest and dot-eating order for Ms. Pac-Man; both methods start Ms. Pac-Man at the bottom and differ in which regions they leave for last.Source: §E.4 · p. 25, Figure 14 · p. 26

F.1. Task and data

The MATH fine-tuning setup defines training and checkpoint-selection data, a binary correctness-based fitness, and four solution-text descriptors. Correctness gates the descriptors, making fitness and behavior signals terminal rewards in the token-level MDP.

F.1. Task and data

  • Training uses 5,584 levels 3–5 MATH problems with boxed reference answers; checkpoint selection uses 384 same-level test problems excluding MATH-500.The test set combines MATH-500, Minerva, and English text-only OlympiadBench; base models are Qwen3-1.7B-Base and Granite-4.1-3B-Base.Source: §F.1 · p. 27
  • Fitness is 1 when a solution finishes within 4096 tokens and its last boxed expression matches the reference under Math-Verify, and 0 otherwise.The training reward standardizes this score using the base model’s first-batch mean and standard deviation.Source: §F.1 · p. 27
  • Four descriptors are computed from solution text up to the last boxed answer, and multiplying each by correctness assigns incorrect solutions the neutral value 0.Fitness and descriptors are revealed only when the solution is complete, so both are terminal rewards in the token-level MDP; Table 9 lists the raw features and scales.Source: §F.1 · p. 27, Table 9 · p. 27

F.2. Training

This section describes RePPO’s LLM fine-tuning and evaluation, including benchmark findings and qualitative examples of responses to a triangle-area problem and a constrained-seating problem. It also notes that learned diversity depends on the descriptors being optimized and that RePPO responses to the triangle-area problem vary in length.

F.2. Training

  • Each RePPO member uses rank-32 LoRA actor adapters and its own rank-64 critic adapter, and training follows PPO without a KL penalty or entropy bonus.Each iteration samples eight solutions for each of the same 128 problems per member, and checkpoints are selected by validation mean accuracy every 10 iterations.Source: §F.2 · p. 27

F.3. Baselines and evaluation

  • Evaluation samples eight solutions per member and 40 from the base model, measuring pass@1, population pass@5j, and member identifiability.Population pass@5j uses an unbiased estimator; identifiability is held-out logistic-regression accuracy on TF-IDF word unigrams and bigrams.Source: §F.3 · p. 28

F.4. Additional results

  • Prompting instructions shift member styles only moderately, even when they explicitly request extreme descriptor values.This pattern is shown for the prompting baselines in Figure 15.Source: §F.4 · p. 28, Figure 15 · p. 28
  • Qwen3 pass@40 rises by 2 points and Granite pass@40 by 9 points after fine-tuning; RePPO’s lower pass@1 is mainly attributable to one member per population.The Qwen3 code-writing member produces long solutions and guesses program outputs, while a Granite member rarely uses an explicit reasoning block.Source: Table 11 · p. 28, §F.4 · p. 28

F.5. Qualitative examples

  • RePPO members maintain distinct, consistent solution styles, while Ind-PPO members are hard to distinguish in both models.Qwen3 members vary in code use and list formatting; Granite members differ mainly in how they reason within or omit the <think> block.Source: §F.5 · p. 29
  • Some learned styles are undesirable: Qwen3’s code outputs are correct only half the time, and Granite verification may repeat checks.Both behaviors maximize the defined descriptors, showing that descriptor design determines the diversity RePPO produces.Source: §F.5 · p. 29
  • For the cited triangle example, the reported area is 12 square units; the passage states this result directly without specifying side lengths or a calculation method.Source: §F.5 · p. 36
  • For seating 6 girls and 2 adjacent boys, the worked block method counts 7! × 2! arrangements, yielding 10,080.Treat the boys as one block among seven units, then account for their two internal orders.Source: §F.5 · p. 37
  • The cited RePPO results report Member 0 as correct with a response length of 217 tokens and Member 2 as correct with a response length of 401 tokens.Source: §F.5 · pp. 40–41
  • Ind-PPO Members 0 and 1 are both reported correct on the example problem.Source: §F.5 · p. 42

G. Pixel-based locomotion

Pixel-based locomotion tests RePPO with large convolutional policies trained directly from images. Across three tasks, it is more behaviorally diverse than Ind-PPO and achieves higher qVS in every task, with task-dependent fitness tradeoffs.

G. Pixel-based locomotion

  • RePPO trains populations of 16 pixel-input convolutional policies on HalfCheetah, Humanoid, and Walker2D, using 2.5 M-parameter policies and 262 M total training steps.Archive-based baselines are omitted because they would require storing and training thousands of such policies.Source: G. Pixel-based locomotion · p. 43
  • Vendi Scores are 1.9–4.2 for RePPO versus 1.2–1.5 for Ind-PPO, and RePPO achieves higher qVS in every task.RePPO also has higher mean fitness on HalfCheetah and Walker2D, while trading part of it for diversity on Humanoid.Source: G. Pixel-based locomotion · p. 43, Table 12 · p. 43
  • Figure 16 shows RePPO policies spread across contact patterns, while Ind-PPO policies concentrate in a small region.The figure presents behavior descriptors for pixel-based policies, with one seed per task.Source: G. Pixel-based locomotion · p. 43, Figure 16 · p. 43

H. LunarLander

On LunarLander, RePPO preserves behavioral diversity when the descriptor is observed at a single landing event, while trading some mean fitness for diversity. Its diversity matches Sep-CMA-MAE’s with higher mean fitness, producing the highest qVS.

H. LunarLander

  • LunarLander tests RePPO with discrete actions and a sparse descriptor recording horizontal position and vertical speed at first ground contact.The signal is nonzero at only one step per episode; episodes without contact receive descriptor (0, 0). Diversity necessarily costs quality because reward favors fuel-efficient, gentle landings near the pad.Source: H. LunarLander · pp. 43–44
  • RePPO and Ind-PPO train 32 policies with 64-unit MLPs for 65.5 M steps; Sep-CMA-MAE uses a 32 × 32 grid archive under the same environment-step budget.All methods use five seeds.Source: H. LunarLander · p. 44
  • RePPO doubles Ind-PPO’s Vendi Score and matches Sep-CMA-MAE’s with higher mean fitness, giving it the highest qVS.All three methods find task-solving policies, with best fitness between 270 and 288.Source: H. LunarLander · p. 44, Table 13 · p. 44
  • RePPO’s population lands across the screen width, varying mainly in position, while Ind-PPO’s policies land at nearly the same spot.Figure 17 shows one seed per method; Sep-CMA-MAE’s archive is thinned to one fittest policy per each of 32 CVT cells.Source: H. LunarLander · p. 44, Figure 17 · p. 44

H.1. Computational Resources

  • Atari RePPO uses about 4 GB of GPU memory; LLM population training took 21 h for Qwen3 and 64 h for Granite across five RTX A6000 GPUs.These times correspond to about 100 and 320 GPU hours, respectively.Source: §H.1 · p. 44

Related papers

Loading 2610.36425v1…