PaperlayerView exact revision on arXiv

Source-linked AI summary

Understanding Evolution Strategies for LLM Reasoning: Broader Reasoning Coverage than GRPO

Yunpeng Ba, Zhi Zheng, Yue Xie, Jiaqing Li, Xialiang Tong, Tao Zhong, Mingxuan Yuan, Zhichao Lu, Xuyang Wu, Zhenkun Wang

arXiv:2608.27351v1cs.LG

TL;DR

The paper addresses the limited understanding of ES optimization behavior and its scope relative to GRPO for LLM reasoning. It combines theoretical analysis, empirical comparisons, parameter-drift analysis, and hyperparameter studies, finding broader reasoning coverage under ES alongside sparse functional effects and scale-dependent population requirements. These results position ES as a distinct reasoning post-training paradigm, while continual-learning behavior over longer multi-task horizons remains open.

  • Problem

    Existing evidence insufficiently characterizes ES’s optimization behavior, its reasoning-coverage advantage over GRPO, capability preservation under parameter drift, and scalable hyperparameter settings.

  • Method

    The paper theoretically and empirically compares ES with GRPO, analyzes the functional concentration of ES updates, and evaluates ES hyperparameters and estimator designs.

  • Results

    ES improves Pass@1 and achieves higher Pass@K than GRPO without the same entropy collapse, while held-out capabilities remain largely preserved under appropriate settings.

  • Takeaways & Limitations

    ES is a distinct reasoning post-training paradigm, with broader coverage than GRPO and smaller effective population-size requirements for larger pretrained models.

  • Takeaways & Limitations

    The effects of ES on previously acquired capabilities may depend on task and training configuration, especially over longer continual-learning horizons spanning multiple tasks.

Abstract

from arXiv · show

Evolution Strategies (ES) have recently emerged as a memory-efficient post-training paradigm for LLM reasoning. However, the optimization behavior of ES remains understudied, making it hard to define its advantage scope compared to mainstream post-training paradigms (e.g., Group Relative Policy Optimization (GRPO)). By systematically investigating ES dynamics and mechanisms, this paper first identifies a performance advantage of ES over GRPO, theoretically and empirically showing that ES can lead to broader reasoning coverage, thereby better exploiting the reasoning capabilities of pretrained LLMs. Theoretically, we show that verifier-projected Jensen-Shannon diversity across the ES population is helpful to higher Pass@K performances. Empirically, unlike GRPO, which exhibits entropy collapse, ES improves Pass@1 while attaining higher Pass@K than GRPO. We further develop a sequential GRPO-ES training strategy that combines GRPO's strength in Pass@1 with ES's gains in Pass@K. Second, we find that despite substantial whole-model parameter drift, the task-performance gains of ES are only contributed to a sparse subset of larger-magnitude updates. This functional sparsity suggests that large parameter movement need not imply widespread functional change, and held-out evaluations further show that it does not necessarily lead to catastrophic forgetting. Finally, we study how hyperparameter design affects the effectiveness of ES, demonstrating that ES requires a smaller population size in a larger LLM. These findings position ES as a distinct reasoning post-training paradigm rather than a less effective, memory-efficient alternative to GRPO.

1 Introduction

The paper studies whether ES offers broader reasoning coverage and better capability preservation than GRPO, while identifying the conditions that make ES effective and scalable. It finds that ES improves Pass@1 and broader Pass@K coverage, concentrates task-relevant effects in sparse updates, and benefits from specific normalization and population-size choices.

  • Motivation: ES searches over parameter-perturbed policies, whereas GRPO samples responses from one policy and backpropagates token-level relative advantages.ES uses forward evaluations and reward-weighted perturbations, avoiding backpropagation and enabling memory-efficient, parallelizable post-training.
  • Open boundary: ES capability preservation may vary across tasks and training configurations, especially in continual-learning scenarios with repeated adaptation.The paper identifies longer-horizon multi-task continual learning as an area requiring further study.
  • RQ1: Reasoning coverage: ES improves Pass@1 while achieving higher Pass@K than GRPO across GSM8K and DeepScaleR post-training settings.The paper also reports that ES avoids the entropy collapse observed under GRPO and proposes ES→GRPO and GRPO→ES compositions to combine their strengths.
  • RQ2: Capability preservation: ES’s task-relevant effects are concentrated in a small subset of larger-magnitude updates, while most parameter changes contribute little after perturbation cancellation.Held-out capabilities remain largely preserved under appropriate settings, so substantial whole-model drift need not imply widespread functional change or catastrophic forgetting.
  • RQ3: Optimization design: Effective ES training requires z-score reward normalization, while the two-point estimator provides no advantage for discrete-reward reasoning.The population size required for effective optimization decreases as pretrained model scale increases.

2 Preliminaries

The preliminaries define verifier-reward optimization for reasoning trajectories and contrast GRPO’s single-policy, token-level updates with ES’s population-based parameter-space updates. They also explain why GRPO can suffer entropy collapse and how ES replaces backpropagation with reward-weighted perturbation estimates.

  • Reasoning objective: An LLM maps a reasoning prompt to a chain-of-thought trajectory and final answer, which together form the response evaluated by a verifier.The verifier assigns a scalar reward to the complete response, defining the optimization objective.
  • GRPO: GRPO samples a response group from one behavior policy, normalizes their verifier rewards, and applies each response-level advantage across its entire trajectory.The group reward supplies a prompt-specific baseline without a learned critic, while the PPO-style clipped objective updates token probabilities.
  • GRPO: GRPO can reduce policy entropy when high-probability actions receive positive advantages, concentrating exploration during repeated updates.This concentration can reduce the chance of finding correct responses through repeated sampling even when single-sample accuracy improves.
  • ES: ES estimates an update direction by evaluating Gaussian parameter perturbations and aggregating their rollout rewards without differentiating through the model.In the described implementation, rewards are standardized within each population before the center model is updated.
  • Comparison: GRPO backpropagates token-level advantages through a single policy, whereas ES converts population-level reward differences directly into a parameter-space update.This mechanism-level distinction motivates comparing their post-training behavior, capability preservation, and optimization scalability.

3 RQ1: Does ES Exhibit the Same Post-Training Characteristics as GRPO?

ES differs from GRPO by preserving broader reasoning coverage: population diversity can support higher Pass@K, and experiments show gains at both single- and repeated-sampling budgets.

  • Theoretical mechanism: ES population diversity can increase the chance that repeated sampling finds a correct answer compared with matched sampling from one policy.Parameter perturbations create heterogeneous policies with different success rates; sampling across members can improve discovery.
  • Theoretical mechanism: Reward weighting improves population success when weights are positively correlated with member success rates.The analytical improvement over uniform averaging is proportional to the covariance between weights and success probabilities.
  • Theoretical mechanism: Under sufficient conditions, the ES center update can preserve the population advantage and improve expected Pass@K over the initial policy.The result depends on verifier-relevant heterogeneity, reward-aligned selection, and sufficiently small center-transfer error.
  • Empirical comparison: During GSM8K post-training, ES keeps held-out GPQA entropy largely stable and finishes above the base model on Pass@16 and Pass@32, unlike GRPO.GRPO’s entropy declines substantially and its Pass@16 and Pass@32 finish below the base model.
  • Empirical comparison: Across representative tasks and both settings, ES improves Pass@1, Pass@16, and Pass@32, while GRPO’s larger Pass@1 gains often accompany lower large-K performance.GRPO falls below the corresponding base model on Pass@16 and Pass@32 in 15 of 18 Easy-Setting comparisons.
  • Sequential compositions: Sequential ES→GRPO and GRPO→ES training adds non-dominated Pass@1–Pass@K trade-offs under the same total update budget.ES→GRPO attains the highest Pass@32 on the Hard-Setting math average while retaining most of GRPO’s Pass@1 gain.

4 RQ2: Does ES Necessarily Cause Catastrophic Forgetting?

ES moves substantially farther from initialization than GRPO, but its task-relevant gains are concentrated in a small subset of larger-magnitude updates. Held-out results suggest that this drift does not necessarily cause catastrophic forgetting, although preservation may vary across tasks and configurations.

  • 4.1 ES Induces Greater Parameter Drift: 40.7–44.1 times farther from initialization than GRPO, Full ES confirms substantially greater whole-model parameter drift across four models.The drift may partly reflect accumulated random-walk movement in weakly reward-relevant directions.
  • 4.2 ES Effects Are Concentrated in a Small Subset of Larger-Magnitude Updates: 77.6–93.0% of nonzero updates fall within magnitudes up to τ = 1.5 × 10−3, leaving only 7.0–22.4% above the threshold.Larger-magnitude updates therefore form a smaller subset of changed coordinates.
  • 4.2 ES Effects Are Concentrated in a Small Subset of Larger-Magnitude Updates: Target-task Pass@1 remains broadly stable as small-magnitude updates are removed, with noticeable degradation emerging only at high update sparsity.Retaining larger updates preserves most performance gains, indicating magnitude sparsity with corresponding functional concentration.
  • 4.2 ES Effects Are Concentrated in a Small Subset of Larger-Magnitude Updates: The largest ES updates occur mainly in LayerNorm weights and attention projections, concentrating substantial changes in normalization and attention parameters.Reported maxima include 0.01171875 for Llama-3.2-3B-Instruct and 0.015625 for DeepSeek-R1-Distill-Qwen-1.5B.
  • 4.3 Large Parameter Drift Does Not Necessarily Indicate Broad Forgetting: ES generally maintains held-out performance despite large parameter drift, with positive average Pass@32 changes under ES but negative changes under GRPO across three Easy Setting models.In the Hard Setting, ES also achieves higher average Pass@32 than GRPO on held-out non-mathematical benchmarks.
  • 4.3 Large Parameter Drift Does Not Necessarily Indicate Broad Forgetting: Capability preservation may be inconsistent across tasks and repeated updates, since exploratory work suggests ES effects depend on the task and training configuration.This boundary is especially relevant to continual-learning scenarios requiring repeated adaptation while preserving prior capabilities.

5 RQ3: What Parameter Settings and Estimators Make ES Effective and Scalable?

The paper evaluates reward normalization, perturbation scale, population size, and estimator choice to identify effective and scalable ES configurations. Z-score normalization helps optimization, smaller populations become more adequate as model scale grows, and two-point estimation offers no advantage for reasoning tasks.

  • 5.1 Reward Normalization, Perturbation Scale, and Population-Size Scaling: Z-score reward normalization yields higher mean rewards than no normalization throughout evaluated post-initialization updates in the matched one-point ES ablation.Normalization makes updates depend on relative rather than absolute reward scales.
  • 5.1 Reward Normalization, Perturbation Scale, and Population-Size Scaling: Perturbation scale σ controls both exploration radius and objective smoothing, with excessively small or large values risking overfitting or unstable training.σ should limit early reward overfitting while maintaining stable reward-guided progress.
  • 5.1 Reward Normalization, Perturbation Scale, and Population-Size Scaling: At update 300, N = 16 is within 0.01 of N = 64 for the 1.5B and 3B models, whereas only N = 32 meets this criterion for the 0.5B model.The N = 16-to-N = 64 gap falls from 0.0352 to 0.0051 and 0.0030 as model scale increases.
  • 5.2 Why Two-Point ZO Intuition Does Not Directly Transfer to ES for Reasoning: Two-point ES provides no training-reward or held-out advantage over one-point ES in the matched GSM8K experiment.Weak coupling between regenerated autoregressive responses can prevent the variance-reduction intuition from transferring from supervised tasks.

6 Conclusion and Future Work

The paper positions ES as a distinct reasoning post-training paradigm, reporting broader Pass@K coverage, sparse task-relevant updates, preserved held-out capabilities, and stable training with smaller populations for larger models.

  • Conclusion: ES improves Pass@1 while maintaining broader Pass@K coverage than GRPO, which can exhibit entropy collapse and declining Pass@K.The conclusion presents this as both a theoretical and empirical finding.
  • Conclusion: Held-out evaluations indicate that substantial ES parameter movement does not necessarily erase existing capabilities.This contrasts with previous reports attributing catastrophic forgetting to parameter drift.
  • Proof foundations: The theoretical appendix derives the verifier-projected diversity result using JS identities, Taylor expansions, and information-theoretic inequalities.The proof uses data processing, Pinsker, arithmetic–geometric mean, Taylor, and Jensen inequalities.
  • Diversity interpretation: The analysis distinguishes population diversity from within-policy sampling entropy: identical GRPO member policies have zero cross-parameter JS despite potentially nonzero sampling entropy.This clarifies why population-level diversity measures a different quantity from single-policy exploration.

A.3 Proof and Illustration of Lemma 3

The lemma’s illustration shows how success-calibrated fitness can make verifier-level population diversity exploitable while preserving total success probability under a suitable center-policy shift.

  • Illustration: Success-calibrated fitness makes success/failure predictive JS exploitable through the selected-policy shift.The construction uses a prompt-adaptive oracle gate and derives the resulting shift analytically.
  • Illustration: The center policy may use different responses while preserving total verifier success probability.This connects the analytical comparator to the actual ES center through an empirically testable condition.
  • Coverage bound: For K-sample success, f_K(p)=1−(1−p)^K is K-Lipschitz, enabling a transfer bound from success-probability divergence to coverage.The proof combines the Lipschitz property with Jensen’s inequality and a margin condition.

B Complete Majority-Vote Results

The appendix documents majority-vote evaluations corresponding to Pass@K results and reports positive held-out ES metric changes while GRPO declines on larger-K metrics.

  • Majority-vote evaluation: Maj@16 and Maj@32 results correspond to the Pass@K results in the main tables and use deterministic plurality voting after task-specific answer normalization.The appendix reports these metrics for the Easy and Hard settings.
  • Majority-vote results: The Easy Setting and Hard Setting tables report Maj@K results across target and held-out mathematical benchmarks.The table captions identify the corresponding Pass@K comparisons and the best Base/GRPO/ES results.
  • Held-out evaluation: Averaged across three models, all five ES held-out metric changes are positive, whereas GRPO decreases Pass@16 and Pass@32.Changes are averaged equally over five non-GSM8K tasks.

D Complete Results for Magnitude-Thresholded ES

Magnitude-thresholding shows that ES performance is largely preserved when smaller updates are removed, while estimator diagnostics distinguish exact estimator identity from finite-sample variance behavior.

  • Thresholding protocol: Thresholding zeros coordinates with 0 < |∆θ_i| ≤ τ, and update sparsity is computed over nonzero ES changes.Pass@1 is estimated from 32 retained responses per problem.
  • Thresholded ES results: Near 78% update sparsity, Pass@1 remains close to Full ES across all four evaluated endpoints.The changes are −0.351, −0.130, +0.488, and +0.169 percentage points for Qwen2.5-1.5B, Llama-3.2-3B, Qwen2.5-7B, and DeepSeek-R1-Distill-Qwen-1.5B, respectively.
  • Thresholded ES results: As thresholding expands, Qwen2.5-1.5B-Instruct and Llama-3.2-3B-Instruct degrade at approximately 92% sparsity, DeepSeek degrades more gradually, and Qwen2.5-7B improves across the tested range.The endpoints therefore have different robustness profiles under increasingly aggressive ablation.
  • Estimator diagnostic: Antithetic ES and two-point ZO are exactly the same estimator when objective, perturbations, scale, and paired differences match.Their finite-budget variance can still differ because two-point ZO uses two evaluations per direction rather than one.
  • Estimator diagnostic: Variance diagnostics compare regenerated-rollout, supervised cross-entropy, and fixed-rollout objectives under common or independent randomness.A ratio below one indicates variance reduction from subtraction in the diagnostic.

E.4 Diagnostic Results

The diagnostic finds that antithetic subtraction is unstable for regenerated reasoning rollouts because positive and negative rewards are weakly coupled, except at a small perturbation scale.

  • Weak positive–negative reward correlation in regenerated GSM8K rollouts makes paired subtraction increase raw scalar variance.The instability is localized to regenerating autoregressive responses with terminal rewards, rather than parameter perturbation itself.
  • At σ = 10−4, common seeds yield κpair = 0.8194, while independent seeds remain above one.At the two larger tested scales, paired subtraction does not reduce raw variance under either seed mode.
  • Near the tested base checkpoint, regenerated-rollout rewards provide weaker positive–negative coupling than fixed-batch supervised or fixed-rollout objectives.Thus, raw paired subtraction does not reliably cancel evaluation variance except at small scales.

F Population-Size Scaling Trajectories

Population-size trajectories show that larger models can optimize effectively with smaller ES populations, while the implementation uses full-parameter one-point ES with standardized rewards and fixed evaluation protocols.

  • F Population-Size Scaling Trajectories: Figure 5 displays smoothed mean-reward trajectories for N ∈{8, 16, 32, 64}, with shading representing local standard deviation of raw-minus-smoothed residuals.The vertical ranges expose within-model differences.
  • F Population-Size Scaling Trajectories: N = 8 and N = 16 decline late for Qwen2.5-0.5B-Instruct, whereas N = 32 and N = 64 continue improving.For the 1.5B and 3B models, N = 16, N = 32, and N = 64 remain substantially closer throughout training.
  • Implementation: The training and testing hardware uses NVIDIA A100-SXM4-80GB GPUs, while prompts follow each model’s official chat template and task-specific answer format.Answer extraction uses boxed answers, <answer> tags, or one Python code block depending on the task family.
  • Implementation: Full-parameter one-point ES perturbs every model parameter, evaluates verifier rewards from generated responses, restores the center, and applies reward-standardized updates.The procedure reconstructs seeded Gaussian directions and uses α as the center-update scale.
  • Implementation: Mathematical verifier rewards are 1 for correct responses, 0.1 for correctly formatted but incorrect responses, and 0 for format failures.The reported comparisons use task-specific training configurations and offline evaluations retaining 32 responses per problem at temperature 0.6, with K ∈{16, 32}.

G.5 Datasets and Setting Rationale

The paper uses easy and hard reasoning settings to assess ES across model scales and domains, while situating its evaluation and estimator choices relative to prior ES, GRPO, and zeroth-order work.

  • G.5 Datasets and Setting Rationale: The Easy Setting trains on GSM8K across three instruction-tuned model scales and evaluates held-out CommonsenseQA, HotpotQA, Countdown-Task-GOLD, GPQA, and MBPP.No held-out dataset contributes post-training rewards.
  • G.5 Datasets and Setting Rationale: The Hard Setting combines the DeepSeek-R1-Distill-Qwen-1.5B backbone with DeepScaleR mathematical RL data and aligns with representative mathematical GRPO evaluations.It is a harder post-training protocol, not a claim that every item is uniformly harder than every GSM8K item.
  • G.6 Evaluation Metrics: Pass@1 averages per-response correctness, while Pass@K uses a without-replacement estimator and Pass@32 counts problems with at least one correct response.Maj@K uses deterministic plurality voting after answer normalization and treats failed extraction as abstention.
  • Estimator Context: Antithetic ES is algebraically identical to two-point ZO under matched objectives and perturbations, but regenerated stochastic responses can weaken covariance and make subtraction unreliable.This distinguishes reasoning evaluation protocols from fixed-objective zeroth-order fine-tuning.
  • I Pass@K Profiles: ES and GRPO are compared through Pass@K profiles, with GRPO leading at K = 1 and ES overtaking it as K grows while remaining above the Base Model.Sequential composition benefits are order-dependent: ES→GRPO is strongest on AIME24, whereas GRPO→ES leads on AIME25 for K ≥2.
Loading 2608.27351v1…