PaperlayerView exact revision on arXiv

Source-linked AI summary

Evolution Strategies at Scale: LLM Fine-Tuning Beyond Reinforcement Learning

Xin Qiu, Yulu Gan, Conor F. Hayes, Qiyao Liang, Yinggan Xu, Roberto Dailey, Elliot Meyerson, Babak Hodjat, Risto Miikkulainen

arXiv:2509.24372v3cs.LGcs.AIcs.NE

TL;DR

LLM fine-tuning commonly relies on RL, but ES has been considered unscalable to billion-parameter models and difficult under long-horizon rewards. This paper directly optimizes full LLM parameter spaces with ES and finds strong, stable performance relative to RL across models and tasks, while identifying a scope boundary in dataset-dependent comparisons.

  • Problem

    RL dominates LLM fine-tuning, but long-horizon rewards, base-model sensitivity, and the perceived difficulty of full-parameter exploration leave ES scalability insufficiently established.

  • Method

    The paper applies a memory-efficient, GPU-parallel ES implementation to direct full-parameter fine-tuning of multi-billion-parameter LLMs without dimensionality reduction.

  • Results

    ES consistently surpasses standard RL-based fine-tuning across tested models, with stronger Countdown performance, stable improvements across base LLMs, lower hyperparameter sensitivity, and less reward hacking in conciseness tuning.

  • Takeaways & Limitations

    ES is a viable inference-only, backpropagation-free post-training paradigm that expands LLM fine-tuning beyond gradient-based RL.

  • Takeaways & Limitations

    Performance comparisons on the math reasoning task should be interpreted in light of both algorithmic and custom-dataset differences.

Abstract

from arXiv · show

Fine-tuning large language models (LLMs) for downstream tasks is an essential stage of modern AI deployment. Reinforcement learning (RL) has emerged as the dominant fine-tuning paradigm, underpinning many state-of-the-art LLMs. In contrast, evolution strategies (ES) has largely been overlooked due to the widespread belief that it does not scale to modern model sizes. This paper overturns this assumption by demonstrating the first successful application of ES to full-parameter fine-tuning of LLMs at the billion-parameter scale, without dimensionality reduction. ES can indeed search over extremely high-dimensional parameter spaces and outperform established RL implementations across multiple axes, including improved tolerance to long-horizon and delayed rewards, robustness across diverse base LLMs, reduced susceptibility to reward hacking, and improved training stability. These findings suggest that ES is not merely a viable alternative to RL, but a fundamentally different and powerful backpropagation-free post-training paradigm that opens a new direction for LLM fine-tuning beyond current RL-based approaches.

1. Introduction

RL dominates LLM fine-tuning but faces long-horizon reward, base-model sensitivity, and scalability concerns. This paper addresses the belief that ES cannot optimize billion-parameter LLMs by applying full-parameter, parallelized ES and comparing it with RL.

  • Motivation: RL is the predominant LLM fine-tuning paradigm, but long-horizon outcome rewards produce low sample efficiency and high-variance gradient estimates, while token-level credit assignment is difficult.RL performance is also described as sensitive to the choice of base LLMs.
  • Challenge: Direct ES optimization appears infeasible for modern LLMs because prior full-parameter applications used much smaller models and parameter-space exploration was assumed less efficient than action-space exploration.Existing workarounds restrict ES to final layers, low-dimensional adapters, or action-space search.
  • Contribution: ES directly fine-tunes the full parameter space of multi-billion-parameter LLMs without dimensionality reduction, using a memory-efficient implementation parallelized across GPUs.The study evaluates ES against state-of-the-art RL methods across several reasoning tasks and LLM families.
  • Results: ES outperforms RL across multiple dimensions, including sparse long-horizon rewards, small-population optimization, and inference-only memory savings.On Countdown, ES obtains significantly better fine-tuned models under response-level rewards, while using a population of 30 and avoiding gradient calculations.
  • Implication: The paper argues that ES is a viable post-training paradigm that expands LLM fine-tuning beyond gradient-based RL through outcome-only optimization and distributed parameter-space exploration.The conclusion frames this as a first milestone rather than a final limit on scale.

2. Related Work

Related work positions ES as a population-based alternative to RL fine-tuning and reviews prior evolutionary optimization of models at much smaller scales. It also distinguishes parameter-space exploration from the action-space exploration used by most LLM RL methods.

  • Evolution and LLMs: Evolutionary algorithms have also been applied to prompt optimization, LLM evolutionary operators, and model merging, alongside broader evolutionary optimization research.These directions differ from directly fine-tuning all LLM parameters with ES.
  • RL for fine-tuning: PPO and GRPO are predominant RL fine-tuning methods, but RL faces long-horizon reward, base-model sensitivity, reward hacking, and run-instability challenges.PPO uses a clipped surrogate objective and typically a value model, whereas GRPO replaces the value model with group advantages from multiple responses.
  • Traditional ES: Prior ES applications generally optimized spaces containing no more than millions of parameters, leaving direct billion-parameter LLM optimization largely unexplored.Related work includes ES for decision transformers up to around 2.5 million parameters and evolutionary algorithms extended to high-dimensional spaces.
  • Parameter-space exploration: Parameter-space exploration injects one perturbation across an action trajectory, whereas action-space exploration perturbs decisions at each step; prior parameter-space studies tested models far smaller than LLMs.The reviewed literature reports promising sparse-reward results but limited scale relative to modern LLMs.

3. Method

The method samples Gaussian perturbations of a pretrained LLM, evaluates each perturbed model with a reward function, and updates parameters using normalized reward-weighted perturbations. Its implementation reduces memory use and distributes evaluations while retaining direct full-parameter optimization.

  • Algorithm design: The method is based on a simplified Natural Evolution Strategies variant with fixed-covariance perturbation noise, following the design of OpenAI ES.The paper incorporates several scalability modifications beyond the basic algorithm.
  • Basic ES algorithm: Each ES iteration samples N Gaussian-perturbed models, evaluates their rewards, normalizes those rewards, and updates parameters using the weighted perturbations.Noise is independent across parameter dimensions and scaled by σ; the update uses learning rate α.
  • Implementation details: The implementation parallelizes perturbed-model evaluations across processes and reconstructs noise from stored random seeds instead of storing full perturbation tensors.Layer-level in-place perturbation and restoration limits temporary additional memory to a tensor the size of one layer per evaluation process.
  • Implementation details: Reward normalization uses within-iteration z-scores so reward scales have mean 0 and standard deviation 1 across iterations and tasks.The implementation also absorbs σ into the learning rate α to simplify computation and parameter setup.
  • Algorithm design: The study intentionally omits rank transformation, mirrored sampling, weight decay, virtual batch normalization, and Adam to isolate the core ES algorithm.This design tests whether strong performance can be obtained without auxiliary enhancements.

4. Empirical Studies

Across Countdown, conciseness, math reasoning, and puzzle tasks, ES showed strong, stable fine-tuning performance and competitive or better results than RL baselines. These experiments also indicate distinct behavioral advantages, including better reward tradeoffs, reduced reward hacking, and greater run-to-run reliability.

  • 4.1. Performance in the Countdown task: ES substantially improved over PPO, GRPO, and Dr.GRPO across Qwen2.5 and Llama3 models, scaling across model types and sizes.The comparison included Qwen2.5 models from 0.5B–7B and Llama3 models from 1B–8B.
  • 4.1. Performance in the Countdown task: ES produced higher conciseness rewards at lower KL divergence than GRPO, forming a dominant Pareto front without a KL penalty.GRPO required augmenting the reward with a KL-divergence penalty to obtain its reported tradeoffs.
  • 4.2. Behavioral differences between ES and RL in fine-tuning for conciseness: ES avoided reward hacking while matching GRPO’s reward under low KL penalties, whereas GRPO sometimes produced short responses containing nonsensical symbols.Higher GRPO KL penalties prevented hacking but required problem-specific tuning and extensive search.
  • 4.1. Performance in the Countdown task: ES was more reliable across four runs, with GRPO showing 15.5× higher reward standard deviation.The analysis used configurations exceeding a mean reward cutoff of > 0.85.
  • 4.3. ES applied to Math reasoning tasks: On math benchmarks, ES improved the base model and achieved competitive performance against PPO, GRPO, and Dr.GRPO implementations.The comparison favored highly refined RL baselines, while ES used a vanilla implementation with simple hyperparameters.
  • 4.4. Solving challenging puzzle problems: ES significantly improved performance on ARC-AGI and Sudoku relative to the base models, extending evaluation beyond reasoning benchmarks.The base LLMs failed severely on both puzzle problems before ES fine-tuning.

5. Discussion

The discussion contrasts ES with RL in optimization behavior and systems requirements, emphasizing ES’s stability, simplicity, parallelizability, and reduced computational overhead.

  • Algorithmic advantage of ES vs. RL: Parameter-space exploration gives ES lower-variance rollouts and more stable gradient estimates than token-level action-space exploration in RL.RL’s step-wise noise can increase reward-hacking opportunities, including sampling a nonsensical symbol that exploits a conciseness reward.
  • Algorithmic advantage of ES vs. RL: ES optimizes a solution distribution rather than a single solution, making isolated reward hacks less likely to generalize and improving robustness to parameter perturbations.
  • Algorithmic advantage of ES vs. RL: ES uses a simple fixed hyperparameter set across tasks and base models, whereas effective GRPO implementations depend on numerous empirically developed design choices.The discussion specifically cites length normalization, clipping, and KL-penalty placement as consequential RL choices.
  • Engineering benefits of ES vs. RL: ES parallelizes with lower systems overhead because machines exchange random seeds and scalar rewards rather than maintaining complex asynchronous actor-learner communication.This can reduce wall-clock time or support larger populations as GPU availability grows.
  • Engineering benefits of ES vs. RL: ES avoids gradient computation and synchronization, reducing memory and engineering overhead while allowing each population member to use large batch sizes.The discussion also reports substantially lower FLOP cost than RL methods using backpropagation and additional reference models.
  • Engineering benefits of ES vs. RL: Because ES only evaluates perturbed models, inference-oriented kernels for repeated forward passes, large batches, and parameter perturbations may be specialized for future fine-tuning.

6. Future Work

Future work examines why ES remains effective at billion-parameter scale, how its parameter-space smoothing differs mechanistically from RL, and how ES and RL might be combined.

  • Future directions: A population of 30 effectively optimized billions of parameters, motivating analysis of how such small populations work and whether LLMs’ low intrinsic dimensionality contributes.
  • Mechanistic understanding: The authors hypothesize that RL’s sequence-level gradient estimates become high-variance on long sequences and sharp policies, increasing susceptibility to reward hacking.
  • Mechanistic understanding: ES injects Gaussian noise in parameter space to smooth jagged reward landscapes, potentially producing more stable, efficient, and robust optimization as models and policy sharpness increase.The authors explicitly describe this mechanism as a plausible hypothesis requiring direct evidence.
  • Combining ES and RL: ES and RL could be alternated, using ES for broader exploration and RL gradients for finer local search.

7. Conclusion

The conclusion presents ES as a scalable, robust, and general alternative to RL for full-parameter LLM fine-tuning without dimensionality reduction.

  • Conclusion: ES fine-tunes billion-parameter LLMs directly in the full parameter space and consistently surpasses standard RL-based fine-tuning across the reported evaluations.The conclusion highlights Countdown, math reasoning benchmarks, puzzle problems, conciseness, stability, and reduced reward hacking.

Impact Statement

The impact statement identifies easier fine-tuning and reduced reward hacking as potential benefits of ES, especially for non-expert developers and ethical behavior.

  • Ease of use: ES lowers the barrier to LLM fine-tuning because experimenters assign model-level scores rather than designing nuanced gradient-based reward functions.The paper connects this simplification to customized AI applications by non-experts.
  • Reward-hacking: ES is described as less susceptible to reward hacking, potentially helping preserve base-model ethical guardrails and reducing overfitting to specific training examples.
  • Broader impact: Combining easier use with reduced reward hacking may lower the risk of unintended ethical misbehavior when non-experts fine-tune LLMs.

A.1. Experimental Setup

The experiments compare ES and RL across reasoning and conciseness fine-tuning settings, using controlled evaluation protocols and task-specific datasets. ES uses fixed population-based hyperparameters, while RL baselines receive separate sweeps because of greater sensitivity.

  • Countdown experiments: Countdown experiments compare Qwen2.5 models from 0.5B–7B and Llama3 models from 1B–8B, with PPO-z and GRPO-z baselines tuned through hyperparameter sweeps.The setups use TinyZero for PPO-z and two group sizes for GRPO-z.
  • Countdown experiments: ES uses population size N = 30, noise scale σ = 0.001, and learning rate α = 5 × 10−4 across experiments, with matched total sample evaluations against RL.ES reports test performance after 500 iterations, using 200 training samples and 2000 testing samples.
  • Conciseness experiments: Conciseness fine-tuning uses Qwen-2.5-7B-Instruct, repeated runs, and checkpoints every 200 iterations for both ES and GRPO.The fine-tuning dataset contains only two prompt-solution examples, while evaluation uses additional examples.
  • Conciseness experiments: The conciseness reward is the negative absolute difference between generated-response length and the verified solution length.This directly rewards responses whose string length matches the shortest correct answer.
  • Conciseness experiments: Behavior is evaluated by mean conciseness reward and mean KL divergence from the base model, with lower KL indicating greater preservation of base-model behavior.KL divergence also reflects the extent of fine-tuning changes.
  • Implementation detail: The ES implementation uses partially correlated layer-wise noise, whose preliminary performance was not significantly different from true i.i.d. noise.The random-number generator is reinitialized for each layer using the perturbed model’s seed.

A.2. ES Hyperparameter Ablations

The ablations examine how ES population size, noise scale, and learning rate affect GSM8K performance. Larger populations generally help modestly, while N = 30 offers a practical performance–cost trade-off and ES remains relatively robust across hyperparameter settings.

  • Evaluation setup: The ablations use 512 fixed training prompts and evaluate GSM8K pass@1 with greedy decoding after 300 training steps.ES samples at most 512 tokens per prompt.
  • Population size: N = 30 emerged as a reasonable trade-off between GSM8K performance and computational cost across Qwen2.5-1.5B and Qwen2.5-3B.Larger populations generally improved pass@1, but the differences were modest.
  • Population size: Larger population sizes generally yielded better pass@1 performance across both Qwen2.5 models, with modest differences between settings.The comparison covers N ∈ {16, 30, 64, 128}.
  • Noise scale and learning rate: Noise scale σ and learning rate α had a moderate effect on final GSM8K performance for both Qwen2.5-1.5B and Qwen2.5-3B.The ablations evaluate final pass@1 across combinations of σ and α.

A.3. Examples of Countdown Task Outputs

The Countdown task asks models to construct arithmetic expressions that match target values using supplied numbers and basic operations. Example outputs illustrate the required reasoning format and show a fine-tuned model checking and correcting its formulas.

  • Task definition: Countdown tests constrained symbolic reasoning by requiring arithmetic expressions that use given numbers and basic operations to reach a target value.For example, 100 × (6 + 3) + 50 = 950 solves one instance.
  • Output format: The prompt requires hidden reasoning in <think> tags and a final equation in <answer> tags, using each supplied number once.The example asks for an equation equal to 17 from [49, 41, 73].
  • Fine-tuned output: After 400 ES iterations, a Qwen2.5-3B model adjusted formulas by comparing generated answers with correct ones and verifying them.The example is presented as an output from the Countdown task.

A.4. Additional Results for the Countdown Experiments

Additional Countdown experiments show that ES is more stable and consistently stronger than RL across learning settings, model families, and sizes. The results also examine reward trade-offs, checkpointing, computational cost, parameter shifts, and transfer to Sudoku.

  • Run-to-run stability: ES had the lowest variance across three Countdown runs, with statistically significant differences from GRPO variants at p < 0.05.The comparison used three independent random seeds and the same experimental setup across ES and GRPO variants.
  • Learning-rate analysis: The ES Pareto front dominated GRPO across learning rates and β values, while most GRPO Pareto-optimal models stayed below 0.8 mean reward and 0.4 KL divergence.Higher GRPO learning rates increased both mean reward and KL divergence, creating a reward–divergence trade-off.
  • Reward distributions: On Qwen2.5-7B-Instruct, ES shifted reward distributions toward higher rewards, whereas GRPO remained near the base distribution unless using high learning rates and low β.Those aggressive GRPO settings were described as often breaking model performance.
  • Training curves and accuracy: ES consistently outperformed RL across all tested models throughout Countdown training and delivered the largest improvements across model families.These comparisons cover PPO, GRPO, Dr.GRPO, and ES relative to their respective base models.
  • Additional analyses: ES also improved Sudoku performance despite severe base-model failures, while parameter changes in most models remained close to a random walk.The largest Qwen2.5-7B-Instruct model showed systematic small-magnitude edits during conciseness fine-tuning, and ES used MATH500 to select a pseudo-validation checkpoint.
Loading 2509.24372v3…