Source-linked AI summary
DART-ES: Difficulty-Aware Reweighting and Targeted Replay for Fine-Tuning LLMs with Evolution Strategies
Zhishen Sun, Hongzhan Wang, Sizhe Dang, Guang Dai, Haishan Ye
TL;DR
Small rollout groups can make problem-difficulty estimates sparse and sensitive to sampling noise, while ES population feedback offers a more finely resolved solvability signal. DART-ES uses current and historical population pass rates to guide difficulty reweighting and rare-sample replay. It improves standard ES across mathematical reasoning and other evaluated settings, with lower runtime and memory than GRPO under the evaluated settings.
Problem
Small rollout groups can make problem-difficulty estimates sparse and sensitive to sampling noise, while ES offers a more finely resolved population pass rate for estimating solvability.
Source: §1 · p. 2Method
DART-ES aggregates current and historical population pass rates into a dynamic difficulty state that guides perturbation reweighting and rare solvable sample replay.
Source: §1 · pp. 2–3Results
DART-ES consistently improves standard ES across mathematical reasoning and extends to other evaluated settings; GSM8K average accuracy is 73.53% versus ES at 72.07%.
Source: §1 · p. 3, §4.1 · p. 7, §5 · p. 9Takeaways & Limitations
Abstract
from arXiv · showhide
Evolution Strategies (ES) enable memory efficient full parameter fine-tuning of large language models (LLMs) using only forward computation. However, standard ES uniformly averages rewards across problems and compresses problem level population feedback into a single scalar, making it difficult to capture how the learning value of each problem changes with model capability. To address this limitation, we propose Difficulty-Aware Reweighting and Targeted Replay for Evolution Strategies (DART-ES). DART-ES estimates the local solvability of each problem from its pass rate across the perturbation population and aggregates historical observations to construct a dynamic difficulty state. This shared state jointly guides continuous difficulty reweighting and rare solvable sample replay, thereby improving perturbation direction evaluation and training data allocation without introducing an additional difficulty model or backpropagation. Extensive experiments show that DART-ES achieves good fine-tuning performance. DART-ES outperforms ES on all five base models and improves the average accuracy from 72.07\% to 73.53\%, exceeding the 73.26\% achieved by GRPO on GSM8K. Across five challenging mathematical reasoning benchmarks, DART-ES achieves an average accuracy of 49.20\%, compared with 48.34\% for ES and remains competitive with strong 7B models trained with RL. Further experiments show consistent gains in instruction tuning, code generation and the Countdown task with a 14B model, demonstrating strong generalization across tasks and scalability to larger models. Beyond performance gains, DART-ES also shows clear advantages in system efficiency. It reduces runtime per step by 15.2\%--50.2\% and peak memory usage per GPU by 21.1\%--51.1\% compared with GRPO. Despite performing full parameter fine-tuning, DART-ES also requires less runtime and GPU memory than GRPO+LoRA.
1 INTRODUCTION
Standard ES reduces the memory burden of LLM fine-tuning but weights problems uniformly, overlooking how their learning value changes with model capability. DART-ES uses population pass rates and historical difficulty states to guide problem weighting and replay, with reported gains in performance and efficiency.
1 INTRODUCTION
- ES uses forward passes and avoids storing activations, gradients, and optimizer states, but standard ES gives equal weight to mastered, unsolved, and sparsely solved problems.RL methods such as GRPO use backpropagation and can require substantial GPU memory and system resources.Source: §1 · pp. 1–2
- ES’s larger perturbation population provides a more finely resolved pass rate than small rollout groups, capturing both local solvability and potentially effective perturbation directions.This population signal motivates using difficulty information for both direction evaluation and training-data allocation.Source: §1 · p. 2
- DART-ES derives a dynamic difficulty state from current population pass rates and historical observations, then uses it for difficulty-aware reweighting and rare solvable sample replay.The shared state guides perturbation direction evaluation and training data selection without an additional difficulty model or backpropagation.Source: §1 · p. 2
- 73.53% average GSM8K accuracy exceeds GRPO’s 73.26%, while DART-ES averages 49.20% across five challenging math benchmarks and improves on ES across tasks and scales.The reported efficiency gains include 15.2%–50.2% less runtime per step and 21.1%–51.1% less peak GPU memory than GRPO.Source: §1 · pp. 2–3
2 RELATED WORK
Related work covers reinforcement learning with verifiable rewards, difficulty-aware data selection and weighting, and evolution strategies for LLM fine-tuning. DART-ES draws on these areas by using population feedback to address problem weighting and data allocation.
Reinforcement Learning with Verifiable Rewards
- RLVR uses task-specific verifiers to convert response checks into scalar rewards for tasks such as mathematics and code; PPO uses a clipped policy objective and value estimation, while GRPO removes the critic model.GRPO estimates advantages from the relative rewards of multiple responses to the same problem.Source: §2 · p. 3
Difficulty Aware Data Selection, Weighting, and Curriculum Learning
- Difficulty-aware reasoning post-training includes curriculum ordering, offline sample selection, online filtering, and difficulty-based adjustment of policy updates and data reuse.These approaches aim to give difficult but learnable problems greater influence during training.Source: §2 · p. 3
Evolution Strategies for Fine Tuning LLMs
- Evolution strategies optimize by sampling parameter perturbations, evaluating scalar rewards, and aggregating population directions; later work extends ES to full-parameter LLM fine-tuning and improves optimization or perturbation efficiency.Natural Evolution Strategies and OpenAI ES establish foundations in natural-gradient optimization and large-scale parallelization, respectively.Source: §2 · p. 3
3 METHOD
DART-ES supplements ES with a dynamic difficulty state derived from current and historical population pass rates, using it to reweight perturbation directions and target replay.
3.1 EVOLUTION STRATEGIES
- ES samples Gaussian parameter perturbations, evaluates the resulting population, and updates full model parameters from population rewards.Its memory-efficient implementation needs only forward generation, reward evaluation, and parameter perturbation, without gradients or optimizer states.Source: §3.1 · p. 4
3.2 DIFFICULTY-AWARE REWEIGHTING AND TARGETED REPLAY FOR EVOLUTION STRATEGIES
- DART-ES preserves ES’s perturbation and update forms while using current and historical population pass rates to guide reward reweighting and later data selection.The shared difficulty state informs both current direction evaluation and subsequent training-batch construction.Source: §3.2 · p. 4, §3.2.3 · p. 7
3.2.1 HISTORICAL DIFFICULTY ESTIMATION BASED ON POPULATION PASS RATES
- The parameter neighborhood pass rate is the proportion of perturbed models answering a problem correctly; an EMA smooths observations and tracks solvability relative to the current model.The state updates only when a problem appears in the current batch and supports both reweighting and replay.Source: §3.2.1 · pp. 4–5
3.2.2 CONTINUOUS DIFFICULTY REWEIGHTING BASED ON HISTORICAL PASS RATES
- Difficulty reweighting gives more influence to problems with lower historical pass rates, strengthening observed directions that solve them without preferring directions for problems with no current successes.DART-ES changes batch-average reward construction while retaining the base ES parameter-update form.Source: §3.2.2 · p. 6
3.2.3 RARE SOLVABLE SAMPLE REPLAY
- Replay retains problems that are not mastered but have shown successful directions, prioritizing lower historical pass rates while limiting per-epoch visits and adding cooldowns.Replay fills the target share of each batch when eligible samples are available; otherwise, fresh samples fill the remaining positions.Source: §3.2.3 · pp. 6–7
4 EXPERIMENTS
Experiments evaluate DART-ES on GSM8K and challenging math benchmarks, then across tasks and model scales and in runtime and memory usage.
4.1 FINE-TUNING RESULTS ON GSM8K
- 73.53% average accuracy on GSM8K exceeds ES (72.07%), ESSAM (73.28%), GRPO (73.26%), and GRPO+LoRA (72.42%) across five base models.Table 1 reports the comparison across base models.Source: Table 1 · p. 7, §4.1 · p. 7
4.2 FINE-TUNING RESULTS ON MATH
- 49.20% average accuracy across five challenging math benchmarks improves on ES (48.34%) and exceeds the Qwen2.5-Math-7B base model (26.54%).DART-ES was trained on MATH Level 3 to 5 data and showed stable results across benchmarks with different problem types and difficulty levels.Source: Table 2 · p. 8, §4.2 · p. 8
4.3 GENERALIZATION AND SCALABILITY PERFORMANCE
- DART-ES outperforms standard ES on all three general alignment and capability benchmarks and reaches 67.9% on 14B Countdown, versus 63.6% for ES.It also outperforms ES on LiveCodeBench v5, with 19.71% accuracy versus 18.28%.Source: Table 3 · p. 8, §4.3 · p. 9
4.4 RUNTIME AND GPU MEMORY USAGE
- 15.2%–50.2% lower mean per-step runtime and 21.1%–51.1% lower per-GPU peak memory than GRPO; DART-ES also uses less runtime and memory than GRPO+LoRA.Despite full-parameter fine-tuning, DART-ES uses less per-GPU peak memory than GRPO+LoRA on all five models.Source: Figure 2 · p. 9, §4.4 · p. 9
5 CONCLUSION
DART-ES applies a dynamic difficulty state to full-parameter LLM fine-tuning with evolution strategies, and experiments report improvements across tasks and model scales.
5 CONCLUSION
- DART-ES builds a dynamic difficulty state from current and historical population pass rates to guide difficulty reweighting and rare solvable sample replay.The shared state guides perturbation direction evaluation and training data allocation without an additional difficulty model or backpropagation.Source: §5 · p. 9
- DART-ES consistently improves standard ES on mathematical reasoning and extends to instruction tuning, code generation, and a 14B model.The conclusion also reports lower runtime per step and peak memory per GPU than GRPO and GRPO+LoRA under the evaluated settings.Source: §5 · p. 9
- Difficulty-aware population feedback provides an effective and efficient approach to LLM post-training with evolution strategies.Source: §5 · p. 10
A DETAILED ES PROCEDURE AND MEMORY EFFICIENT IMPLEMENTATION
The implementation reconstructs perturbations from random seeds and processes them layer by layer, avoiding storage of full perturbation vectors and aggregated update tensors.
A DETAILED ES PROCEDURE AND MEMORY EFFICIENT IMPLEMENTATION
- Storing one random seed and scalar reward per population member replaces storing a full parameter perturbation for each member.Resetting each seed reproduces the same layerwise perturbations during evaluation, restoration, and update.Source: A DETAILED ES PROCEDURE AND MEMORY… · p. 14
- Independent perturbed models are distributed across evaluation workers, which return batch-average rewards before the main worker updates and synchronizes parameters.The passage states this design allows population evaluation to scale with available GPUs and reduces iteration wall-clock time.Source: A DETAILED ES PROCEDURE AND MEMORY… · p. 14
- Layerwise noise is applied in place, then regenerated from the same seed to restore parameters, requiring only a temporary noise tensor the size of one layer per worker.Source: A DETAILED ES PROCEDURE AND MEMORY… · p. 14
- Population z-score normalization of batch-average rewards reduces reward-scale changes across iterations and tasks.Generation follows task-specific settings; the cited implementation uses greedy decoding to isolate parameter-space exploration.Source: A DETAILED ES PROCEDURE AND MEMORY… · p. 14
- Perturbations are regenerated seed by seed and layer by layer, then their weighted contributions are accumulated directly into the updated parameters without storing a full search direction.Source: A DETAILED ES PROCEDURE AND MEMORY… · p. 14
B EXPERIMENTAL SETUP
The experiments evaluate DART-ES across mathematical reasoning, code generation, instruction tuning, and larger-model settings, using task-specific datasets, models, baselines, and reward functions.
B EXPERIMENTAL SETUP
- Mathematical reasoning uses GSM8K and MATH; evaluation includes five challenging benchmarks, while code, instruction tuning, and Countdown use task-specific datasets and evaluations.GSM8K uses its full training set; MATH trains on Level 3–5 problems and evaluates on OlympiadBench, MATH500, Minerva, AIME2024, and AMC.Source: B EXPERIMENTAL SETUP · p. 16
- The study selects five smaller models for GSM8K, Qwen2.5-Math-7B for MATH, task-specific 1.5B models, and Qwen2.5-14B-Instruct for scalability tests.The 14B experiments use 200 Countdown training examples and 2,000 evaluation examples.Source: B EXPERIMENTAL SETUP · p. 16
- GSM8K comparisons include GRPO, GRPO+LoRA, standard ES, and ESSAM; MATH comparisons use prior 7B RL models and the best-average-performance ES checkpoint reported by Qiu et al. (2025).For instruction following, code generation, and larger-model experiments, standard ES is the main fine-tuning baseline.Source: B EXPERIMENTAL SETUP · p. 16
- Task-specific settings specify population, batch, perturbation, update, and replay parameters, with GSM8K hyperparameters varying across base models.The settings also include generation limits and, for DART-ES, EMA decay, difficulty weight, replay ratio, mastery threshold, and zero-pass patience.Source: B EXPERIMENTAL SETUP · pp. 16–17, Table 4 · p. 17
- Reward design varies by task: GSM8K and Countdown combine answer accuracy with format reward, MATH uses accuracy, code uses execution tests, and instruction tuning uses a learned reward model.For GSM8K and Countdown, difficulty weights apply only to answer correctness; the format reward remains auxiliary and unweighted.Source: B EXPERIMENTAL SETUP · p. 18
C.1 EFFECT OF DIFFICULTY-AWARE REWEIGHTING ON PERTURBATION DIRECTION EVALUATION
Difficulty-aware reweighting changes some perturbation coefficients despite strong overall correspondence with raw-reward coefficients, distinguishing directions by which problems they solve.
C.1 EFFECT OF DIFFICULTY-AWARE REWEIGHTING ON PERTURBATION DIRECTION EVALUATION
- 3.98%–5.58% of coefficients change sign across the five base models, while most reweighted coefficients remain near their raw-reward counterparts.Figure 3 compares separately standardized raw and reweighted rewards for the same perturbations; in one example, equal raw rewards yield coefficients −1.009 and +0.142 after reweighting.Source: Figure 3 · p. 20, §C.1 · pp. 20–21
- Reweighting separates directions with identical raw rewards because their correctly solved problems receive different historical difficulty weights.The example directions have equal answer accuracy, but their weighted sums over correctly solved problems differ.Source: §C.1 · pp. 20–21
- The resulting update coefficients incorporate both the number of successful solutions and the historical solvability of the corresponding problems.The comparison retains the format reward term in both scoring rules.Source: §C.1 · pp. 20–21
C.2 EFFECT OF TARGETED REPLAY ON TRAINING SAMPLE ALLOCATION
Replay visits concentrate on problems with low but nonzero current population pass rates, complementing regular traversal as training progresses.
C.2 EFFECT OF TARGETED REPLAY ON TRAINING SAMPLE ALLOCATION
- 83.94%–88.95% of replay visits have 0 < p < 0.5, compared with 4.45%–40.97% of regular traversal visits across five base models.For Qwen2.5-3B-It, the interval accounts for 86.61% of replay visits and 4.76% of regular traversal visits.Source: §C.2 · p. 21
- As regular traversal shifts toward higher-pass-rate problems during training, replay continues to direct visits toward challenging problems with observed successful directions.The historical difficulty state guides replay selection and complements difficulty-aware reweighting.Source: §C.2 · p. 21
- Problems with 0 < p < 0.5 remain challenging while some population members solve them, providing both successful and failed outcomes for direction evaluation.When p is 0 or 1, correctness outcomes are identical across the current population and cannot distinguish directions on answer correctness.Source: §C.2 · p. 21
C.3 SMOOTHING AND DYNAMIC TRACKING OF HISTORICAL DIFFICULTY STATES
The historical EMA generally predicts subsequent population pass rates better in the middle and late training stages, while smoothing can lag during early changes.
C.3 SMOOTHING AND DYNAMIC TRACKING OF HISTORICAL DIFFICULTY STATES
- EMA has higher overall MSE than the preceding pass rate for Qwen2.5-0.5B-It by 22.08% and LLaMA3.2-1B-It by 29.27%.Relative MSE exceeds 1 for some models in the early stage, consistent with a possible response lag when solvability changes rapidly.Source: §C.3 · p. 23
- 8.46%–38.10% lower MSE in the middle stage and 33.51%–39.55% lower MSE in the late stage are reported for EMA predictions across models and visit sources.Relative MSE below 1 indicates lower error for EMA than for the preceding pass rate.Source: §C.3 · pp. 22–23, Figure 5 · p. 23
- In selected improving cases, problems leave the replay buffer when historical EMA reaches the 0.5 mastery threshold; a fluctuating case re-enters after exiting.A single pass rate above the threshold does not necessarily trigger exit because the mastery criterion uses historical EMA.Source: §C.3 · pp. 23–24, Figure 6 · p. 24
- The shared historical state connects direction evaluation with replay selection, using accumulated problem-level feedback to guide both update coefficients and additional training visits.The EMA integrates population feedback for each problem over time.Source: §C.3 · pp. 21, 24
C.4 DYNAMIC MAINTENANCE AND VISIT ALLOCATION IN TARGETED REPLAY
DART-ES dynamically updates replay-buffer membership using historical pass rates and feedback from unsuccessful visits, while eligibility and visit constraints regulate replay allocation. Realized replay follows planned slots when candidates suffice and is limited by available eligible candidates otherwise.
C.4 DYNAMIC MAINTENANCE AND VISIT ALLOCATION IN TARGETED REPLAY
- Replay-buffer membership is continuously updated from historical pass rates and consecutive visits without a successful direction, subject to per-epoch replay limits and cooldowns.The number of additional visits depends on currently eligible candidates.Source: §C.4 · pp. 24–25
- Buffer sizes for Qwen2.5-0.5B-It and LLaMA3.2-1B-It fell from peaks of 6,867 and 7,012 to 2,683 and 2,061, while membership continued changing.Mastery exits made up 66.03% to 90.97% of exits; problems can enter and exit repeatedly.Source: §C.4 · pp. 24–25, Figure 7 · p. 26
- 10.19%, 8.88%, and 8.57% realized replay ratios fell below nominal ratios of 15%, 20%, and 20% when eligible candidates were often scarce.Insufficient candidates accounted for 35.25%, 62.35%, and 63.46% of iterations, respectively; replay visits equaled the smaller of planned slots and eligible candidates.Source: §C.4 · p. 25, Figure 8 · p. 27
D ABLATION STUDY
On GSM8K with Qwen2.5-0.5B-Instruct, reweighting and targeted replay each improve on standard ES, and their combination performs best. Historical difficulty estimation further improves each matched configuration and provides a shared reference for both mechanisms.
D ABLATION STUDY
- 55.34% accuracy with reweighting and targeted replay exceeded 53.44% for standard ES on GSM8K.The combined configuration also exceeded reweighting alone by 0.45 percentage points and replay alone by 0.60 percentage points.Source: D ABLATION STUDY · p. 28, Table 5 · p. 28
- Reweighting alone reached 54.89% accuracy and targeted replay alone reached 54.74%, versus 53.44% for standard ES.These gains were 1.45 and 1.30 percentage points, respectively.Source: D ABLATION STUDY · p. 28, Table 5 · p. 28
- Historical difficulty estimation raised accuracy to 55.12% with reweighting, 54.97% with replay, and 55.65% with both mechanisms.The corresponding gains over matched configurations without historical estimation were 0.23, 0.23, and 0.31 percentage points; replay feedback also updates the shared state used in reweighting.Source: D ABLATION STUDY · p. 28
E DETAILED PER-STEP RUNTIME AND GPU MEMORY RESULTS
Under the reported GSM8K configurations, DART-ES had the shortest runtime per step and lowest peak memory use per GPU across five base models. These results were measured against GRPO baselines using multiple memory-saving techniques, while DART-ES used none of those additional techniques.
E DETAILED PER-STEP RUNTIME AND GPU MEMORY RESULTS
- GRPO and GRPO+LoRA enabled several memory-saving techniques, whereas DART-ES enabled none; measurements used four 32-GiB GPUs.Runtime per step was wall-clock time for one full training iteration, and GPU memory was the mean of peak usage across the four GPUs.Source: E DETAILED PER-STEP RUNTIME AND GPU… · p. 29
- 15.2%–50.2% lower runtime per step than GRPO and 41.9%–73.9% lower than GRPO+LoRA were reported across five base models.DART-ES had the shortest per-step runtime across the evaluated model families and scales.Source: E DETAILED PER-STEP RUNTIME AND GPU… · p. 29, Table 6 · p. 29
- 21.1%–51.1% lower peak memory usage per GPU than GRPO and 7.1%–40.8% lower than GRPO+LoRA were reported across the evaluated models.DART-ES performed full-parameter fine-tuning and had the lowest measured peak memory use at every evaluated scale.Source: E DETAILED PER-STEP RUNTIME AND GPU… · p. 29, Table 7 · p. 29