Source-linked AI summary
EvE: An Alternate Optimizer to Adam
Shashank Raj, Kalyanmoy Deb
TL;DR
Adam’s long training runs make it difficult to rank many candidate configurations cheaply and prune poor ones early. EvE combines differential-evolution exploration with a targeted Adam fallback, using gradients only when a candidate fails to improve. Under matched evaluation budgets, it wins or ties Adam on most tested scalable benchmark cells and completes real neural-network tasks faster, with some final-quality trade-off.
Problem
Adam can require many epochs before its training curve is informative, making it a poor fit for cheaply ranking and pruning configurations during hyperparameter or architecture search.
Source: §1 · pp. 1–2Method
EvE uses a four-member steady-state differential-evolution population and triggers a short Adam burst for an individual only when its proposed candidate fails to improve.
Source: §1 · p. 2Results
EvE wins or matches Adam on most tested scalable benchmarks and finishes real neural-network training tasks 1.7–3.9× faster at a modest cost in final quality.
Source: §1 · p. 2, §7 · p. 10Takeaways & Limitations
EvE is a fast, gradient-aware proxy for search-heavy work before training a single model, rather than a replacement for Adam as a final-stage trainer.
Source: §1 · p. 2, §7 · p. 10On minibatch objectives, sampling noise means best-so-far values are not guaranteed to be non-increasing step to step.
Source: §4 · p. 5
Abstract
from arXiv · showhide
Adam and its variants dominate neural network training, but a single run only reveals whether a configuration works well after most of its budget is spent, a poor fit for hyperparameter or architecture search, where configurations must be ranked cheaply and pruned early. We introduce EvE (Evolutionary Explorer), a steady-state, population-of-four differential evolution (DE) optimizer with a targeted Adam fallback: each iteration proposes one candidate via DE, running a short burst of gradient descent only if the DE step fails to improve on the incumbent. Selection is greedy, so on a deterministic objective the best-so-far value is provably monotone non-increasing, and since gradients are used only as a targeted rescue, per-iteration cost stays within a constant factor of a single Adam step regardless of dimension. Under a fixed, evaluation-cost-matched budget, EvE wins or ties Adam on 76% of 70 (problem, dimension) cells across seven scalable benchmarks up to one million variables. On three real neural-network tasks (an MLP on MNIST, and LoRA fine-tuning of a 1.5B-parameter language model on two datasets) EvE finishes the same charged budget 1.7-3.9x faster, at a modest cost in final quality (about one accuracy point on MNIST, 9-11% higher relative test loss on the two fine-tuning tasks; on GSM8K, Adam is about 5 accuracy points more accurate, and fine-tuning lowers accuracy below the base model for both). Inside successive halving on UCI Adult, EvE completes hyperparameter and architecture searches 3.1-3.5x faster, ranking configurations about as consistently with Adam as Adam does with itself across seeds (Kendall's tau 0.66-0.69). EvE is not a total replacement for Adam as a final-stage trainer, but a fast, gradient-aware proxy for the search-heavy, budget-constrained regime one level up.
1 INTRODUCTION
Gradient-based optimizers are cheap and effective, but their long training curves make them poorly suited to ranking and pruning many configurations early. EvE combines population-based exploration with a targeted Adam fallback as a search proxy, not a replacement for final-stage Adam training.
1 INTRODUCTION
- Adam can require many epochs before its training curve is informative, limiting its fit for cheaply ranking and pruning configurations in hyperparameter or architecture search.Gradient-free population methods explore several regions at once, but can waste evaluations rediscovering directions gradients would reveal.Source: §1 · pp. 1–2
- EvE updates one member of a four-individual differential-evolution population per iteration and uses a short Adam burst only when the candidate fails to improve.A greedy accept-or-reject rule makes the best-so-far value non-increasing on deterministic objectives; the gradient rescue keeps per-iteration cost within a constant factor of one Adam step regardless of dimension.Source: §1 · p. 2
- Under a fixed, evaluation-cost-matched budget, EvE wins or matches Adam on most tested scalable benchmarks and finishes neural-network tasks 1.7–3.9× faster at modestly lower final quality.The paper positions EvE as a fast proxy for search-heavy work before training a single model, not as a replacement for final-stage Adam.Source: §1 · p. 2
2 RELATED WORK
Prior optimizers either refine a single point with differentiable information or explore populations without gradients. EvE combines population exploration with an individually gated gradient fallback.
Gradient-based optimization
- First-order adaptive methods and their variants refine a single point using local differentiable information, while EvE relaxes that constraint.The cited alternatives modify moment estimates, add curvature, or use information geometry.Source: §2 · p. 2
Evolutionary and population-based optimization
- Differential evolution, evolution strategies, and particle swarm optimization explore without gradients, but their sample efficiency falls as dimension grows.These population methods are used mostly for hyperparameter and architecture search rather than direct large-network training.Source: §2 · p. 2
Hybrid methods
- Unlike prior hybrids that apply a separate polishing phase to the population or its best member, EvE triggers Adam rescue per individual only when mutation fails.Thus, gradient cost scales with how often mutation stalls, not with population size.Source: §2 · p. 2
3 METHOD
EvE searches in normalized coordinates with a four-member, steady-state differential-evolution population, then conditionally applies polynomial mutation and an Adam fallback. Greedy selection compares candidates with the incumbent, using a shared minibatch for that iteration when the objective is stochastic.
3.1 SETUP
- EvE minimizes a possibly non-convex, not-everywhere-differentiable objective over bounded variables by searching the unit cube and mapping candidates affinely to real coordinates.The population has four members, keeping several candidates available for the fallback to polish rather than attempting to cover the full space.Source: §3.1 · p. 3
3.2 PROXIMITY INITIALIZATION
- The default neural-network initialization draws one uniform anchor and places three independent Gaussian jitters around it, with σ = 0.1.This creates a compact neighborhood whose DE differences and pbest pull are informative immediately; synthetic benchmarks instead use Latin Hypercube Sampling.Source: §3.2 · p. 3
3.3 STEADY-STATE DIFFERENTIAL EVOLUTION
- Each iteration challenges one slot using a rand-to-pbest/1 differential-evolution update built from three distinct donor slots.The pbest pull biases offspring toward the best known point, and the offspring is clamped to the unit cube before evaluation against the incumbent.Source: §3.3 · p. 3
3.4 POLYNOMIAL MUTATION (PM)
- On synthetic benchmarks, polynomial mutation perturbs one coordinate and replaces the DE child only if it scores strictly better.Its bounded, self-scaling perturbation avoids infeasible values, but is disabled by default for network training because a single-coordinate nudge at millions of parameters essentially never wins and adds cost without measurable benefit.Source: §3.4 · pp. 3–4
3.5 ADAM FALLBACK
- When the child fails to beat its target, EvE runs a short Adam burst in real coordinates, retaining per-slot moment state across bursts.The refined point replaces the child only if it improves on that child; the fallback can use another local gradient optimizer, though only Adam is evaluated.Source: §3.5 · p. 4
3.6 SELECTION AND COMMON RANDOM NUMBERS
- The iteration’s winning candidate replaces its target slot only when it strictly beats the incumbent.For minibatch objectives, all evaluations in an iteration, including the fallback burst, use the same newly drawn batch (common random numbers).Source: §3.6 · p. 4
4 THEORETICAL PROPERTIES
For deterministic objectives, greedy selection guarantees a non-increasing best-so-far value, while EvE’s per-iteration cost remains within a constant factor of one Adam step at any dimension. The minibatch guarantee is weaker because sampling noise can change incumbent values.
Monotone best-so-far
- For deterministic objectives, EvE’s best-so-far value never worsens because each iteration changes only one slot and accepts a candidate only when it improves that slot.The property holds regardless of the DE, polynomial mutation, or Adam-fallback settings.Source: §4 · p. 4
- With minibatch loss, best-so-far values are not literally non-increasing step to step because refreshing the challenged slot on a new batch can change its value through sampling noise.The guarantee is only that a candidate beats the incumbent on the same batch before acceptance.Source: §4 · p. 5
Per-iteration cost
- EvE’s per-iteration cost stays within a constant factor of one Adam optimizer at any dimension, assuming a gradient costs O(1) forward passes and the objective cost is Ω(n).The worst case includes the Adam fallback firing every iteration; the constant population size keeps the resulting cost in the same order as plain Adam steps.Source: §4 · p. 5
Memory cost
- EvE uses 13× the live parameter count in total storage versus 4× for plain Adam, a fixed roughly 3–4× memory multiplier.The overhead does not grow with problem size.Source: §4 · p. 5
Reproducibility
- A run is exactly reproducible from one integer seed because its two persistent random generators are not reseeded mid-run.One generator is used at initialization and the other for subsequent donor selection, jitter, and mutation seeds.Source: §4 · p. 5
5 EXPERIMENTS
Under evaluation-cost-matched budgets, EvE performs competitively on scalable synthetic benchmarks and completes neural-network training and UCI Adult searches faster, often with a modest quality trade-off.
5.1 EXPERIMENTAL SETUP
- Experiments compare methods under shared evaluation-cost budgets, converting measured backward-pass wall time into evaluation-equivalent units.Budgets were 2,000 for synthetic benchmarks, 3,000 for MLP/MNIST, and 300 for LoRA/Alpaca.Source: §5.1 · p. 5
5.2 SYNTHETIC SCALABLE BENCHMARKS
- The synthetic comparison used seven standard scalable objectives, 21 seeds per problem–dimension cell, and a budget of 2,000 evaluations.Dimensions ranged from 2 to 10^6; significance testing used paired, Holm-Bonferroni-corrected Wilcoxon signed-rank tests.Source: §5.2 · p. 6
- Across 70 synthetic benchmark cells, EvE was significantly better in 33, Adam in 17, and 20 were statistically indistinguishable.The comparison covered seven scalable problems across dimensions from 2 to 10^6.Source: §5.2 · p. 6
5.3 NEURAL NETWORK TRAINING: MLP ON MNIST
- On MLP/MNIST, EvE completed the shared budget 3.7–3.9× faster than baselines, at a cost of about one accuracy point.EvE took 49s versus 182–192s, reaching 97.0% accuracy versus the Adam family’s 98.0–98.2%.Source: §5.3 · p. 6
5.4 HYPERPARAMETER AND ARCHITECTURE SEARCH: UCI ADULT INCOME
- In both UCI Adult search spaces, EvE completed successive-halving sweeps 3.1–3.5× faster while reaching validation accuracy within about half a point of the baseline cluster.Its first-rung ranking of 48 configurations agreed with Adam about as consistently as Adam agreed with itself across seeds (Kendall’s τ 0.66–0.69 versus 0.67–0.69).Source: §5.4 · pp. 7–8
5.5 REAL-WORLD FINE-TUNING: LORA ON QWEN2.5-1.5B-INSTRUCT
- On LoRA fine-tuning of Qwen2.5-1.5B-Instruct on Alpaca, EvE finished 1.7–2.0× faster, with roughly 11% higher test loss than the Adam family.EvE took 35.0±0.6s versus 58.8–70.7s for the baselines; test loss was 1.240 ± 0.046 versus 1.115–1.124.Source: §5.5 · p. 8
6 DISCUSSION
EvE’s relative advantage depends on the task: it performs well on many multimodal synthetic problems and runs faster on real training and search tasks, with practical trade-offs and configuration requirements.
6.1 WHEN EVE HELPS AND WHEN ADAM DOES
- EvE wins or ties Adam in 53 of 70 synthetic cells, with its strongest results on multimodal problems and at extreme dimensions.Adam is strongest on Sphere and on Schwefel from n = 50; the methods are statistically indistinguishable on Rosenbrock at every dimension.Source: §6.1 · p. 8
- Across three neural-network tasks, EvE finishes the same charged budget 1.7–3.9× faster, while trailing Adam in final accuracy or loss.The reported quality gaps are about one MNIST accuracy point, 9–11% relative test loss on two LoRA tasks, and five GSM8K accuracy points; most evaluations go to population upkeep rather than gradient steps.Source: §6.1 · p. 8
- On UCI Adult, successive-halving searches reach comparable accuracy in roughly one-third of the wall-clock time of every baseline in both search spaces.Figure 2 plots wall-clock time against best validation accuracy and reports means and standard deviations over five seeds.Source: Figure 2 · p. 9
6.2 PRACTICAL CONSIDERATIONS
- EvE’s learning rate needs its own search, its four-member population has the widest seed spread on both LoRA tasks, and its population and mutation settings are fixed per task family.Its memory footprint is non-trivial: 13× the live parameter count against Adam’s 4×, with measured peak memory of 2.95×.Source: §6.2 · pp. 9–10
7 CONCLUSION
The conclusion presents EvE as a fast, gradient-aware proxy for search-heavy training, rather than a replacement for Adam as a final-stage trainer. It also emphasizes reproducibility through explicit algorithms, settings, seeds, budgets, and statistical procedures.
7 CONCLUSION
- EvE trades some final quality for faster runs, winning or tying Adam on many synthetic cases and finishing real training and UCI Adult searches faster.The conclusion frames it as a proxy for the search-heavy step before training a single model, not a final-stage replacement.Source: §7 · p. 10
- Future work includes adapting population, mutation, and initialization automatically and extending search-loop evidence across datasets.Source: §7 · p. 10
- The paper states that its executable pseudocode, explicit hyperparameters, fixed seeds, cost-accounting rule, and statistical procedures support regenerating its results.Results aggregate 21 seeds for synthetic benchmarks and five for neural-network tasks, with one labeled single-run budget-extension check.Source: REPRODUCIBILITY STATEMENT · p. 10
AI USE STATEMENT
The appendix records the paper’s limited AI-tool use and details how evaluation costs, memory, minibatches, and method settings are handled. It also explains when polynomial mutation is used and how learning-rate selection and training time are assessed.
AI USE STATEMENT
- Grammarly was used for writing corrections, and Claude Sonnet for the literature survey.Source: AI USE STATEMENT · p. 10
A.1 EVALUATION-COST ACCOUNTING
- Backward-pass time is measured on the run’s hardware and converted into forward-equivalent cost; runs stop once cumulative cost reaches the shared budget.Forward calls are charged by the number of candidates scored, or one per model-state call; each backward-pass charge is added once.Source: §A.1 · p. 14
A.2 VECTORIZED UPDATES AND MEMORY COST
- EvE’s four-slot population and persistent Adam moments require 13× parameter-and-optimizer-state storage versus Adam’s 4×, while measured peak allocated memory is 2.95× Adam’s.The MLP/MNIST measurement is below the 3.25× storage-only ratio because activations and allocator overhead do not scale with population size.Source: §A.2 · pp. 14–15, Figure 4 · p. 15
A.3 HOW THE BATCH ADVANCES
- One new minibatch is drawn at each iteration and reused for that slot’s refreshed value, candidate evaluations, and any fallback burst.The other population slots retain their most recently scored values until their own turns.Source: §A.3 · pp. 15–16
A.4 WHY PM HELPS THE BENCHMARKS BUT NOT NEURAL NETWORK TRAINING
- Polynomial mutation is retained for synthetic benchmarks but skipped in neural-network training, where a one-coordinate perturbation among millions is unlikely to beat the clean child.Its accept/reject gate prevents a mutant from worsening the child, but still incurs an extra evaluation.Source: §A.4 · p. 16
A.5 LEARNING-RATE SELECTION: TEST VERSUS VALIDATION
- Validation-based learning-rate selection leaves MNIST and Alpaca comparisons stable, but widens EvE’s GSM8K test-loss gap from about 9% to about 11%.For GSM8K, the selected rate changes from 10−2 under test selection to 10−4 under validation selection.Source: §A.5 · p. 16
A.6 TRAINING-TIME SUMMARY
- Table 3 reports wall-clock time in seconds under the shared budget, comparing EvE with the range across eight baselines.The UCI Adult entries are total times for full successive-halving sweeps.Source: §A.6 · p. 16, Table 3 · p. 17
B SYNTHETIC BENCHMARKS: PARAMETERS AND DETAILED ANALYSIS
The synthetic benchmark sweep uses benchmark-specific settings and a Latin Hypercube initialization, with a small informal check motivating that initialization. Results vary with landscape: EvE is strongest on several multimodal problems, while Adam remains competitive or wins on others.
Parameters
- The benchmark settings use EvE fallback learning rate 0.05 and seven steps, while Adam uses a fixed 0.1 rate without a per-problem sweep.The Adam rate was chosen once, so individual cells could differ under another rate; the Sphere objective is recentered so its optimum lies inside the search box.Source: B SYNTHETIC BENCHMARKS: PARAMETERS AND… · p. 17, Table 4 · p. 17
Latin Hypercube population initialization
- Latin Hypercube Sampling spreads the four initial points across the domain; an informal matched-budget spot check favored it on most tested cells, but not at full sweep scale.The benchmark uses LHS, while proximity initialization remains the neural-network default.Source: B SYNTHETIC BENCHMARKS: PARAMETERS AND… · p. 17
Per-problem analysis
- EvE wins or ties every Ackley, Griewank, and Rastrigin cell, while Adam is significantly better on every Sphere dimension.At n = 10^6, Ackley medians are 6.68 for EvE and 19.6 for Adam; Griewank medians are 3.71 × 10^6 and 1.60 × 10^7, respectively. Rosenbrock cells are statistically indistinguishable; Adam leads on Schwefel from n = 50 upward, while it diverges numerically on Zakharov at n ≥ 10^5.Source: B SYNTHETIC BENCHMARKS: PARAMETERS AND… · pp. 17–18
- Table 5’s per-problem counts classify cells as EvE-better, statistically indistinguishable, or Adam-better using paired-seed Wilcoxon tests with Holm-Bonferroni correction.The table summarizes ten dimensions per problem, with 21 seeds per cell; full min/median/max values are in Table 1.Source: Table 5 · p. 18
C MLP/MNIST: ARCHITECTURE AND HYPERPARAMETERS
The MNIST comparison fixes the model, data split, and charged budget across optimizers; on UCI Adult, EvE speeds successive-halving hyperparameter search while finding nearly comparable validation accuracy.
C MLP/MNIST: ARCHITECTURE AND HYPERPARAMETERS
- The MNIST protocol uses the same 1,863,690-weight, two-hidden-layer MLP and shared initialization, with test accuracy taken from the lowest-validation-loss checkpoint.All methods use the same 90%/10% training-validation split and stop at evaluation-cost budget B = 3,000.Source: C MLP/MNIST: ARCHITECTURE AND… · pp. 18–19
- EvE finishes the shared MNIST budget in about 49s versus 182–192s for baselines, reaching 97.0% test accuracy versus 98.0–98.2% for Adam-family methods.Its training loss sits behind the Adam family but ahead of plain SGD; its learning-rate sensitivity lies between SGD and the Adam family.Source: C MLP/MNIST: ARCHITECTURE AND… · p. 19
D UCI ADULT: HYPERPARAMETER AND ARCHITECTURE SEARCH
- Successive halving promotes the top third of configurations through five increasing cost budgets, resuming rather than restarting training at each rung.The UCI Adult benchmark is chosen as a standard, hyperparameter-sensitive AutoML setting.Source: D UCI ADULT: HYPERPARAMETER AND… · pp. 19–20
D.1 HYPERPARAMETER SEARCH: LEARNING RATE AND WEIGHT DECAY
- 3.4–3.5× faster, EvE completes the UCI Adult sweep in 21.8±0.3s versus 73.6–76.8s for eight baselines, with best validation accuracy 0.8544 ± 0.0018 versus 0.8549–0.8591.EvE was faster than every baseline on all five seeds.Source: §D.1 · p. 20, Table 8 · p. 20
D.2 ARCHITECTURE SEARCH: WIDTH AND DEPTH
On UCI Adult architecture search, EvE reaches similar validation accuracy faster while often selecting smaller, shallower networks; at the first search rung, its configuration rankings agree with Adam about as well as Adam agrees with itself across seeds.
D.2 ARCHITECTURE SEARCH: WIDTH AND DEPTH
- 3.1–3.5× faster, EvE completes the architecture sweep in 21.5 ± 0.9s versus 66.9–75.8s, reaching 0.8544 ± 0.0010 versus baselines’ 0.8533–0.8589 validation accuracy.EvE beats SGD momentum by 0.11 points and trails the other seven baselines by 0.38–0.45 points.Source: §D.2 · p. 21, Table 9 · p. 21
- Across all methods, width 512 and depth 2 are each selected in 30 of 45 winning configurations, while EvE selects each in only 2 of its 5 runs.EvE chooses width 256 in its other three runs and depth 1 in its other three, yet its accuracy remains close to the baseline cluster.Source: §D.2 · p. 21
D.3 DOES EVE RANK CONFIGURATIONS LIKE ADAM DOES?
- At the first SHA rung, EvE–Adam Kendall’s τ is 0.663 versus Adam’s cross-seed 0.694 on hyperparameter search, and 0.691 versus 0.670 on architecture search.These are mean values over five seeds for rankings of the same 48 configurations.Source: §D.3 · p. 21, Table 10 · p. 21
- The ranking agreement is imperfect and measured only on UCI Adult at the first rung; on architecture search, Adam agrees more closely with AdamW (0.848) than with EvE (0.691).EvE’s disagreement with Adam is comparable to Adam’s own run-to-run reordering, but the result does not establish later-rung agreement.Source: §D.3 · pp. 21–22
E LORA/ALPACA: ARCHITECTURE AND HYPERPARAMETERS
For LoRA fine-tuning on Alpaca, EvE spends the same charged budget faster than the baselines but ends with higher test loss. The protocol keeps the base model frozen and adapts only LoRA parameters.
E LORA/ALPACA: ARCHITECTURE AND HYPERPARAMETERS
- The experiment uses Qwen2.5-1.5B-Instruct with frozen base weights and LoRA adapters; only adapter parameters are trainable, and each method uses the same protocol.Alpaca sequences are capped at 512 tokens, and each method is tested at its own best learning rate under budget B = 300.Source: E LORA/ALPACA: ARCHITECTURE AND… · p. 22, Table 11 · p. 22
- EvE’s population is anchored at the adapter’s real starting values, preserving the LoRA zero-initialized starting point instead of overwriting it with noise.The other three population members are placed by proximity jitter around that anchor.Source: E LORA/ALPACA: ARCHITECTURE AND… · pp. 22–23
- 1.7–2.0× faster, EvE exhausts the Alpaca budget in 35.0±0.6s versus 58.8–70.7s for baselines, with median test loss 1.240 versus the Adam-family cluster’s 1.115–1.124.EvE’s loss is roughly 11% higher, and its run-to-run range is wider than any baseline’s.Source: E LORA/ALPACA: ARCHITECTURE AND… · p. 23
F.1 SAME-TASK FINE-TUNING: LORA ON GSM8K
On GSM8K, EvE shows the same speed–quality pattern under a 900-evaluation budget, but lower test loss does not translate into stronger downstream accuracy. Under flexible scoring, fine-tuning lowers accuracy below the base model for both optimizers.
F.1 SAME-TASK FINE-TUNING: LORA ON GSM8K
- The test-loss comparison evaluates fit to GSM8K’s answer distribution, while the downstream accuracy check tests whether fine-tuning improves correct problem solving.The authors report that selecting learning rates by held-out test loss rather than downstream accuracy can lower token loss without improving arithmetic.Source: §F.1 · pp. 23–25
- GSM8K uses its own training and test splits, answer-only loss masking, and a 900-evaluation budget—three times the Alpaca budget and a departure from the paper’s usual matched-budget protocol.The sequence cap is 512 tokens; model, LoRA setup, learning-rate grid, seeds, and population initialization otherwise match Alpaca.Source: §F.1 · p. 24
- At three times the per-method budget, EvE takes 100.4 ± 4.7s versus 193.8–198.7s for baselines, a 1.9–2.0× advantage, while median test loss is 0.640 versus 0.585–0.590.EvE’s median test loss is roughly 9% higher; choosing its learning rate on validation loss instead gives 0.654, about 11% higher.Source: §F.1 · p. 24
- The authors use a fixed 100-question probe before full scoring because the initial probe found every fine-tuned checkpoint below the untouched base model.Longer training to 10,000 and 30,000 evaluations did not restore probe accuracy, motivating the full test-set evaluation.Source: §F.1 · pp. 24–25
- Under flexible scoring on all 1,319 test questions, Adam reaches 59.4% versus EvE’s 54.4% at 900 evaluations, while the base model’s 73.5% exceeds every fine-tuned checkpoint by at least 14 points.At 10,000 evaluations Adam scores 58.2% versus EvE’s 53.7%; at 30,000, the difference is not significant (52.6% versus 54.4%).Source: §F.1 · p. 25, Table 12 · p. 25
- Strict scoring alone would misleadingly favor fine-tuned checkpoints because the base model rarely uses the required “#### N” answer format.Flexible scoring accepts a boxed answer or the completion’s last number when that format is absent.Source: §F.1 · p. 25