PaperlayerView exact revision on arXiv

Source-linked AI summary

Can LLMs Beat Classical Hyperparameter Optimization Algorithms? A Study on autoresearch

Fabio Ferreira, Lucca Wobbe, Arjun Krishnakumar, Frank Hutter, Arber Zela

arXiv:2603.24647v5cs.LGstat.ML

TL;DR

The paper asks whether LLMs can replace classical HPO for tuning a small language model under a fixed compute budget. It benchmarks classical, LLM-based, and hybrid methods, introducing Centaur to share CMA-ES state with an LLM. Classical methods outperform pure LLM agents in fixed spaces, while Centaur achieves the strongest results and works with a 0.8B LLM.

  • Problem

    The paper studies whether LLM-based HPO can outperform classical optimizers on autoresearch under a fixed compute budget.

  • Method

    The study benchmarks nine classical, LLM-based, and hybrid HPO methods, including Centaur, which shares CMA-ES’s internal state with an LLM.

  • Results

    Centaur achieves the best result in the experiments, while classical methods outperform pure LLM agents in fixed search spaces and direct code editing only narrows the gap.

  • Takeaways & Limitations

    LLMs are most effective as complements to classical optimizers rather than replacements, with a 0.8B LLM sufficient for hybrid optimization.

  • Takeaways & Limitations

    The study evaluates a single task and a limited set of open-weight and frontier models, so the generality of its findings remains uncertain.

Abstract

from arXiv · show

The autoresearch repository enables an LLM agent to optimize hyperparameters by editing training code directly. We use it as a testbed to compare classical HPO algorithms against LLM-based methods on tuning the hyperparameters of a small language model under a fixed compute budget. When defining a fixed search space over autoresearch, classical methods such as CMA-ES and TPE consistently outperform LLM-based agents, where avoiding out-of-memory failures matters more than search diversity. Allowing the LLM to directly edit source code narrows the gap to the classical methods but does not close it, even with frontier models available at the time of writing such as Claude Opus 4.6 and Gemini 3.1 Pro Preview. We observe that LLMs struggle to track optimization state across trials. In contrast, classical methods lack the domain knowledge of LLMs. To combine the strengths of both, we introduce Centaur, a hybrid that shares CMA-ES's interpretable internal state, including mean vector, step-size, and covariance matrix, with an LLM. Centaur achieves the best result in our experiments, and a 0.8B LLM already suffices to outperform all classical and pure LLM methods. Unconstrained code editing requires larger models to be competitive with classical methods. We further analyze search diversity, model scaling from 0.8B to frontier models, and ablate the fraction of LLM-proposed trials in Centaur. All in all, our results suggest that LLMs are most effective as a complement to classical optimizers, not as a replacement. Code is available at https://github.com/ferreirafabio/autoresearch-automl & interactive demo at https://ferreirafabio.github.io/autoresearch-automl.

1 Introduction

This study tests whether classical HPO or LLM-based agents can optimize autoresearch under matched compute budgets. It finds that classical methods generally lead, while Centaur combines classical optimization state with LLM reasoning.

  • Scope: The paper benchmarks classical, LLM-based, and hybrid HPO methods on autoresearch under identical 24-hour budgets with three seeds.The benchmark supports both fixed hyperparameter selection and direct code-editing optimization.
  • Main findings: Classical methods outperform LLM agents in a fixed search space, while direct code editing narrows but does not close the gap, even for frontier models.Karpathy Agent (Code) is competitive with classical methods, but classical methods find a similar configuration about four times faster.
  • Centaur: Centaur shares CMA-ES’s mean vector, step-size, and covariance matrix with an LLM to combine landscape learning with transformer-training knowledge.The design addresses the complementary strengths and weaknesses of classical optimizers and LLMs.
  • Analyses: The study analyzes search diversity, out-of-memory rates, model scaling, and the fraction of LLM-proposed trials.These analyses support the comparison between pure LLM, classical, and hybrid optimization.

2 Related Work

Prior HPO work spans classical optimizers and newer LLM-assisted pipelines. This paper positions Centaur as distinct because it shares a classical optimizer’s full internal state with an LLM.

  • Classical HPO: Classical HPO includes random search, Bayesian optimization, SMAC, TPE, and evolution strategies such as CMA-ES.The study focuses on single-task methods without multi-fidelity or transfer to isolate learning the optimization landscape from scratch.
  • LLM-based HPO: Recent methods use LLMs as surrogate models, samplers, direct hyperparameter suggesters, or components of evolutionary and bandit-based optimizers.Examples include LLAMBO, SLLMBO, LLaMA-ES, and LLM-guided regional search.
  • Positioning: Centaur differs from prior methods by sharing the classical optimizer’s full internal state with the LLM.The paper identifies this explicit state sharing as its key idea.

3 Experimental Setup

The experiments tune a small transformer on nanochat under matched hardware and time constraints. Hyperparameters are extracted automatically from the training script, while their ranges remain manually specified.

  • Task and hardware: Each trial trains a small decoder-only transformer on FineWeb for five minutes using one NVIDIA H200 GPU, optimizing validation bits-per-byte.The optimizee is evaluated on the nanochat task.
  • Evaluation protocol: All methods run for 24 hours with three seeds, and failed out-of-memory trials receive val_bpb = 100.0.Fast failures can produce more trials, while the finite penalty remains compatible with surrogate models.
  • LLM infrastructure: The study uses Qwen3.5 models at 0.8B and 27B parameters, plus frontier-model experiments through the Gemini API.Training VRAM is capped at 80 GB, with remaining memory reserved for self-hosted LLM inference.
  • Search space: AST parsing extracts 14 hyperparameters from top-level ALL_CAPS literal assignments in train.py.The extraction automates hyperparameter selection, but the search ranges still require domain knowledge and manual specification.

4 Centaur: CMA-ES Guided LLM Optimization

Centaur combines CMA-ES proposals with selective LLM intervention while exposing CMA-ES’s interpretable state. Its design uses classical search to maintain optimization structure and LLM reasoning to modify selected candidates.

  • Algorithm: Centaur uses CMA-ES on every trial and invokes the LLM on 30% of trials with the optimizer’s proposal and internal state.The LLM receives the mean vector µ, step-size σ, covariance matrix C, top configurations, and recent trial history.
  • Algorithm: On LLM turns, Centaur passes µ, σ, C, and optimization history to the LLM, which may override CMA-ES’s proposed configuration.The LLM overrides the proposal in nearly all observed LLM turns.
  • Design rationale: CMA-ES is selected because its mean, scalar step-size, and labeled covariance matrix are interpretable to an LLM.The paper contrasts this with TPE density estimators and Gaussian-process posteriors, which are harder to summarize in natural language.

5 Results

Across fixed-space, unconstrained, and hybrid settings, classical methods generally outperform pure LLM agents, while Centaur achieves the strongest overall results by combining CMA-ES state with LLM suggestions.

  • 5.1 Classical methods outperform LLMs in fixed search spaces: Within the fixed search space, classical HPO methods consistently outperform pure LLM approaches, led by Centaur (0.9763) and TPE (0.9768).Several pure LLM methods perform worse than random search, whereas direct code editing is more competitive.
  • 5.1 Classical methods outperform LLMs in fixed search spaces: OOM avoidance matters more than search diversity: top methods have OOM rates at or below 16%, while the bottom four exceed 36%.LLAMBO (Paper) is highly diverse but reaches a 48% OOM rate, whereas TPE and CMA-ES maintain rates of 11% and 16%.
  • 5.2 Unconstrained code editing is viable but requires model scale: Direct code editing makes Karpathy Agent competitive with classical methods, but it remains slower, with classical methods finding similar configurations approximately 4× faster.It is the only pure LLM method described as competitive with classical approaches.
  • 5.2 Unconstrained code editing is viable but requires model scale: Scaling from 0.8B to 27B is essential for unconstrained code editing but provides no fixed-HP advantage, with Karpathy Agent Code improving from 0.9910 to 0.9814.The fixed-HP comparison changes only from 0.9904 to 0.9908.
  • 5.3 Hybrid optimization: best of both worlds: Centaur outperforms CMA-ES while using the LLM on only 30% of trials, reducing cross-seed standard deviation from 0.0036 to 0.0005.LLM trials contribute 25% of incumbent improvements, and Centaur [0.8B] already outperforms all classical and pure LLM methods.
  • 5.3 Hybrid optimization: best of both worlds: Centaur achieves the best reported result, with Opus 4.6 reaching 0.9739±0.0012 and establishing a new benchmark leader.The gain over open-weight variants is attributed to higher-quality suggestions during the LLM’s 30% of turns, not better OOM avoidance or more trials.

6 Conclusion

Across identical-budget experiments, Centaur achieved the best result, while classical methods generally outperformed pure LLM agents and code editing only narrowed the gap. The study’s findings are limited by evaluation on a single task and a small set of models.

  • Centaur achieved the best result, with a 0.8B LLM already outperforming all classical and pure LLM methods.Centaur used the LLM on only 30% of trials.
  • Within a fixed search space, classical methods consistently outperformed LLM agents, while direct code editing narrowed but did not close the gap.The comparison was conducted under identical budgets.
  • The study evaluated a single task with Qwen3.5 0.8B and 27B, Gemini 3.1 Pro Preview, and Claude Opus 4.6.The authors state that more benchmarking is needed to determine generality.

A.1 Convergence by Trial Number

The appendix reports convergence by trial number to assess sample efficiency rather than wall-clock cost. These curves can differ because LLM methods spend additional time on inference between trials.

  • Trial-number and wall-time views can differ substantially because LLM inference adds time between trials.A method can therefore appear competitive per trial while its wall-time curve is compressed.
  • Trial-number convergence measures sample efficiency rather than wall-clock cost, using the same methods as Figure 1.Figure 4 reports mean ± std across 3 seeds.
  • Figure 5 compares 0.8B and 27B methods by trial number using mean ± std across 3 seeds.

A.2 Per-Seed Convergence

The per-seed convergence plots replace standard-deviation bands with individual seed trajectories overlaid on the mean. This exposes cross-seed variance and whether average behavior depends on an outlier.

  • Per-seed plots show individual trajectories and the mean, revealing cross-seed variance and possible outlier-driven averages.Thin transparent lines represent seeds, while thick solid lines represent the mean.
  • Figure 6 shows per-seed convergence for all 27B methods against wall-time.
  • Figure 7 compares per-seed wall-time convergence for the 0.8B and 27B methods.

A.3 Frontier Model Comparison

The frontier-model experiments compare Gemini variants with Qwen3.5-27B under the same optimization framework. Neither Flash variant outperformed Qwen3.5-27B, while Gemini 3.1 Pro Preview was competitive but did not improve the mean.

  • Gemini 2.5 Flash and Gemini 3.1 Flash-Lite runs stopped after 16–18 hours because no meaningful improvement was observed beyond that point.Gemini 3.1 Pro Preview ran for the full 24-hour budget with 3 seeds.
  • Karpathy Agent with Gemini 3.1 Flash-Lite had extremely high failure rates of 87–94%, mostly runtime errors rather than OOM.The passage reports that 93% of failures were runtime errors.
  • Gemini 3.1 Pro Preview was competitive with Qwen3.5-27B but did not improve the mean for code editing or Centaur.For code editing, the reported values were 0.9826±0.0004 versus 0.9814±0.0046; for Centaur, 0.9767±0.0013 versus 0.9763±0.0005.
  • Neither Gemini 2.5 Flash nor Gemini 3.1 Flash-Lite outperformed Qwen3.5-27B.

A.4 Centaur LLM Ratio Ablation

Centaur performs best when the LLM contributes selectively rather than dominating the optimization trajectory. The default r=0.3 allocation balances CMA-ES control with occasional domain-informed LLM suggestions.

  • A.4 Centaur LLM Ratio Ablation: At r=0.3, Centaur balances classical and LLM contributions, while higher LLM ratios degrade performance, especially for the 27B model at r=0.8.The r=0.8 configuration performs worse than CMA-ES alone.
  • A.5 Incumbent Traces: Centaur’s traces are evaluated through incumbent trajectories, where colored points mark new best results and the staircase shows best-so-far performance.The traces plot incumbent behavior against cumulative wall-time and show the best seed for each method.
  • A.6 Qualitative Agent Behavior: Centaur improves over CMA-ES alone, with LLM suggestions contributing 6 of 24 incumbent improvements in seed 0.LLM trials comprised 88 of 275 total trials, close to the 30% target ratio.
  • A.6 Qualitative Agent Behavior: The LLM’s overrides added transformer-specific and hardware-aware choices, including all-short attention, power-of-2 batch sizes, and a higher learning rate.In one incumbent-producing trial, the LLM changed LLLL attention to SSSS, batch size 61 to 64, total batch size 133143 to 131072, and learning rate 0.208 to 0.3.
Loading 2603.24647v5…