PaperlayerView exact revision on arXiv

Source-linked AI summary

Using Large Language Models for Hyperparameter Optimization

Michael R. Zhang, Nishkrit Desai, Juhan Bae, Jonathan Lorraine, Jimmy Ba

arXiv:2312.04528v2cs.LGcs.AI

TL;DR

Hyperparameter choices shape model complexity and generalization, while conventional automated optimization includes black-box methods such as random and Bayesian search. This paper prompts LLMs to iteratively propose hyperparameters and training code; on standard benchmarks, LLMs improve on traditional tuning with small search budgets, while code generation beats random search and is comparable to LLM search with human-specified configurations.

  • Problem

    Hyperparameters influence model inductive bias and generalization, motivating automated optimization, while this paper investigates whether LLMs can optimize them.

    Source: §1 · p. 1

  • Method

    LLMs receive dataset, model, and hyperparameter descriptions, propose configurations, and use recorded performance to suggest subsequent configurations; they can also generate training code.

    Source: §1 · pp. 1–2

  • Results

    With small search budgets, LLMs improved upon traditional hyperparameter tuning; code generation performed better than random search and comparably to LLM search with human-specified configurations.

    Source: §1 · pp. 1–2

  • Takeaways & Limitations

    LLM-generated training code can reduce the initial search for configuration spaces unlikely to succeed and avoids requiring human specification of hyperparameters and search spaces.

    Source: §1 · p. 2

    Full-scale GPT-4 HPO can be costly: 32 HPOBench tasks with trajectory 10 using chain-of-thought cost approximately $8 as of March 2024.

    Source: §5 · p. 10

Abstract

from arXiv · show

This paper explores the use of foundational large language models (LLMs) in hyperparameter optimization (HPO). Hyperparameters are critical in determining the effectiveness of machine learning models, yet their optimization often relies on manual approaches in limited-budget settings. By prompting LLMs with dataset and model descriptions, we develop a methodology where LLMs suggest hyperparameter configurations, which are iteratively refined based on model performance. Our empirical evaluations on standard benchmarks reveal that within constrained search budgets, LLMs can match or outperform traditional HPO methods like Bayesian optimization across different models on standard benchmarks. Furthermore, we propose to treat the code specifying our model as a hyperparameter, which the LLM outputs and affords greater flexibility than existing HPO approaches.

1 Introduction

The introduction frames hyperparameter optimization as difficult when budgets are constrained or practitioners lack expertise, and proposes iterative LLM-guided tuning. Experiments report improvements over traditional tuning in small-budget settings and extend the approach to code generation.

1 Introduction

  • Existing HPO methods leave practitioners to design search spaces and typically struggle during the initial search phase, while tuning can be challenging with constrained budgets or limited expertise.Hyperparameters influence model inductive bias and generalization.Source: §1 · p. 1
  • The paper proposes prompting LLMs with dataset, model, and hyperparameter descriptions, then iteratively using model performance to request further configurations.The proposed approach was effective at proposing hyperparameters during initial search phases.Source: §1 · p. 1
  • With small search budgets, LLMs improved upon traditional hyperparameter tuning, and they performed well over evaluation horizons of up to 100 evaluations.The experiments also found LLMs significantly faster than random search for Vision Transformers and ResNets on CIFAR-10, including with generic architecture-agnostic prompts.Source: §1 · pp. 1–2
  • With five evaluations, code generation performed better than random search and comparably to LLM search using human-specified configurations.Generated training code can reduce the initial search over configuration spaces unlikely to succeed.Source: §1 · p. 2

2 Method

The method treats hyperparameter optimization as an iterative process in which LLM proposals are evaluated and validation metrics guide subsequent proposals. It covers known search spaces, alternative prompting strategies, and code generation when the search space is unknown.

2.1 Background

  • HPO can be formulated as bilevel optimization: choose hyperparameters to minimize validation loss after model parameters are trained to convergence on the training objective.The formulation assumes unique solutions for simplicity.Source: §2.1 · p. 2

2 Method

  • The method alternates between LLM-proposed hyperparameter settings and training runs, returning validation metrics to the LLM until the search budget is exhausted.Prompts describe the problem, hyperparameters, and, when known, their search spaces.Source: Figure 1 · p. 2, §2.2 · p. 3

2.2 LLMs for Hyperparameter Optimization

  • For known search spaces, the chat prompt retains full conversational history, while the compressed prompt stores the problem description and configuration-metric history in one message.Compressed prompts are cheaper and achieve similar performance to chat prompts.Source: §2.2 · p. 3
  • When the search space is unknown, the LLM can generate model and optimizer code, treating the code string as the hyperparameter and avoiding human specification of the search space.If generated code fails to produce a valid output, the method can re-prompt with the error message.Source: §2.2 · pp. 3–4

3 Related Work

Related work spans model-free and structured HPO methods, gradient-based approaches, and applications of LLMs to decision-making and AutoML. The paper positions its work as using LLMs to make decisions during HPO for machine-learning tasks.

Hyperparameter Optimization (HPO)

  • Grid and random search are early model-free HPO methods, while multi-fidelity methods such as Hyperband and Successive Halving use early stopping but depend on randomness and do not fully leverage HPO structure.These methods are easy to parallelize and have shown promising practical results.Source: §3 · p. 4
  • Bayesian optimization builds a surrogate from past evaluations and uses uncertainty in an acquisition function, but faces sensitivity to surrogate parameters and scalability problems.The paper reports that LLMs can exceed BO in the initial tuning phase and notes the potential benefit of a hybrid strategy.Source: §3 · p. 4
  • Gradient- or conditioning-based HPO methods can be more scalable and efficient, but often require differentiable objectives, implementation effort, and deployment alongside the model.OptFormer instead fine-tunes transformers on offline data to transfer-learn a surrogate for gradient-based HPO.Source: §3 · p. 4

Decision making with LLMs

  • Prior LLM optimization work studied linear regression, Traveling Salesman, and prompt optimization; this paper applies LLM decision-making to HPO across classical machine-learning algorithms and deep neural networks.Other work integrates LLMs into AutoML, including finding competitive architectures on neural architecture search benchmarks.Source: §3 · p. 4

4 Results

The results evaluate LLM-based hyperparameter optimization on benchmark tasks and deep-learning models, then examine how prompt information affects tuning. On HPOBench and CIFAR-10, LLM approaches outperform random search in the reported comparisons.

4 Results

  • The evaluation spans eight HPOBench datasets and four model types, plus CIFAR-10 ViT and ResNet tuning and code-generation experiments.The HPOBench setup comprises 32 tasks.Source: §4 · p. 4, §4.1 · p. 5

4.1 Hyperparameter Optimization on HPOBench

  • GPT-4 Turbo beats random search on 81.25% of 32 tasks and achieves the highest mean and median improvement and consistently better mean rank.Each method receives 10 evaluations.Source: Table 1 · p. 5, §4.1 · p. 5

4.2 CIFAR-10

  • On CIFAR-10, across Vision Transformer and ResNet tuning, LLM-based approaches achieve lower best validation loss than random search after 30 iterations, with the best loss tracked across iterations.The config-based LLM approach using explicit hyperparameter ranges performs similarly to the unconstrained LLM.Source: Figure 3 · p. 7

4.3 Ablation Studies on Prompt Information and Robustness

  • The prompt-information ablation varies how much dataset and architecture detail is supplied when tuning a Vision Transformer.Conditions include adding CIFAR-10 information or both CIFAR-10 and the correct architecture; the supplied passages do not report the ablation outcome.Source: §4.3 · p. 8

3. Adding dataset and incorrect architecture (“...tune a convolutional neural network on CIFAR-10”)

With five evaluations, code generation found better initial settings than competing approaches. Prompt detail could improve early hyperparameter selection, while added noise produced similar performance.

3. Adding dataset and incorrect architecture (“...tune a convolutional neural network on CIFAR-10”)

  • More prompt information could improve hyperparameter selection in the first two steps, but all conditions reached similar loss by iteration 10.The reported loss at iteration 10 is approximately 0.65.Source: §3 · p. 8, Figure 4 · p. 8
  • Performance was similar when loss and accuracy feedback was perturbed by a uniformly sampled factor from 0.9 to 1.1 at each step.The authors connect this robustness to stochasticity in neural-network validation metrics.Source: §3 · p. 8, Figure 4 · p. 8
  • In the sampled trajectories, learning-rate choices varied with the optimizer, and larger batch sizes were not tried after batch size 256 yielded higher loss.GPT-4 selected SGD once and Adam for the remaining proposals in both runs.Source: §3 · p. 8, Figure 5 · p. 9

4.4 Code generation

  • Code-generation HPO treats model and optimizer source code as hyperparameters, then tunes generated functions through argument-specific calls and validation feedback.The first iteration generates code; later iterations generate function calls with specific arguments.Source: §4.4 · p. 9
  • With 5 evaluations, code generation obtained better initial settings than competing approaches, as measured by minimum test loss.Table 4 reports minimum test loss after 5 tuning iterations.Source: §4.4 · p. 9, Table 4 · p. 9

5 Discussion

The discussion identifies dataset contamination, reproducibility, and cost as limitations, while proposing research directions for understanding LLM performance and using LLMs as research assistants. The appendices document prompts, transition messages, and optimizer settings.

5 Discussion

  • Benchmarks may appear in LLM training data, but prompting with function outputs and no benchmark details reduces the possibility of direct memorization.The authors suggest private or synthetic datasets for future studies.Source: §5 · pp. 9–10
  • GPT-4 inference procedures are undisclosed, and temperature 0 does not always produce deterministic outputs, complicating replication.The authors suggest open-source LLMs for reproducible benchmarks.Source: §5 · p. 10
  • GPT-4 HPO can cost about $8 for 32 HPOBench tasks at trajectory length 10, while a compressed-prompt, ten-round run costs five cents.The authors note costs can rise with more iterations and may be worthwhile when the underlying experiment is expensive.Source: §5 · p. 10

5.1 Future Directions

  • The mechanisms behind LLM effectiveness in HPO are not well understood; more parameters and training enabling useful memorization is offered as one hypothesis.The authors suggest this may partly explain the performance jump from GPT-3.5 to GPT-4.Source: §5.1 · p. 10
  • LLMs can provide natural-language feedback on error messages, but may hallucinate, misread model architecture, or recommend inappropriate regularization.The authors propose grounding suggestions in verified technical knowledge and improving dialogue systems for debugging.Source: §5.1 · pp. 10–11

5.2 Ethics Statement

  • LLM hyperparameter tuning may make it easier to find well-performing settings and lower training costs, potentially broadening access to machine learning experiments.The authors note that benefits and risks are likely similar to those of other AutoML research and flag risks from code generation.Source: §5.2 · p. 11

A Appendix

  • The appendix directs readers to HPOBench details, code-generation experiments, 2D landscapes, tuning-assistant dialogues, and LLM response trajectories.Source: A Appendix · p. 15

B.1 Prompts; B.2 Transition Messages; B.3 Other Experiment Details

  • The appendix uses a generic message plus Sklearn documentation for prompts, leaves prompts untuned, reports one LLM seed at temperature 0, and uses SMAC's default RF and GP surrogate settings.The last-attempt prompt warns that the budget is nearly exhausted.Source: §B.1 · p. 15, §B.2 · p. 16, §B.3 · p. 16

B.4 Results

Across HPOBench tasks, GPT-4 Turbo consistently beats random search and is comparable to Bayesian optimization, including trajectories up to 100 evaluations. Figures 7 and 8 show minimum validation error after 10 evaluations across benchmark tasks.

B.4 Results

  • Figure 7 compares minimum validation error after 10 function evaluations across HPOBench neural-network tasks and optimizer methods.The caption describes GPT-4 Turbo as performing well against random search and BO with GP or RF surrogate models; aggregate results across four models are in Table 1.Source: Figure 7 · p. 17
  • Figure 8 reports minimum validation error after 10 evaluations for HPOBench tasks tuning random forests and SVMs.Aggregate results across four models are reported in Table 1.Source: Figure 8 · p. 18
  • GPT-4 Turbo consistently beats random search and is comparable to Bayesian optimization on HPOBench, including trajectories up to 100 evaluations.The 60- and 100-iteration comparison covers 24 tasks across SVMs, random forests, and logistic regression; neural networks were excluded because they were too costly to train for those lengths.Source: §B.4 · p. 16, Table 5 · p. 19

B.5 Additional Results

The appendix evaluates an added system prompt and reports mean and median changes. The prompt is used throughout the remaining experiments.

B.5 Additional Results

  • The authors use the “You are a machine learning expert” system prompt throughout the rest of the experiments.They relate this choice to conditioning on good performance.Source: §B.5 · p. 19
  • The appendix evaluates adding the system prompt “You are a machine learning expert” and reports mean and median changes.Source: Table 6 · p. 19, §B.5 · p. 19
  • Table 6 is identified as an evaluation of system-prompt effects, but the supplied caption gives no outcome values.Source: Table 6 · p. 19

C CIFAR-10 Experiment Details

The CIFAR-10 experiments use three random seeds and evaluate test accuracy once after 20 epochs. In the code-generation setup, the LLM writes a model-and-optimizer function, which is validated and can be regenerated after errors.

C CIFAR-10 Experiment Details

D Code Generation

  • For code generation, the LLM writes PyTorch code for a model-and-optimizer function using dataset and feature descriptions, then tunes its hyperparameters through function calls and training feedback.The dataset is assumed to be preprocessed and available as a DataLoader; the function takes tunable hyperparameters as arguments.Source: D Code Generation · pp. 20–21
  • Generated model-and-optimizer code is checked for valid Python and the required function signature and outputs; errors are returned to the LLM for regeneration.Source: D Code Generation · p. 21

E 2-dimensional Landscape Experiment

The experiment tests whether GPT-4 can optimize toy functions and whether prompt details affect its search. Across most tasks, performance beats random search, though results vary across seeds.

E 2-dimensional Landscape Experiment

  • The study tests Rosenbrock, Branin, Himmelblau, Ackley, and well- and ill-conditioned quadratic functions, including randomly shifted versions of the first set.The shifts use a fixed c with c_i sampled from U(0, 1) to mitigate potential memorization.Source: E 2-dimensional Landscape Experiment · p. 21
  • Across 3 seeds, GPT-4 outperformed random search on most tasks, but its performance varied between seeds.The figure reports optimization in most situations with few evaluations, including after random argument shifts to mitigate memorization.Source: E 2-dimensional Landscape Experiment · p. 22, Figure 9 · p. 24

E.1 Prompts

  • Adding a sentence that the functions are deterministic produced the same or better performance on 8 of 10 tasks.Prompt 3 adds that the training process is deterministic and yields a nonnegative loss.Source: E 2-dimensional Landscape Experiment · p. 22, Table 9 · p. 23
  • The experiment compares a practical black-box optimization prompt with variants that add a compute budget or frame the task as hyperparameter optimization.Prompt 3 further states that training is deterministic and yields nonnegative loss.Source: §E.1 · p. 22

F LLMs as Tuning Assistants

The paper presents LLMs as interactive assistants for debugging and tuning, while noting that their advice can rely on unsupported assumptions. Examples include responses to NaN errors and low-accuracy models.

F LLMs as Tuning Assistants

  • LLMs can give users feedback on error messages and answer follow-up questions, but may assume a neural network and suggest regularization when a model is underfitting.The paper notes that an expert might first ask what model is being trained and about validation and training performance.Source: F LLMs as Tuning Assistants · p. 25
  • For NaN errors with Adam at learning rate 0.1, reduce it to 0.001 or 0.0001; if divergence continues, try gradient clipping, check for NaN or infinite input values, and verify the model architecture.Source: F LLMs as Tuning Assistants · p. 26
  • For low accuracy, GPT-4 suggests increasing epochs, layers, or neurons for underfitting, while its dropout suggestion may not help underfitting models.The response also discusses learning rate, batch size, activation functions, initialization, optimizer, and momentum.Source: F LLMs as Tuning Assistants · p. 27, Figure 11

G.2 HPOBench

The HPOBench examples show GPT-4 reviewing successive configurations and their losses, sometimes identifying improvement and sometimes noting a plateau or worse performance.

G.2 HPOBench

  • In Task 1, the initial configuration had loss 0.19283 and the second had 0.19731; increasing the learning rate and reducing batch size did not improve performance.A later configuration showed loss 0.21525 and some improvement relative to the preceding configuration.Source: §G.2 · p. 28
  • A configuration with alpha 0.001, batch size 16, depth 2, learning rate 0.001, and width 512 reached loss 0.17937.The accompanying analysis says reducing depth and increasing width continued to be effective, while a smaller batch size may also have contributed.Source: §G.2 · p. 28
  • Another configuration with alpha 0.001, batch size 16, depth 3, learning rate 0.001, and width 256 also reached loss 0.17937, described as a plateau.The analysis says this matched the previous best configuration.Source: §G.2 · p. 28
  • A configuration with alpha 0.001, batch size 8, depth 2, learning rate 0.001, and width 512 had loss 0.21525, worse than the previous best.The analysis cautions that reducing depth and increasing width may not always improve performance.Source: §G.2 · p. 28

Related papers

Loading 2312.04528v2…