Efficiently Estimating Optimal Hyperparameter Scaling Laws through Power-Law Entropy Search

Zhiliang Chen; Sebastian Ament; David Eriksson; Maximilian Balandat; Bryan Kian Hsiang Low; Eytan Bakshy; Jihao Andreas Lin

Source-linked AI summary

Efficiently Estimating Optimal Hyperparameter Scaling Laws through Power-Law Entropy Search

Zhiliang Chen, Sebastian Ament, David Eriksson, Maximilian Balandat, Bryan Kian Hsiang Low, Eytan Bakshy, Jihao Andreas Lin

arXiv:2609.01431v2Sep 1, 2026cs.LGcs.AI

TL;DR

Optimal hyperparameter scaling laws can guide production-scale LLM configurations, but conventional estimation requires expensive grid searches and repeated experiments. PLES uses cost-aware multi-fidelity Bayesian optimization to select experiments that reduce scaling-law uncertainty, and it reaches accurate laws across synthetic, surrogate, and actual training settings with less than one-tenth the computational budget of conventional baselines.

Problem
Estimating optimal hyperparameter scaling laws conventionally requires costly grid searches, and the laws must be repeatedly re-derived as architectures and settings change.
Method
PLES is a cost-aware acquisition function that uses multi-fidelity Bayesian optimization, GP-based estimates, and uncertainty reduction across scales to choose experiments.
Results
Less than one-tenth of the computational budget is required for PLES to converge to accurate scaling laws across synthetic benchmarks, real-data surrogates, and actual LLM pre-training runs.
Takeaways & Limitations
PLES provides a posterior-uncertainty stopping criterion and can recover multiple hyperparameter scaling laws even when acquisition targets one hyperparameter.
Takeaways & Limitations
PLES assumes the optimal hyperparameter scaling law has a known power-law form; deviations can bias coefficient estimates and degrade held-out-scale extrapolation.

Abstract

from arXiv · show

Optimal hyperparameter scaling laws describe how the best hyperparameters for large language model (LLM) training change with model and data scale, enabling practitioners to predict optimal configurations at production scales without expensive large-scale tuning. However, estimating these scaling laws conventionally requires exhaustive grid searches over thousands of training runs, consuming enormous computational resources. We introduce Power-Law Entropy Search (PLES), a computational cost-aware acquisition function built on multi-fidelity Bayesian optimization that efficiently estimates optimal hyperparameter scaling laws through adaptive experimentation. A key innovation in PLES is that it searches for candidates that reduce the overall uncertainty of a scaling law estimate, instead of optimizing a single objective function. At each iteration, PLES selects the candidate configuration that maximally reduces the uncertainty of the scaling law estimates per unit computational cost, naturally favoring informative small-scale experiments. We evaluate PLES on synthetic benchmarks, surrogate models fitted to real LLM training data, and actual LLM pre-training runs. Across all settings, PLES converges to accurate optimal hyperparameter scaling laws using less than one-tenth of the computational budget required by conventional grid search and other baselines.

1 Introduction

Optimal hyperparameter scaling laws enable production-scale LLM configurations without directly tuning at those scales, but conventional estimation requires costly repeated grid experiments. PLES instead uses cost-aware adaptive experimentation to estimate the whole scaling-law function efficiently.

Scaling laws describe how optimal hyperparameters change with model and data size, guiding production-scale LLM development without large-scale HPO.
Over 3,700 LLMs and nearly one million NVIDIA H800 GPU hours illustrate the computational cost of conventional scaling-law estimation.
PLES balances experiment cost against information gained across model and data scales using multi-fidelity Bayesian optimization and a GP surrogate.
PLES estimates the entire scaling-law function by exploiting information from experiments at multiple scales rather than optimizing one objective function.
PLES achieves more accurate scaling laws than conventional baselines under equal computational budgets, including for LLM pretraining.
PLES estimates converge toward the ground-truth scaling law while uncertainty intervals indicate confidence in the estimate.

2 Related Works

The paper distinguishes optimal hyperparameter scaling laws from conventional loss scaling laws and situates PLES within information-driven experimental design. Unlike related work, it uses multi-fidelity Bayesian optimization and validates the method on actual LLM training runs.

Conventional LLM scaling laws describe validation-loss changes with model and data scale, whereas optimal hyperparameter laws describe how loss-minimizing hyperparameters change across scales.
Bayesian experimental design selects experiments to maximize information gain about a latent quantity, which motivates PLES’s information-driven small-scale experimentation.
Related work studies scaling-law experiment design but does not focus on optimal hyperparameters across model and data scales.
The related approach models scaling-law parameters with a Gaussian mixture model, while this paper adopts multi-fidelity Bayesian optimization.
This paper verifies its method on actual LLM training runs rather than only hypothetical experiments from an existing benchmark.

3 Problem setup

The problem setup defines the optimal hyperparameter at each model and data scale and represents its scaling law as a power-law function. The goal is to estimate its coefficients from small-scale experiments and infer the optimum at a held-out target scale.

For configuration (N, D, θ), the optimal hyperparameter minimizes evaluation loss, with training cost often approximated as 6ND.
An optimal hyperparameter scaling law maps model scale N and data scale D to the loss-minimizing hyperparameter θ* at each scale.
The held-out scale is the largest target model and data scale, where θ*_max = arg min_θ ℓ(N_max, D_max, θ).
The setup uses a power-law functional form for the optimal hyperparameter scaling law over model and data scales.
Estimating coefficients c, α, and β from small-scale experiments is equivalent to estimating the optimal hyperparameter scaling law.

4 Method

The method sequentially estimates optimal hyperparameter scaling laws with a GP surrogate, Thompson sampling, Bayesian linear regression, and the cost-aware PLES acquisition function. PLES prioritizes experiments that reduce coefficient uncertainty efficiently, while probabilistic uncertainty supports stopping and can inform multiple scaling laws.

4 Method: PLES maintains a GP surrogate over model size, data size, hyperparameters, and observed loss, then uses Thompson samples to estimate optima across scales.The resulting optimal-hyperparameter estimates are aggregated across scales for scaling-law estimation.
4 Method: The power law log θ∗ = log c + α log N + β log D becomes a Bayesian linear regression over coefficients w = (log c, α, β)⊤.The posterior mean and covariance quantify the estimated coefficients and their uncertainty.
4 Method: The differential entropy logdetΣw(GP(D)) represents uncertainty about the current power-law coefficients and therefore the scaling-law estimate.Lower coefficient variance corresponds to lower empirical uncertainty in the estimated scaling law.
4 Method: PLES selects the candidate with the largest reduction in power-law coefficient uncertainty per unit computational cost.A cost-cooling factor d can favor smaller-scale experiments; fantasy observations from the current GP estimate the candidate’s updated information.
4 Method: For multiple hyperparameters, PLES can reduce a weighted sum of differential entropies, while targeting one law may also recover another accurately.The reported example targets learning-rate uncertainty but also recovers the batch-size scaling law.
4 Method: Held-out-scale uncertainty provides a stopping criterion: when it is small, further experimentation does not significantly improve the estimated power law.This criterion follows from the probabilistic estimate of the power-law coefficients.

5 Experiments

PLES is evaluated on synthetic, surrogate, and real LLM training settings, where it estimates optimal hyperparameter scaling laws more accurately and with substantially less compute than baselines. Additional analyses examine uncertainty reduction, recovery of multiple scaling laws, candidate-scale selection, and computational overhead.

5.3 Main Results: PLES consistently achieves smaller held-out-scale percentage error and reaches the best attainable model loss in less than one-tenth of canonical-baseline compute.The comparison uses equal total compute budgets, shared starting configurations where applicable, and 10 random seeds per method.
5.4 Analysis of scaling law uncertainty with increasing computational cost: As experimentation proceeds, GP variance and scaling-law uncertainty decrease while the estimated learning-rate law converges toward the ground truth.At iteration 6, held-out-scale uncertainty is sufficiently small to stop; another iteration does not significantly improve the estimate.
5.5 Results on more than one hyperparameters (learning rate and batch size): PLES recovers the batch-size scaling law even though its acquisition objective targets uncertainty in the learning-rate scaling law.The paper suggests acquired candidates can lie in regions containing optima for multiple hyperparameters simultaneously.
5.6 Analysis of the scale of candidates acquired by PLES: Most PLES-proposed candidate scales cost less than 6% of the expensive held-out scale, reflecting its cost-aware preference for cheaper experiments.The figure caption reports a stricter description of most candidates as costing less than 2% of the held-out scale.
5.7 Computational complexity: PLES has per-iteration acquisition complexity O(C · K · Q), with Thompson sampling and inner minimizations fully parallelizable across candidates and samples.The resulting computational overhead is described as negligible compared with experiment costs.

6 Limitations

PLES assumes that the optimal hyperparameter scaling law follows a known functional form, which may not hold universally.

6 Limitations: If optimal hyperparameters deviate from the assumed power-law form, coefficient estimates can become biased and held-out-scale extrapolation accuracy can degrade.The authors identify more flexible functional representations with tractable uncertainty quantification as future work.

7 Conclusion

PLES efficiently estimates optimal hyperparameter scaling laws through adaptive experimentation. Across synthetic, surrogate, and actual LLM pre-training settings, it achieves accurate estimates at less than one-tenth the computational budget of conventional grid search and other baselines.

Less than one-tenth the computational budget is sufficient for PLES to converge to accurate optimal hyperparameter scaling laws across three experimental settings.The settings include synthetic benchmarks, surrogate models fitted to real LLM training data, and actual LLM pre-training runs.
PLES provides a stopping criterion based on posterior uncertainty at the target scale, allowing experimentation to end once sufficient confidence is achieved.
PLES recovers scaling laws for multiple hyperparameters simultaneously even when its acquisition function reduces uncertainty for only one.
PLES is presented as a practical framework for reducing the computational cost of developing future LLMs.The authors connect this value to repeated scaling-law re-estimation as architectures and training recipes evolve.

A Pseudocode for PLES

PLES evaluates candidate experiments by how much they reduce uncertainty in one or more power-law scaling laws. Its pseudocode updates a GP with a fantasy observation, computes entropy reduction, and selects the highest-value candidate.

PLES accepts the current GP, candidate configurations with model and data scales, and a cost-cooling factor, then outputs an acquisition value for each candidate.
For each candidate, PLES draws a fantasy observation, updates the GP, and computes the resulting power-law uncertainty.
The acquisition value is the reduction from logdetΣw(GP(D)) to logdetΣw(GP(D∪x)), and the highest-value candidate is selected.
Multiple hyperparameters: For two hyperparameters, PLES models separate power laws with independently estimated coefficient vectors and Bayesian linear-regression posteriors.
Multiple hyperparameters: The two regression problems share scale design matrix X but use different targets and noise matrices.
Multiple hyperparameters: The modified acquisition function maximizes the sum of entropy reductions to reduce uncertainty in both scaling laws simultaneously.

C Details of Synthetic Function

The synthetic benchmark models language-model training loss using learning rate and batch size as optimization variables and model and data size as fidelity variables. It uses a log-space quadratic loss landscape with prescribed power-law optima and compute cost proportional to N·D.

The four-dimensional benchmark uses learning rate η and batch size B as optimization variables, with model size N and data size D as fidelity variables.All four inputs are parameterized in log10 space.
The evaluation cost is cost(N, D) = N · D, making small-N, small-D queries cheaper than high-fidelity queries.
The optimal learning rate follows a joint power law in model and data size, decreasing with model size and increasing mildly with data size.
The optimal batch size follows a power law in data size alone, with exponent d = 0.5.
Around the optimum, loss is modeled as a smooth quadratic bowl in log-space with zero learning-rate–batch-size cross-term.
The additive irreducible-loss term improves with scale according to a Chinchilla-style formulation.

D Using GP as Oracle Ground-truth

The surrogate experiments fit a Gaussian Process to real LLM training-run data and use it as ground truth for validating PLES. The fitted surrogate closely matches observed optimal learning rates and batch sizes, with RMSE below 0.01.

The surrogate GP is fitted to training-run data from Li et al. (2025) and used as ground truth to validate PLES.
Observed optimal learning rates and batch sizes are close to the GP’s predicted optima at each model and data scale.The training-run optima are shown as yellow stars and GP predictions as red stars.
RMSE values are all smaller than 0.01, supporting the GP surrogate as a fit to the real training data.

E Budget Allocation for Grid Search

Grid search should distribute experiments across scales rather than concentrate the budget on cheap small models. Wide scale coverage estimates the scaling-law slope more reliably and strengthens the baseline.

Spending the entire tuning budget on small models is a poor use of compute for predicting optimal hyperparameters at much larger scales.The task resembles regression extrapolation, where densely sampling only small scales limits large-scale prediction.
The grid-search baseline allocates experimental runs evenly across all scales to provide a stronger comparison.Uniform allocation is preferred over focusing the budget at small scales.
Crowded points at small scales estimate the scaling-law slope poorly, and small slope errors propagate into larger uncertainties at larger scales.This makes precise small-scale optima insufficient for reliable large-scale extrapolation.
Spreading experiments across a wide range of scales gives a better estimate despite making each individual measurement coarser.The benefit comes from improving slope estimation across the extrapolation range.