Source-linked AI summary
Practical Multi-fidelity Bayesian Optimization for Hyperparameter Tuning
Jian Wu, Saul Toscano-Palmerin, Peter I. Frazier, Andrew Gordon Wilson
TL;DR
Expensive validation-error evaluations make deep-network hyperparameter tuning difficult, motivating cheaper multi-fidelity observations. The paper proposes taKG, which combines trace observations with multiple continuous fidelity controls and provides a convergent acquisition-optimization method. Experiments report improvement over established alternatives and strong performance without trace observations.
Problem
Deep-network hyperparameter tuning is bottlenecked by the time required to evaluate validation error, motivating cheaper fidelity-based proxies.
Method
taKG combines trace observations with multiple continuous fidelity controls, while an unbiased stochastic-gradient method optimizes the acquisition function.
Results
The experiments report significant improvement over FaBOLAS, Hyperband, and BOCA, with strong performance also when trace observations are unavailable.
Takeaways & Limitations
The approach supports practical multi-fidelity Bayesian optimization for iteratively trained models, including batch or sequential use and available gradient information.
Takeaways & Limitations
The neural-network experiments omit some benchmarks and evaluate only selected methods in batch settings.
Abstract
from arXiv · showhide
Bayesian optimization is popular for optimizing time-consuming black-box objectives. Nonetheless, for hyperparameter tuning in deep neural networks, the time required to evaluate the validation error for even a few hyperparameter settings remains a bottleneck. Multi-fidelity optimization promises relief using cheaper proxies to such objectives --- for example, validation error for a network trained using a subset of the training points or fewer iterations than required for convergence. We propose a highly flexible and practical approach to multi-fidelity Bayesian optimization, focused on efficiently optimizing hyperparameters for iteratively trained supervised learning models. We introduce a new acquisition function, the trace-aware knowledge-gradient, which efficiently leverages both multiple continuous fidelity controls and trace observations --- values of the objective at a sequence of fidelities, available when varying fidelity using training iterations. We provide a provably convergent method for optimizing our acquisition function and show it outperforms state-of-the-art alternatives for hyperparameter tuning of deep neural networks and large-scale kernel learning.
1 INTRODUCTION
Hyperparameter evaluation is costly, motivating multi-fidelity Bayesian optimization with cheaper training-based approximations. The paper introduces taKG, which uses trace observations and multiple continuous fidelities, provides a convergent optimizer, and improves over established alternatives.
- Motivation: Evaluating validation error for hyperparameter settings can require substantial time and computational power, motivating cheaper low-fidelity approximations.These approximations may use fewer training iterations, training points, or validation points.
- Motivation: Trace observations record performance across training iterations, allowing later evaluations to continue from previously completed iterations.This can reduce computation when evaluating a larger iteration count after a smaller one.
- Approach: The proposed trace-aware knowledge gradient (taKG) jointly leverages trace information and multiple continuous fidelity controls while selecting hyperparameters and training or validation sizes.It supports simultaneous control of iterations, training data, and validation data.
- Approach: Two taKG variants address settings where sampling costs remain substantial throughout the fidelity space or where cost and information vanish near zero fidelity.The 0-avoiding variant avoids repeatedly sampling near-zero fidelities without requiring a tuned fixed cost.
- Optimization: The paper introduces an unbiased stochastic gradient estimator and multistart stochastic gradient ascent that converges to a local stationary point of the acquisition function.The method is designed for acquisition functions that cannot be evaluated in closed form.
- Results: Numerical experiments report significant improvement over FaBOLAS, Hyperband, and BOCA, with strong performance also when trace observations are unavailable.The approach remains applicable with continuous fidelity controls in that setting.
2 THE taKG AND taKG∅ACQUISTION FUNCTIONS
The paper formulates multi-fidelity Bayesian optimization with trace observations and defines taKG and taKG∅ to value information relative to sampling cost. It also provides a stochastic-gradient optimization procedure with almost-sure convergence under stated conditions.
- 2.1 Problem Setting: The framework models the objective as g(x, s), with full-fidelity objective f(x) = g(x, 1) and continuous fidelity controls s ∈ [0, 1]^m.Fidelity controls are scaled so 0 is lowest fidelity and 1 is highest; evaluation cost may depend on x and s.
- 2.1 Problem Setting: Trace evaluations return observations across fidelity values, while non-trace controls return only the selected fidelity; evaluation cost is assumed non-decreasing in each fidelity component.For training iterations, observing a fidelity provides the performance trace from zero through the evaluated iteration level.
- 2.2 Valuing Trace Observations: L_n(x, S) is the expected loss after observing x at fidelities S and then choosing the solution minimizing posterior expected full-fidelity loss.It is computed by simulating observations, updating the Gaussian-process posterior, optimizing the resulting predicted loss, and averaging over replications.
- 2.5 Efficiently maximizing taKG and taKG∅: A stochastic gradient estimator enables acquisition optimization despite simulation-based evaluation, and stochastic gradient ascent converges almost surely to a connected set of stationary points of taKG∅ under the stated assumptions.The estimator is unbiased, and the convergence result assumes conditions including a compact hyperrectangle, positive bounded cost, and a suitable diminishing stepsize sequence.
- 2.3 Trace-aware Knowledge Gradient: taKG ranks a point and fidelity set by expected-loss reduction per unit sampling cost, using VOI_n(x, S) = L_n(∅) − L_n(x, S).The cost of observing a fidelity set is assigned to the elementwise maximum fidelity needed to obtain that set.
- 2.4 0-avoiding Trace-aware Knowledge Gradient: taKG∅ modifies the value of information so fidelities with a zero component receive zero value, avoiding repeated sampling at extremely low fidelities.Under regularity conditions, its value of information is continuous and remains zero when the smallest component of the selected fidelity is zero, discouraging nearby low-fidelity samples.
3 NUMERICAL EXPERIMENTS
The experiments evaluate taKG∅ across synthetic optimization, neural-network hyperparameter tuning, and large-scale kernel learning, using sequential, batch, and derivative-enabled comparisons. Across these settings, taKG∅ generally outperforms competing methods, while batch evaluations can further improve performance in some neural-network experiments.
- Synthetic functions: Synthetic benchmarks use Branin, Rosenbrock, and Hartmann functions with one or two continuous fidelity controls and compare sequential and batch methods.Evaluation costs model training-data visits, training iterations, and fixed validation overhead; batch cost is the maximum individual evaluation cost.
- Synthetic functions: Batch 8-taKG∅ performs well relative to batch 8-EI, 8-KG, and Hyperband on the synthetic benchmarks.Hyperband is treated as a batch method despite varying parallelism during its operation.
- Neural networks: On MNIST, sequential taKG∅ outperforms KG, EI, and FaBOLAS, while batch-size-4 taKG∅ improves over batch KG, EI, and Hyperband.The neural-network experiments use training-set size and training iterations as fidelity controls; MNIST tunes five hyperparameters for a two-layer feedforward network.
- Neural networks: On CIFAR-10 and SVHN, sequential taKG∅ outperforms competitors, and batch taKG∅ further improves performance on CIFAR-10.With optimized hyperparameters trained for 200 epochs on the full datasets, test classification error is approximately 12% for CIFAR-10 and 5% for SVHN.
- Large-scale kernel learning: In large-scale KISS-GP kernel learning, derivative-enabled taKG∅ finds good solutions more quickly than derivative-enabled knowledge gradient in both sequential and batch settings.The experiment optimizes three GP hyperparameters on 1 million training points using training-point and inducing-point fidelities.
4 CONCLUSION
The paper proposes trace-aware knowledge gradient as a multi-fidelity acquisition function that uses trace observations and multiple continuous fidelities. It extends naturally to batch and derivative settings and is reported to find good solutions more quickly in deep-learning and kernel-learning applications.
- Conclusion: Trace-aware knowledge gradient leverages trace observations and multiple simultaneous continuous fidelities, with natural extensions to batch and derivative settings.The acquisition function uses trace structure to accelerate global optimization in application settings including deep learning and kernel learning.
5 SUPPLEMENTARY MATERIAL
The supplementary material specifies the Gaussian-process modeling assumptions, establishes smoothness and convergence-related results, and details fidelity kernels and synthetic experiment settings.
- Gaussian-process model: The model places a Gaussian-process prior on g and assumes additive independent Gaussian noise with common variance σ2.The noise variance and mean and kernel parameters are treated as Bayesian hyperparameters.
- Trace observations: A single evaluation can yield multiple trace observations; taKG retains two observations per evaluation, making n twice the evaluation count.
- Theoretical analysis: The proofs establish continuity of Ln and use differentiability, compactness, boundedness, and stochastic-gradient conditions to support the convergence analysis.The supplementary arguments derive these properties for the posterior and cost models under continuously differentiable kernels and compact domains.
- Fidelity kernels: For two continuous fidelities, the GP kernel factors into hyperparameter and fidelity kernels, with an intercept added to model a loss that does not diminish.The fidelity controls represent training iterations and training-data amount, and the associated kernel parameters are treated Bayesianly.
- Experiments: The experiments define synthetic test functions and search ranges for feedforward-network learning rate, dropout, batch size, and layer width.The reported ranges are [10−6, 100], [0, 1], [25, 210], and [100, 1000], respectively.