Scalable Bayesian Optimization Using Deep Neural Networks

Jasper Snoek; Oren Rippel; Kevin Swersky; Ryan Kiros; Nadathur Satish; Narayanan Sundaram; Md. Mostofa Ali Patwary; Prabhat; Ryan P. Adams

Source-linked AI summary

Scalable Bayesian Optimization Using Deep Neural Networks

Jasper Snoek, Oren Rippel, Kevin Swersky, Ryan Kiros, Nadathur Satish, Narayanan Sundaram, Md. Mostofa Ali Patwary, Prabhat, Ryan P. Adams

arXiv:1502.05700v2Feb 19, 2015stat.ML

TL;DR

Bayesian optimization needs accurate, inexpensive surrogate distributions for expensive objectives, but Gaussian processes scale cubically with observations. The paper replaces them with adaptive basis regression using neural networks, achieving linear scaling while retaining competitive optimization performance across benchmarks and large-scale applications.

Problem
Gaussian-process Bayesian optimization scales cubically with observations, making objectives requiring many evaluations and massive parallelization difficult to handle.
Method
DNGO uses a neural network to learn adaptive basis functions for Bayesian linear regression as an alternative surrogate model.
Results
DNGO scales linearly rather than cubically and remains competitive with state-of-the-art Bayesian optimization across benchmark, object-recognition, and image-captioning tasks.
Takeaways & Limitations
DNGO is especially well suited to massively parallel hyperparameter optimization while maintaining tractability and principled uncertainty management.
Takeaways & Limitations
The paper notes that sparse Gaussian-process models are another promising route for improving scalability.

Abstract

from arXiv · show

Bayesian optimization is an effective methodology for the global optimization of functions with expensive evaluations. It relies on querying a distribution over functions defined by a relatively cheap surrogate model. An accurate model for this distribution over functions is critical to the effectiveness of the approach, and is typically fit using Gaussian processes (GPs). However, since GPs scale cubically with the number of observations, it has been challenging to handle objectives whose optimization requires many evaluations, and as such, massively parallelizing the optimization. In this work, we explore the use of neural networks as an alternative to GPs to model distributions over functions. We show that performing adaptive basis function regression with a neural network as the parametric form performs competitively with state-of-the-art GP-based approaches, but scales linearly with the number of data rather than cubically. This allows us to achieve a previously intractable degree of parallelism, which we apply to large scale hyperparameter optimization, rapidly finding competitive models on benchmark object recognition tasks using convolutional networks, and image caption generation using neural language models.

1. Introduction

As machine-learning models and hyperparameter spaces grow, tuning becomes increasingly important while GP-based Bayesian optimization faces cubic scaling. The paper proposes DNGO, a neural-network-based approach that scales linearly and achieves competitive results on difficult applications.

Growing model complexity introduces more hyperparameters, making their proper setting critical for performance on difficult problems.
Bayesian optimization provides model-based global optimization for noisy, expensive black-box functions by balancing exploration and exploitation through uncertainty modeling.
GP-based Bayesian optimization has cubic inference-time scaling with observations, limiting optimization when many configurations must be evaluated.
DNGO uses neural networks to learn adaptive basis functions for Bayesian linear regression, scaling linearly with function evaluations while retaining uncertainty characterization.
6.37% and 27.4% classification results on CIFAR-10 and CIFAR-100, plus BLEU scores of 25.1 and 26.7 on Microsoft COCO 2014, demonstrate competitive performance across applications.

2. Background and Related Work

Bayesian optimization uses a probabilistic surrogate and acquisition function to select evaluations for noisy, expensive black-box objectives. Its standard GP formulation offers flexibility and uncertainty but faces computational limitations and motivates scalable alternatives.

Bayesian optimization constructs a probabilistic model over objective functions and queries its posterior to decide where to evaluate the expensive function.
The acquisition function quantifies the promise of a new experiment from posterior mean and variance, balancing exploration and exploitation.
Gaussian processes are commonly used because of their flexibility, well-calibrated uncertainty, and analytic properties.
The standard approach uses a GP surrogate with the expected improvement acquisition function, whose criterion depends on predictive mean, variance, and the lowest observed value.
Bayesian neural networks target uncertainty over network weights, but their full posterior is intractable for most neural-network forms.

3. Adaptive Basis Regression with Deep Neural Networks

DNGO replaces the GP surrogate with adaptive basis regression whose neural-network basis retains uncertainty modeling while reducing scaling with observations. The method uses practical architectural and prior choices to support parallel Bayesian optimization.

Adaptive basis regression: DNGO combines a deep neural network with Bayesian linear regression on the last hidden layer, marginalizing output weights while point-estimating the remaining parameters.The network supplies adaptive nonlinear basis functions, while the Bayesian output layer captures predictive uncertainty.
Scalability: Adaptive basis regression scales linearly with observations and cubically with basis-function dimensionality, enabling a trade-off between evaluation time and model capacity.This shifts the main cubic dependence away from the number of observations.
Scalability: The matrix inversion bottleneck depends on output dimensionality D rather than observation count N, allowing substantially more observations than GP-based optimization.The paper demonstrates this scaling advantage in Figure 1.
Network architecture: The selected architecture is a deep, narrow network with 3 hidden layers and approximately 50 hidden units per layer, using no dropout and modest ℓ2 normalization.These settings were selected with GP-based Bayesian optimization and then used throughout the empirical evaluation.
Quadratic prior: DNGO uses a convex quadratic prior mean centered in the bounded search region, with horseshoe priors that shrink small quadratic effects while preserving large ones.The prior can reduce to a simple offset when the quadratic component is unnecessary, helping address model misspecification.
Network architecture: Activation choice materially changes learned basis functions and uncertainty, while unbounded ReLU activations can produce extremely large uncertainty estimates.The comparison includes tanh, ReLU, and ReLU with tanh on the final hidden layer.

4. Experiments

Experiments evaluate DNGO on benchmark optimization, image caption generation, and convolutional-network recognition, including highly parallel hyperparameter searches that produce competitive models.

4.1. HPOLib Benchmarks: DNGO significantly outperforms SMAC and TPE on HPOLib benchmarks while remaining competitive with Gaussian-process optimization.The comparison indicates that its scalability does not substantially reduce the number of evaluations needed to find minima.
4.2. Image Caption Generation: Up to 800 parallel evaluations were used to optimize the log-bilinear image-captioning model, whose individual evaluations averaged 26.6 hours.The search covered 11 hyperparameters spanning learning, regularization, and architectural choices, including categorical variables and infeasible regions.
4.2. Image Caption Generation: Approximately 2500 experiments, equivalent to over 2700 CPU days, were evaluated in less than one week using 300–800-way parallelism.The experiments were selected and evaluated while cluster availability determined the parallel batch size.
4.2. Image Caption Generation: Ensembling the top LBL models produced a test-set BLEU score of 26.7 and significantly outperformed the reported LSTM-based approaches.The authors attribute the ensemble gain as potentially related to distinct multiple local optima in the hyperparameter space.
4.3. Deep Convolutional Neural Networks: The optimal convolutional-network models achieved test errors of 6.37% on CIFAR-10 and 27.4% on CIFAR-100.The same generic architecture was specialized to each dataset through individualized hyperparameter tuning, with final models trained for 350 epochs on the full training set.
4.3. Deep Convolutional Neural Networks: Parallelized automated hyperparameter tuning obtained models highly competitive with state-of-the-art results in only a few sequential steps.Figure 4 visualizes validation errors over time and the evolving best error as exploration and exploitation are balanced.

5. Conclusion

DNGO enables efficient optimization of noisy, expensive black-box functions while retaining GP-like tractability and uncertainty management. Its linear scaling makes it especially suitable for massively parallel hyperparameter optimization, though alternative scalable probabilistic models remain possible.

DNGO improves scalability from cubic to linear in the number of observations while retaining tractability and principled uncertainty management.
DNGO performs competitively with existing state-of-the-art Bayesian optimization approaches.
DNGO is especially well suited to massively parallel hyperparameter optimization.
Adaptive basis regression with neural networks is one scalable approach, while sparse Gaussian-process models also remain promising alternatives.

A. Convolutional neural network experiment specifications

This section specifies the convolutional neural network experiment setup, including architecture, training, meta-optimization, and hyperparameterization details.

The experiment specifications cover network architecture, training, meta-optimization, and hyperparameterization.

A.1. Architecture

The convolutional neural network architecture is specified in Table 5.

Table 5 specifies the model architecture.

A.2. Data augmentation

The data augmentation procedure corrupts each input through color transformations, scaling, random cropping, horizontal reflection, and pixel dropout. The CNN architecture uses convolution layers followed by ReLU nonlinearities.

Data augmentation: Each input is corrupted using several augmentation procedures described in the section.
Architecture: The CNN architecture is described as generic, with each convolution layer followed by a ReLU nonlinearity.
Data augmentation: HSV augmentation shifts hue, saturation, and value and can stretch saturation and value using sampled global constants.
Data augmentation: Inputs are randomly scaled by a factor sampled from a uniform range.
Data augmentation: Translation augmentation randomly crops each input to 27 × 27.
Data augmentation: Each input is horizontally reflected with probability 0.5.
Data augmentation: Pixel dropout independently drops each input element with random probability D0.

A.3. Initialization and training procedure

The model uses variance-preserving Gaussian initialization, stochastic gradient training with tuned optimization and regularization parameters, and dropout-based averaging for evaluation.

Initialization and training procedure: Convolution-layer weights are initialized with zero-mean Gaussian noise scaled by σ√F_m, while the input layer receives a separate σ_I hyperparameter.This parametrization is chosen to make activation variances invariant to filter dimensionality.
Initialization and training procedure: The model is trained with minibatch stochastic gradient descent and momentum, tuning the momentum and learning rate before scheduled annealing and termination after 200 epochs.The learning rate is reduced by a factor of 0.1 at epochs 130 and 190.
Initialization and training procedure: Weight decay regularizes all layers, while dropout is applied after max-pooling layers with separate rates D1 and D2.The dropout rates are tuned independently for the corresponding max-pooling outputs.
Initialization and training procedure: Performance is evaluated by averaging log-probability predictions over 100 samples from the input-corruption and unit-dropout distributions.The averaging incorporates both input-corruption samples and dropout masks.

B. Additional figures for image caption generation

The image-captioning experiments visualize Bayesian optimization over multimodal language-model hyperparameters, showing how validation BLEU-4 evolves while the search balances exploration and exploitation.

Additional figures for image caption generation: Figure 5 tracks validation BLEU-4 for hyperparameter configurations over optimization iterations using a planar histogram and a completion-order scatter plot.Histogram shading indicates bin counts, the current best score is traced in black, and constraint violations have validation scores of 0.
Additional figures for image caption generation: The visualizations illustrate Bayesian optimization’s tradeoff between visiting unexplored regions and focusing on promising regions.The scatter plot orders experiments by when they finished, while the histogram summarizes score distributions across configurations.
Hyperparameter search space: The log-bilinear model is optimized over 11 hyperparameters covering architecture, context, optimization, representation, regularization, and factorization choices.These include model variant, context size, learning rate, momentum, batch size, hidden-layer size, embedding size, dropout, context decay, word decay, and factors.
Hyperparameter search space: The LBL’s multiplicative variant uses a factored weight tensor, making the factors hyperparameter relevant only to that model.The additive variant instead incorporates image features through an additive bias term.
Hyperparameter search space: Learning rate, context decay, and word decay are optimized in log-space, with transformed values exponentiated before training or spanning several orders of magnitude.The learning rate is exponentiated before entering the training procedure; context and word decay are likewise optimized in log-space.
Hyperparameter search space: Table 6 specifies the hyperparameterization scheme and the optimal configurations found, while Table 7 gives the corresponding multimodal neural-language-model specification.Table 7 defines log-space transformations, parameter ranges, and option sets used in the search.