Source-linked AI summary
Input Warping for Bayesian Optimization of Non-stationary Functions
Jasper Snoek, Kevin Swersky, Richard S. Zemel, Ryan P. Adams
TL;DR
Bayesian optimization needs surrogate models that can represent non-stationary objectives, which standard stationary Gaussian processes model poorly. The paper learns Beta-CDF input warpings, extends them to multi-task optimization, and reports better results faster and more reliably across challenging benchmarks. Its scope includes mild input-dependent length-scale variation, and one benchmark comparison leaves the performance discrepancy with random forests unresolved.
Problem
Stationary Gaussian-process regression struggles with non-stationary functions, although such functions commonly arise in Bayesian optimization of machine-learning hyperparameters.
Method
The method automatically learns bijective input transformations from Beta cumulative distribution functions while marginalizing over their shape parameters, including a multi-task extension.
Results
Across benchmark and machine-learning optimization tasks, modeling non-stationarity improves Bayesian optimization, with better results reached in fewer function evaluations than Snoek et al. (2012).
Takeaways & Limitations
Learned warpings provide an interpretable way to analyze non-stationarities while improving Bayesian optimization performance across the reported tasks.
Takeaways & Limitations
The approach targets mild input-dependent length-scale variations, and the performance discrepancy between Gaussian processes and random forests remains unresolved.
Abstract
from arXiv · showhide
Bayesian optimization has proven to be a highly effective methodology for the global optimization of unknown, expensive and multimodal functions. The ability to accurately model distributions over functions is critical to the effectiveness of Bayesian optimization. Although Gaussian processes provide a flexible prior over functions which can be queried efficiently, there are various classes of functions that remain difficult to model. One of the most frequently occurring of these is the class of non-stationary functions. The optimization of the hyperparameters of machine learning algorithms is a problem domain in which parameters are often manually transformed a priori, for example by optimizing in "log-space," to mitigate the effects of spatially-varying length scale. We develop a methodology for automatically learning a wide family of bijective transformations or warpings of the input space using the Beta cumulative distribution function. We further extend the warping framework to multi-task Bayesian optimization so that multiple tasks can be warped into a jointly stationary space. On a set of challenging benchmark optimization tasks, we observe that the inclusion of warping greatly improves on the state-of-the-art, producing better results faster and more reliably.
1. Introduction
Bayesian optimization depends on surrogate models that accurately represent the objective, but stationary Gaussian processes struggle with realistic non-stationary functions. The paper introduces automatically learned input warpings, extends them to multiple tasks, and reports faster, more reliable optimization.
- Motivation: Stationary Gaussian processes assume translation-invariant covariance, limiting their ability to model objectives whose length scales vary across the input space.This is especially problematic for machine-learning hyperparameters, where poor settings may behave similarly while good regimes are sensitive to small changes.
- Input warping: The paper learns bijective input warpings using Beta cumulative distribution functions to remove major non-stationary effects.The method marginalizes over warping shape parameters and can represent transformations such as logarithmic, exponential, and sigmoidal mappings.
- Multi-task extension: The warping framework extends to multi-task Bayesian optimization by mapping multiple tasks into a jointly stationary space.This lets the tasks be warped onto one another to use their shared structure.
- Empirical results: On continuous-valued benchmark problems, the method outperforms all previous state-of-the-art algorithms.Across four challenging machine-learning optimization tasks, it also consistently reaches a better result in fewer function evaluations than Snoek et al. (2012).
- Empirical results: The empirical study identifies modeling non-stationarity as an important component of effective Bayesian optimization.The reported improvements are both significant and broadly observed across the benchmark and machine-learning tasks.
2. Background and Related Work
Gaussian processes provide flexible function priors for Bayesian optimization, but commonly used stationary kernels and related models constrain how realistic functions are represented. The paper situates input warping alongside multi-task GP modeling and Bayesian optimization with expected improvement.
- Gaussian Processes: Gaussian processes define distributions over functions, with predictive means and uncertainties that can be computed efficiently after conditioning on observations.
- Gaussian Processes: Common covariance functions depend on input differences and are therefore invariant to translations, making them stationary.
- Related Work: Prior work models non-stationarity using multiple Gaussian processes, non-stationary covariance functions, latent-space projections, or output-space transformations.
- Multi-Task Gaussian Processes: Multi-task Gaussian processes share information across correlated datasets, often by deriving each task from a transformed common latent function.
- Bayesian Optimization: Bayesian optimization uses a cheap probabilistic surrogate and proxy optimization to select queries for an expensive black-box objective; this work uses a GP with expected improvement.
3. Input Warping
Input warping models non-stationarity by applying learned bijections before the Gaussian-process kernel, with Beta CDFs providing flexible, interpretable transformations. Priors over Beta shape parameters encode expected warping forms, while hierarchical marginalization allows the transformations to be learned; the framework also supports task-specific warpings.
- Input Warping: Log-normal priors let users encode beliefs about non-stationarity, including slight warping, approximate exponential or logarithmic forms, and sigmoidal contraction.A zero-mean prior centers the model on the identity transformation; the empirical analysis uses variance 0.75.
- Input Warping: The method applies monotonic input transformations before the covariance function, turning non-stationary behavior into a form modeled by a Gaussian process.The kernel is evaluated as K(w(x), w(x̃)), where w is a vector-valued bijection.
- Input Warping: Each input dimension is warped with a Beta CDF whose positive shape parameters determine the transformation’s unique shape.The Beta CDF maps [0, 1] to [0, 1] and supports dimension-wise transformations.
- Input Warping: The Beta-CDF family can represent logarithmic, exponential, sigmoidal, and other monotonic warpings while remaining concisely parameterized.The choice is motivated by expressive monotonic transformations and interpretable parameters.
- Integrating over warpings: A hierarchical Bayesian model places priors over the warping parameters and integrates them out using slice-sampling-based MCMC.This avoids committing to one explicit transformation function.
- Multi-Task Warping: Multi-task warping allows different tasks to map into a jointly stationary space so their shared structure can be exploited despite task-specific input relationships.This is motivated by task differences such as dataset size changing optimal hyperparameter settings.
4. Empirical Analyses
The empirical analyses compare warped and unwarped Gaussian-process Bayesian optimization on machine-learning hyperparameter tasks, continuous benchmarks, and multi-task transfer settings. Across these experiments, learned warpings improve convergence, reliability, or transfer efficiency, while also producing interpretable task- and parameter-specific transformations.
- Experimental Analyses: The study conducts three experiments covering stationary-GP comparisons, continuous HPOLib benchmarks, and multi-task Bayesian optimization.The evaluation uses GP EI MCMC with and without warping and includes machine-learning hyperparameter problems.
- Comparison to Stationary GPs: Warped Bayesian optimization greatly improves convergence over stationary Gaussian processes across four challenging machine-learning hyperparameter problems.On the higher-dimensional convolutional-network problem, it consistently converges to a better solution.
- Learned Warpings: Learned warpings are non-trivial: they are logarithmic-like for several logistic-regression dimensions, absent for one dimension, and intuitive for convolutional-network learning rates.The learning-rate posterior becomes more extreme and certain as observations accumulate, while dropout can yield surprising transformations.
- HPOLib Continuous Benchmarks: On continuous HPOLib benchmarks, input warping makes Gaussian-process optimization at least as good as every compared method and often better.Compared methods include SMAC, TPE, and Spearmint; reported results use ten runs with average validation loss and standard deviation.
- HPOLib Continuous Benchmarks: Warping substantially reduces standard deviations and, in many cases, the number of function evaluations needed to solve benchmark problems.The paper characterizes the resulting performance as more reliable and faster on many instances.
- Multi-Task Warping: Warped multi-task Bayesian optimization outperforms unwarped multi-task and single-task optimization in logistic regression and online LDA transfer experiments.For logistic regression, unwarped multi-task optimization appears stuck in a local minimum, whereas warped optimization escapes by the 20th evaluation.
- Multi-Task Warping: Task-specific warpings align with dataset-scale effects: USPS favors higher L2 penalties than MNIST, while smaller-task training can favor more epochs, smaller minibatches, or higher learning rates.These patterns agree with the reported relationship between dataset size and hyperparameter settings.
5. Conclusion
The paper presents input warping as an efficient way to model non-stationarity in Gaussian-process Bayesian optimization. Experiments report faster, higher-quality, and more reliable optimization, while learned transformations can also be inspected post hoc.
- Input warping uses Beta-distribution cumulative distribution functions to learn bijective transformations that model mild input-dependent length-scale variation.The approach automatically infers transformation forms while remaining computationally efficient.
- The method significantly outperforms Snoek et al. (2012) in evaluations to convergence, value reached, and solution reliability.
- On the continuous HPOLib benchmark subset, input warping performs substantially better than state-of-the-art baselines.
- Learned transformations can be analyzed post hoc, providing interpretable information about the optimized parameter space.
- The paper situates these findings within a broader concern that violated Gaussian-process assumptions can nullify their modeling and computational benefits.