Source-linked AI summary
Dealing with Integer-valued Variables in Bayesian Optimization with Gaussian Processes
Eduardo C. Garrido-Merchán, Daniel Hernández-Lobato
TL;DR
Bayesian optimization with Gaussian processes must handle integer-valued inputs despite GP models assuming continuous variables, because naive rounding can misalign suggested and evaluated points or ignore objective constancy. The paper modifies the GP covariance function after rounding integer-valued inputs, and experiments show that this approach finds solutions closer to optimum in fewer evaluations.
Problem
Gaussian-process Bayesian optimization assumes continuous inputs, so integer-valued variables require approximations whose modeling and evaluation choices can impair optimization.
Method
The approach rounds integer-valued inputs and transforms the GP covariance function so the objective is modeled as constant across real-valued intervals sharing an integer rounding.
Results
The proposed approach finds points closer to the optimum with fewer objective evaluations than the basic approach in both noisy and noiseless synthetic experiments.
Takeaways & Limitations
Modeling the objective's constant intervals improves Bayesian optimization performance on problems involving integer-valued variables.
Takeaways & Limitations
The basic rounding approaches can misalign high-acquisition points with actual evaluations or ignore the objective's constant behavior across rounded intervals.
Abstract
from arXiv · showhide
Bayesian optimization (BO) methods are useful for optimizing functions that are expensive to evaluate, lack an analytical expression and whose evaluations can be contaminated by noise. These methods rely on a probabilistic model of the objective function, typically a Gaussian process (GP), upon which an acquisition function is built. This function guides the optimization process and measures the expected utility of performing an evaluation of the objective at a new point. GPs assume continous input variables. When this is not the case, such as when some of the input variables take integer values, one has to introduce extra approximations. A common approach is to round the suggested variable value to the closest integer before doing the evaluation of the objective. We show that this can lead to problems in the optimization process and describe a more principled approach to account for input variables that are integer-valued. We illustrate in both synthetic and a real experiments the utility of our approach, which significantly improves the results of standard BO methods on problems involving integer-valued variables.
1. Background on Bayesian Optimization
Bayesian optimization uses a Gaussian-process model and an acquisition function to choose expensive, noisy evaluations efficiently. The GP provides predictive mean and uncertainty, while expected improvement guides the next evaluation toward promising candidates.
- BO optimizes expensive, noisy real-valued functions without requiring an analytical expression or gradient computation.The objective is observed through noisy evaluations y = f(x) + ϵ.
- At each iteration, BO fits a probabilistic model to collected observations and selects the next point by maximizing an acquisition function.The process repeats after each new observation, and the GP predictive mean can later be optimized to identify a solution.
- BO is effective because evaluating and maximizing the acquisition function is much cheaper than evaluating the objective itself.This enables careful selection of evaluations while minimizing the number of expensive objective-function calls.
- The GP predictive distribution represents the objective at a candidate point with a mean µ(x) and variance σ2(x).These quantities are computed from observed values, prior covariances, noise variance, and a covariance matrix.
- Expected improvement measures the average utility of evaluating each candidate relative to the best value observed so far.For minimization, the utility is u(y) = max(0, ν − y), where ν is the current best observed value.
2. Dealing with Integer-valued Variables
Integer-valued inputs violate the continuous-variable assumptions of standard Gaussian-process Bayesian optimization, while common rounding strategies can misalign acquisition recommendations with actual evaluations or ignore constant regions.
- Continuous assumptions: Standard Gaussian-process Bayesian optimization assumes continuous inputs, so integer constraints can assign probability to invalid values and produce invalid candidate evaluations.The issue is especially relevant to mixed-variable machine-learning hyperparameter optimization, including learning rates, layer counts, and neurons per layer.
- Comparison: Figure 1 illustrates distinct strategies for handling integer-valued variables, including naive rounding, wrapper rounding, and the proposed modeling approach.The proposed approach is depicted as the third strategy in the figure.
- Naive rounding: Naively optimizing the acquisition function over real values and then rounding can mismatch high-acquisition locations with evaluated points.It can also repeatedly evaluate a previously sampled point when rounding leaves the acquisition function unchanged.
- Wrapper rounding: Rounding inside the objective wrapper aligns high-acquisition locations with evaluations and encourages evaluations at different locations.However, the Gaussian process still ignores that the rounded objective is constant across intervals mapping to the same integer, which can be sub-optimal.
3. Proposed Approach
The proposed approach modifies the Gaussian-process covariance function so integer-valued inputs are rounded before covariance computation. This makes the model represent constant behavior within shared rounding intervals and better captures the objective structure.
- Covariance transformation: The method transforms inputs by rounding integer-valued variables to the nearest integer before evaluating the covariance function.The resulting covariance is k′(x_i, x_j) = k(T(x_i), T(x_j)).
- Modeling constant regions: The transformed covariance makes points that round to the same integer share the objective’s constant behavior and identical uncertainty within those intervals.This follows from assigning perfect correlation when transformed points coincide.
- Uncertainty reduction: After one measurement in each rounded interval, the proposed model’s uncertainty about the objective goes to zero.The paper presents this as a beneficial property illustrated in Figure 1.
- Model comparison: Figure 2 shows that the proposed posterior captures constant behavior along the integer dimension, whereas a standard Gaussian process does not.The comparison uses one continuous dimension and one dimension taking values in {0, 1, 2, 3, 4}.
4. Experiments
The experiments compare the proposed integer-aware BO approach with a basic approach on synthetic objectives and gradient-boosting hyperparameter optimization. Across these settings, the proposed method reaches better solutions with fewer objective evaluations.
- Experimental setup: The study compares the proposed BO approach with a basic approach using Spearmint, Matérn covariance functions, slice-sampled GP hyperparameters, and expected improvement.The experiments cover synthetic objectives and a real gradient-boosting task.
- Synthetic experiments: Synthetic objectives include mixed continuous and integer inputs in two- and four-dimensional settings, with noisy and noiseless evaluations.Integer domains include {0, 1, 2} and {0, 1, 2, 3}.
- Synthetic experiments: The proposed approach finds points closer to the optimum with fewer objective evaluations in both noisy and noiseless synthetic experiments.The reported noise variances are 0.01 and 0.001, respectively.
- Digits experiment: In the digits experiment, the methods optimize learning rate and five possible tree depths for a 100-tree gradient-boosting ensemble.Performance is evaluated using validation-set test log-likelihood over 100 iterations and 100 repetitions.
- Digits experiment: On the digits dataset, the proposed approach finds ensembles with better validation prediction properties using fewer objective evaluations than the basic approach.The experiment uses 70% of the data for training and 30% for validation.
5. Conclusions
The paper proposes an integer-aware GP covariance approach for Bayesian optimization and evaluates it on synthetic and real problems. The approach produces solutions closer to the optimum in fewer iterations than a basic treatment of integer-valued variables.
- Conclusion: The method rounds real values to integers and modifies the GP covariance so the objective remains constant across real values mapping to the same integer.This addresses integer-valued variables within GP-based Bayesian optimization.
- Conclusion: Synthetic and real experiments show that the proposed approach has better modeling properties and yields better optimization results than a basic approach.The evaluation covers problems involving integer-valued variables.
- Conclusion: The proposed BO method finds solutions closer to the optimum in fewer iterations.This is the paper's stated overall conclusion across the evaluated problems.