PaperlayerView exact revision on arXiv

Source-linked AI summary

Practical Bayesian Optimization for Model Fitting with Bayesian Adaptive Direct Search

Luigi Acerbi, Wei Ji Ma

arXiv:1705.04405v2stat.MLq-bio.NCq-bio.QM

TL;DR

Model fitting for simulated or numerically approximated models requires optimization over costly, noisy black-box landscapes. The paper develops BADS, a hybrid of local Bayesian optimization and direct search, and benchmarks it against derivative-free optimizers. Across real model-fitting problems, BADS consistently matches or outperforms the tested alternatives, including vanilla BO, while its scope is limited by surrogate quality and dimensionality.

  • Problem

    Model fitting can require costly exploration of rough or noisy nonconvex landscapes, while BO’s suitability against standard optimizers and its acceptable overhead in moderate-cost regimes remain unclear.

  • Method

    BADS combines MADS direct search with local Gaussian-process Bayesian optimization and is evaluated alongside many derivative-free optimizers on real neuroscience model-fitting problems.

  • Results

    BADS consistently performs on par with or better than tested optimizers across model-fitting problems, including vanilla BO.

  • Takeaways & Limitations

    Hybrid Bayesian optimization can benefit model fitting beyond very costly black-box functions, with BADS offering a practical general-purpose tool.

  • Takeaways & Limitations

    BADS performance generally deteriorates in high dimensions or for pathological objective structure, and the authors recommend models with up to approximately 15 variables.

Abstract

from arXiv · show

Computational models in fields such as computational neuroscience are often evaluated via stochastic simulation or numerical approximation. Fitting these models implies a difficult optimization problem over complex, possibly noisy parameter landscapes. Bayesian optimization (BO) has been successfully applied to solving expensive black-box problems in engineering and machine learning. Here we explore whether BO can be applied as a general tool for model fitting. First, we present a novel hybrid BO algorithm, Bayesian adaptive direct search (BADS), that achieves competitive performance with an affordable computational overhead for the running time of typical models. We then perform an extensive benchmark of BADS vs. many common and state-of-the-art nonconvex, derivative-free optimizers, on a set of model-fitting problems with real data and models from six studies in behavioral, cognitive, and computational neuroscience. With default settings, BADS consistently finds comparable or better solutions than other methods, including `vanilla' BO, showing great promise for advanced BO techniques, and BADS in particular, as a general model-fitting tool.

1 Introduction

Model fitting can require costly derivative-free searches over noisy, nonconvex black-box objectives, while the appropriate optimizer is often unclear. The paper addresses this gap with BADS, a low-overhead hybrid BO method benchmarked on real neuroscience model-fitting problems.

  • Complex neuroscience models often require numerical approximations or stochastic simulations, making maximum-likelihood fitting costly and potentially noisy.
  • Choosing an inadequate derivative-free optimizer can hinder progress, limit model complexity, and cast doubt on findings.
  • Traditional BO targets more expensive problems than typical behavioral-model evaluations, creating stricter overhead and evaluation-budget requirements for model fitting.Typical model evaluations may cost 0.1-10 s, with hundreds to thousands of allowed evaluations.
  • BADS combines BO with direct search to achieve competitive performance at small computational cost.
  • The benchmark evaluates BADS and many common optimizers on real-data model-fitting problems from behavioral, cognitive, and computational neuroscience.
  • BADS combines MADS with local Gaussian-process BO and efficiency heuristics, while traditional surrogate methods may require large overheads or problem-specific tuning.

2 Optimization frameworks

The paper presents direct search and Bayesian optimization as complementary frameworks for difficult black-box objectives. MADS supplies mesh-based exploration and convergence safeguards, while GP surrogates guide exploitation–exploration decisions through acquisition functions.

  • Mesh adaptive direct search (MADS): MADS searches mesh neighborhoods around the incumbent using iteration-dependent directional steps and can incorporate additional search strategies.
  • Mesh adaptive direct search (MADS): The MADS mesh consists of evaluated points combined with a mesh size and fixed viable search directions, including positive and negative coordinate directions.
  • Mesh adaptive direct search (MADS): Each MADS iteration has a SEARCH stage for strategy-proposed mesh points and an optional POLL stage when SEARCH fails to improve the objective.
  • Mesh adaptive direct search (MADS): POLL evaluates opportunistically constructed directions and ensures theoretical convergence to a local stationary point for nonsmooth functions.
  • Mesh adaptive direct search (MADS): Successful iterations retain or enlarge the mesh, whereas unsuccessful iterations shrink it until a stopping criterion is met.
  • Bayesian optimization: BO uses a Gaussian-process surrogate and an acquisition function to balance promising low predicted values against uncertain regions.
  • Bayesian optimization: BADS uses a GP lower confidence bound acquisition function, with ν = 0.2 and δ = 0.1 as recommended settings.

3 Bayesian adaptive direct search (BADS)

BADS alternates fast local Bayesian SEARCH steps with systematic MADS POLL exploration, using a local GP whose training set and uncertainty handling adapt to the optimization state. Its implementation standardizes bounds, updates GP hyperparameters periodically, and modifies sampling for noisy objectives.

  • BADS overview: BADS alternates local BO SEARCH steps with systematic mesh-based POLL exploration, using POLL when SEARCH repeatedly fails.
  • Problem specification: The algorithm accepts starting points, hard bounds, optional plausible bounds, constraints, and additional options.
  • Algorithm: BADS initializes mesh and poll sizes, evaluates an initial design, repeatedly updates the GP, searches, polls if needed, and stops at budget, mesh-size, or stalling criteria.
  • Problem specification: Variables are constrained by hard and plausible bounds, transformed to log space when appropriate, and rescaled into the standardized box [-1, 1]^D.
  • Initial design: The initial design adds D Sobol points to the starting point and tests repeated evaluations when the objective’s stochasticity is unspecified.
  • Gaussian-process model: The default GP uses a constant mean, ARD rational-quadratic kernel, and lower-confidence-bound acquisition function, with hyperparameters selected by MAP estimation and periodically refit.
  • Gaussian-process model: The GP training set locally approximates the objective around the incumbent, prioritizing nearby cached points and rebuilding when the incumbent moves.
  • Search stage: SEARCH uses repeated local BO proposals, while its oracle samples candidate batches around the incumbent and selects candidates by acquisition value.

4 Experiments

The experiments benchmarked BADS against 16 alternative optimizers on artificial functions and six real neuroscience model-fitting studies, using repeated randomized runs under noiseless and noisy conditions. BADS was consistently competitive, while vanilla BO performed poorly across the real problems.

  • Benchmark design: The benchmark included 96 artificial BBOB09 functions across four dimensions and model-fitting objectives from six cognitive and computational neuroscience studies.The real objectives were deterministic for three studies and noisy for three others.
  • Benchmark design: Each algorithm was evaluated in 50 independent runs with randomized starting points and budgets of 500 × D evaluations for noiseless problems and 200 × D for noisy problems.Success was defined by reaching a specified absolute error tolerance from the true optimum or its best estimate.
  • Artificial functions: BADS, fmincon-family methods, and active-update CMA-ES were strongest on noiseless artificial functions, depending on the evaluation budget.CMA-ES became competitive at larger evaluation budgets.
  • Artificial functions: BADS outperformed all other optimizers on heteroskedastic-noise artificial functions, with active CMA-ES variants ranking second.The noise standard deviation increased with distance from the optimum, and homoskedastic results were similar.
  • Real model-fitting problems: Across all real model-fitting problems, BADS performed on par with or better than every tested optimizer, even after accounting for its extra algorithmic cost.CMA-ES or fmincon was typically second-best, but their rankings varied across problems; vanilla BO often performed near random search.

5 Conclusions

The paper presents BADS as a practical hybrid BO toolbox for fitting moderately expensive computational models. Its benchmark results support robust performance across real model-fitting problems, while its applicability is bounded by surrogate quality and dimensionality.

  • Contributions: BADS combines local BO-based SEARCH with a fail-safe, model-free POLL stage to protect against Gaussian-process surrogate failures.The authors attribute its robust performance to alternating between these two stages.
  • Contributions: The associated toolbox is intended to fit moderately expensive computational models out-of-the-box and is available with an interface matching existing optimizers.The package requires no tuning under its default settings.
  • Conclusions: BADS performed as well as or better than widely used and state-of-the-art nonconvex derivative-free methods, including vanilla BO, on real model-fitting problems.The authors present this as evidence that hybrid Bayesian optimization can extend beyond extremely costly black-box functions.
  • Scope and limitations: BADS performance generally deteriorates in high dimensions or on pathological objective structures because its surrogate approximation becomes less effective.The authors recommend pairing BADS with multi-start optimization for models with up to approximately 15 variables and suitable overhead.

Supplementary Material

The supplementary material specifies the Gaussian-process and Bayesian-optimization components used by BADS. It describes posterior calculations, alternative kernels, periodic coordinates, and the local approximation choices underlying the method.

  • Supplement scope: The supplement expands the definitions and implementations of Gaussian processes and Bayesian optimization used in BADS.It also provides fuller algorithmic, benchmark, and numerical-implementation details.
  • Gaussian-process model: The GP posterior is constructed from training points, observed function values, a mean function, a covariance kernel, and independent Gaussian observation noise.Posterior moments are available in closed form at a chosen point.
  • Kernel choices: BADS uses an ARD rational quadratic kernel by default and also considers squared-exponential and Matérn 5/2 kernels.The rational quadratic kernel approaches the squared-exponential kernel as α becomes infinite.
  • Kernel choices: The supplement notes that infinitely differentiable kernels can create extrapolation issues, but this is less problematic for BADS’s local interpolating approximation.The authors report that the rational quadratic kernel performs well empirically.
  • Kernel choices: Periodic coordinates are handled with composite ARD periodic kernels whose periods are determined by the supplied finite hard bounds.The base kernel can be rational quadratic, squared exponential, or Matérn 5/2.

A.2 Construction of the training set

BADS builds a local GP training set around the incumbent and fits its hyperparameters with a MAP objective using data-dependent priors. Additional safeguards address local scale changes, poor fits, and numerical failures.

  • Training-set construction: Training points are selected by increasing length-scale-normalized distance from the current incumbent.The local set contains at least 50 cached points when available, plus up to 10 × D nearby points within a kernel-dependent radius.
  • Hyperparameter fitting: GP hyperparameters are fitted by maximizing their posterior probability under the Gaussian likelihood.The posterior combines the likelihood with a prior over hyperparameters.
  • Hyperparameter fitting: BADS uses bounded independent hyperparameter priors defined from the current training data as an approximate empirical-Bayes procedure.This approach is presented as a faster heuristic than a full hierarchical Bayesian treatment.
  • Adaptive priors: The GP observation-noise prior decreases with poll size, while the GP mean is set to the 90th percentile of current observations to keep exploration local.These choices adapt the surrogate as the search zooms toward smaller scales.
  • Robustness safeguards: If hyperparameter optimization becomes stuck or numerically fails, BADS retries from a prior-informed initialization or retains the previous hyperparameters.The implementation also flags an approximation as inaccurate when the stated diagnostic probability falls below 10^-6.

A.4 Acquisition functions

BADS offers multiple acquisition functions for selecting promising evaluations from a Gaussian-process surrogate, while noting that probability of improvement can overfavor exploitation.

  • Acquisition functions: BADS provides closed-form alternatives to its default GP lower confidence bound, including probability of improvement and expected improvement.These criteria use the GP predictive mean and variance.
  • Probability of improvement: Probability of improvement maximizes the chance of improving on the current best minimum, with ξ controlling an exploration offset.The implementation uses negative PI for consistency with the minimization objective.
  • Probability of improvement: PI can excessively favor exploitation, and selecting ξ to counteract this tendency is difficult.This is presented as a limitation of the acquisition strategy.
  • Expected improvement: Expected improvement measures the expected improvement over the current best minimum and is also available in closed form.BADS uses the negative EI formulation for minimization.

B.1 Problem definition and initialization

BADS defines bounded and optionally constrained optimization problems, initializes a standardized search, and alternates local Bayesian SEARCH with mesh-based POLL exploration.

  • Problem definition: BADS minimizes a possibly noisy black-box objective over hard bounds, optionally restricting solutions through a barrier function with nonpositive violations.Hard bounds may be finite or infinite, while non-bound constraints are represented by c(x) ≤ 0.
  • Problem definition: Plausible bounds identify the region where solutions are expected to lie, while hard bounds define the admissible coordinate limits.Fixed variables and circular dimensions are also supported.
  • Initialization: BADS transforms positive wide-range variables to log space and rescales all variables so plausible bounds map to the standardized box [−1, 1]^D.Function and barrier evaluations are converted back to the original coordinate space.
  • Initialization: The initial design contains x0 plus D Sobol-sequence points projected onto the mesh, while repeated evaluations at x0 can classify objective noise.Users are encouraged to specify whether the objective is noisy because the automatic test can fail.
  • SEARCH and POLL stages: BADS combines an aggressive local SEARCH with a fallback POLL stage, switching after nsearch consecutive failures to find sufficient improvement.The default failure threshold is nsearch = max{D, ⌊3 + D/2⌋}.
  • SEARCH stage: SEARCH candidates are chosen by locally optimizing an acquisition function with an evolutionary heuristic and a covariance matrix, while excluding or projecting infeasible points.The search covariance can use coordinate-wise or weighted-covariance strategies.
  • POLL stage: POLL directions are rescaled using GP length scales and evaluated in acquisition-function order; mesh size expands after success and contracts after failure.A successful SEARCH can skip POLL without changing mesh size.
  • Noisy objective: For noisy objectives, BADS can return GP predictions or average repeated evaluations at the final point to estimate the expected objective value.The repeated-evaluation option uses Nfinal = 10 by default.

C Benchmark

The benchmark evaluates BADS on a large collection of artificial and real optimization problems and compares it with many MATLAB-available optimization methods.

  • Benchmark: The study tests BADS on artificial and real problems and compares its performance with many optimization methods available in MATLAB.The experiments use MATLAB R2015b and R2017a implementations.

C.1 Algorithms

The benchmark compares diverse MATLAB optimizers under standardized domains, tracking performance with error-based measures and accounting for algorithmic overhead and noisy returns.

  • Algorithms: The tested algorithms include freely available methods and MATLAB toolbox methods, classified by noise handling and local or global search behavior.Semi-local methods are identified separately because they may escape local minima but still need multi-start strategies.
  • Algorithms: For noisy problems, BADS, CMA-ES, and bayesopt use uncertainty handling, while only active-set fmincon is tested for the noisy CCN17 subset.The fmincon family was found comparable to random search on those noisy problems.
  • Standardization: Benchmark variables are standardized across methods by mapping plausible regions to [−1, 1]^D and applying log transforms to positive variables spanning over one order of magnitude.Starting points are generated using plausible bounds when available.
  • Performance measures: Deterministic performance is measured by immediate regret, defined as the best value found so far minus the true minimum or its benchmark estimate.A run is successful when this error falls below a specified tolerance.
  • Performance measures: For CCN17 problems without known minima, the reference minimum is the best value found across all algorithms’ runs and approximately 3.75 · 10^5 × D evaluations.The benchmark treats this computational effort as difficult to beat.
  • Performance measures: Effective performance adjusts iteration counts by each algorithm’s fractional overhead, shifting performance curves to reflect computational cost.Algorithm rankings are based on area under the performance curve.
  • Noisy problems: Noisy evaluations are assessed using the true function values at returned solutions, allowing up to three candidate solutions per optimization run.This accounts for algorithms visiting promising neighborhoods but returning different points under noise.

C.3 Alternative benchmark parameters

The benchmark conclusions remain stable under narrower deterministic tolerances and a larger noisy-function evaluation budget. Under these alternative parameters, BADS performs on par with or better than other algorithms.

  • The deterministic benchmark uses ε ∈[0.1, 1] rather than the main text’s wider ε ∈[0.01, 10] range.
  • Vanilla BO’s performance was already abysmal without accounting for its substantial overhead, so the correction was not applied to its results.
  • BADS performs on par with or better than other algorithms under these alternative benchmark parameters.

D Numerical implementation

BADS is implemented as an accessible MATLAB toolbox with tested defaults, no extra toolbox or compilation requirements, and modular components for expert customization. Its Gaussian-process implementation builds on GPML v3.6 with efficiency modifications and bound-constrained hyperparameter optimization.

  • BADS is freely available as a MATLAB toolbox, while a Python version is planned.
  • The toolbox uses a familiar MATLAB optimizer interface and plug-and-play defaults that hide most implementation complexity.It requires neither additional toolboxes nor compiling C/C++ code via mex files.
  • Expert users can customize POLL generation, SEARCH, acquisition functions, and initial design through BADS’s modular architecture.The implementation supports selecting components from a large list and adding new options.
  • The MATLAB Gaussian-process implementation adapts GPML v3.6 with efficiency improvements and added functionality.GP hyperparameters are optimized with fmincon when available, or with a modified bound-constrained GPML minimize function otherwise.
  • GPML v4.0 was released during BADS development and might be supported in future versions.
Loading 1705.04405v2…