Source-linked AI summary
Benchmarking the Performance of Bayesian Optimization across Multiple Experimental Materials Science Domains
Qiaohao Liang, Aldair E. Gongora, Zekun Ren, Armi Tiihonen, Zhe Liu, Shijing Sun, James R. Deneault, Daniil Bash, Flore Mekki-Berrada, Saif A. Khan, Kedar Hippalgaonkar, Benji Maruyama, Keith A. Brown, John Fisher, Tonio Buonassisi
TL;DR
The efficiency of Bayesian Optimization across broad experimental materials domains had been insufficiently evaluated. This work benchmarks BO across five experimental materials datasets and finds that Random Forests perform comparably to Gaussian Processes with ARD, while both outperform GP without ARD.
Problem
Bayesian Optimization performance had not been evaluated across a broad array of experimental materials systems, and missing performance evidence could slow advanced algorithm development.
Method
The study benchmarks BO across five experimental materials science datasets using a pool-based active learning framework.
Results
Random Forests compete with Gaussian Processes using ARD kernels, and both outperform commonly used GP without ARD when paired with the same acquisition functions.
Takeaways & Limitations
The comparison highlights surrogate-model selection and the differing implicit assumptions of Random Forests and Gaussian Processes in materials optimization.
Takeaways & Limitations
The benchmarking framework might have given Random Forests a slight advantage by discretizing the materials domain.
Abstract
from arXiv · showhide
In the field of machine learning (ML) for materials optimization, active learning algorithms, such as Bayesian Optimization (BO), have been leveraged for guiding autonomous and high-throughput experimentation systems. However, very few studies have evaluated the efficiency of BO as a general optimization algorithm across a broad range of experimental materials science domains. In this work, we evaluate the performance of BO algorithms with a collection of surrogate model and acquisition function pairs across five diverse experimental materials systems, namely carbon nanotube polymer blends, silver nanoparticles, lead-halide perovskites, as well as additively manufactured polymer structures and shapes. By defining acceleration and enhancement metrics for general materials optimization objectives, we find that for surrogate model selection, Gaussian Process (GP) with anisotropic kernels (automatic relevance detection, ARD) and Random Forests (RF) have comparable performance and both outperform the commonly used GP without ARD. We discuss the implicit distributional assumptions of RF and GP, and the benefits of using GP with anisotropic kernels in detail. We provide practical insights for experimentalists on surrogate model selection of BO during materials optimization campaigns.
Introduction
The paper addresses the limited quantitative benchmarking of Bayesian Optimization across diverse experimental materials systems. It evaluates BO configurations across five datasets and derives practical guidance for surrogate-model selection.
- Research gap: Few studies have quantitatively analyzed BO acceleration, enhancement, or sensitivity to surrogate-model and acquisition-function choices.
- Benchmark scope: The study benchmarks BO across five experimental materials datasets spanning nanotube polymer blends, silver nanoparticles, perovskites, and additively manufactured polymer structures and shapes.
- Benchmarking framework: The benchmarking framework uses pool-based active learning and acceleration and enhancement factors to compare BO against random sampling.
- Main finding: Random Forests can compete with GP using ARD anisotropic kernels, while both outperform GP without ARD when paired with the same acquisition functions.
- Practical implications: The analysis connects surrogate models’ implicit distributional assumptions with practical guidance for selecting BO algorithms in materials optimization campaigns.
- Resources: The authors provide open-source benchmarking code and datasets to support future algorithmic development.
Results
Across five experimental materials datasets, BO performance varied with surrogate model, acquisition function, and experiment budget. GP with ARD and RF generally outperformed GP without ARD, while LCB1 provided strong performance and different surrogates often followed distinct optimization paths.
- Dataset benchmark: The benchmark spans five materials datasets with three to five input features, one optimization objective, and tens to hundreds of data points.The datasets cover compositions and synthesis-processing parameters from diverse autonomous experimental systems.
- Surrogate models: RF initially performs best at low learning cycles, but GP with ARD leads after Top% = 0.67 and reaches higher targets with fewer experiments.On the crossed barrel case, GP with ARD and RF reached the top 5% target in approximately 30 of 600 candidates, compared with about 90 for GP without ARD.
- Surrogate models: Both RF and GP with ARD outperform GP without ARD, while RF and GP with ARD show comparable performance across the investigated datasets.Their performance distributions overlap substantially across five datasets, supporting similar aggregate effectiveness.
- Acquisition functions: LCB1 outperforms more exploration-heavy LCBλ variants and EI, whereas PI generally performs worse than EI.The paper attributes PI’s weaker performance partly to considering improvement likelihood without considering the magnitude of improvement.
- Performance metrics: BO enhancement and acceleration factors reach up to 16× before diminishing returns emerge deeper in pool-based optimization campaigns.Enhancement and acceleration initially remain small while surrogate models gain accuracy, then decline as information gains diminish.
- Optimization paths: GP with ARD and RF can achieve comparable outcomes through substantially different design-space paths, with Jaccard similarity around 0.6–0.65 at Top% = 0.8.The relative advantage can depend on experiment budget and task: RF excels earlier, while GP with ARD reaches Top% = 0.8 with fewer experiments.
Discussion
Across five experimental materials datasets, RF performs comparably to GP with ARD and both outperform GP without ARD. The comparison links these outcomes to differing distributional assumptions, computational costs, and the usefulness of anisotropic GP kernels.
- Model assumptions: GPs rely on stronger distributional assumptions, while RF predictions are empirical and comparatively distribution-free, affecting predicted means, uncertainties, and experiment selection.These differences persist even when the surrogate models use the same acquisition function.
- Computational cost: RF training is faster than GP training in the benchmark, with average running-time ratios t_RF : t_GP : t_GP ARD = 1 : 1.32 : 1.54.The stated general complexities are t_RF = O(nlog(n) · n_dim · n_tree) and t_GP = O(n^3 + n^2 · n_dim).
- Surrogate-model comparison: RF is a capable BO surrogate whose performance across five datasets is consistently comparable to GP and can match GP with ARD.The paper attributes this partly to RF predictions being empirical estimates from an ensemble of decision trees.
- Surrogate-model comparison: GP with anisotropic kernels removes the performance gap between RF and GP, whereas GP without ARD performs relatively worse.The reported improvement is associated with using independent kernel parameters for feature dimensions.
- Benefits of GP ARD: ARD lengthscales identify relevant input directions and indicate suitable ranges and sampling densities for subsequent experiments.The paper therefore highlights GP with anisotropic kernels as both a generalizable surrogate and a source of feature-relevance information.
Methods
The study uses pool-based Bayesian optimization that alternates surrogate prediction, acquisition-based experiment selection, and model updating. Gaussian processes and random forests provide predictive estimates, while EI, PI, and LCB guide selection under exploration–exploitation trade-offs.
- Surrogate models: For Gaussian processes, a prior is constructed from collected observations and used to derive posterior predictions for a new observation.The GP covariance matrices are formed using either isotropic or anisotropic kernels, and predictive standard deviation comes from the covariance matrix diagonal.
- Surrogate models: For random forests, predictions are represented through the objective-value predictions of individual decision trees in the forest.The text notes that alternative aggregation methods, such as the median, could be examined in future studies.
- Acquisition functions: The study evaluates expected improvement, probability of improvement, and lower confidence bound acquisition functions.LCB uses an adjustable ratio between exploitation and exploration.
- Optimization workflow: Two initial experiments are randomly drawn without replacement from the original pool to initialize the collected dataset.The initial collection is then used to begin sequential optimization.
- Optimization workflow: At each planning stage, the surrogate estimates the predicted mean and standard deviation for every remaining experimental action.These estimates are inputs to the acquisition function.
- Optimization workflow: The next experiment is selected by maximizing the acquisition function, after which its observed outcome is added to the collection and incorporated into the surrogate model.Planning and inference alternate until no undiscovered data points remain.
- Benchmarking: Benchmarking includes statistical baselines and compares overlap between optimization paths that begin from the same two initial data points.Identical paths yield maximum overlap, whereas drastically different paths produce the least overlap.