PaperlayerView exact revision on arXiv

Source-linked AI summary

Enhancing Bayesian Optimization and Active Learning Through Kernel Diversity

Heng Zhang, Haotian Xiang, Qin Lu, Konstantinos D. Polyzos, Tara Javidi

arXiv:2608.24721v1cs.LGcs.AI

TL;DR

BO and Bayesian active learning depend on hyperparameter selection, yet point estimates can suffer from misspecification and fully Bayesian MCMC treatments are computationally expensive. KENDO replaces hyperparameter sampling with an adaptively weighted kernel ensemble and disagreement-aware acquisition, yielding BO, active-learning, and multi-objective variants. Across reported experiments, the approach combines competitive or superior performance with substantial computational savings, including up to 5× for BO and 27× for active learning.

  • Problem

    Hyperparameter and kernel selection can cause model misspecification in BO and Bayesian active learning, while fully Bayesian MCMC treatments are computationally expensive.

  • Method

    KENDO combines an explicit, adaptively weighted kernel ensemble with disagreement-aware acquisition strategies for BO, active learning, and random-scalarization multi-objective optimization.

  • Results

    KENDO-BO and KENDO-AL achieve competitive or superior reported performance over MCMC-based approaches while reducing computational cost by up to 5× and 27×, respectively.

  • Takeaways & Limitations

    Kernel diversity and adaptive weighting provide a shared self-correcting mechanism for optimization and predictive active learning, with a natural random-scalarization extension to multiple objectives.

  • Takeaways & Limitations

    The framework uses moment matching that may underestimate uncertainty when kernel predictions are highly disparate, and its kernel dictionary requires manual specification.

Abstract

from arXiv · show

Hyperparameter selection remains a key challenge in Bayesian optimization (BO) and Bayesian active learning (AL), as model misspecification can lead to suboptimal performance, while more accurate fully Bayesian treatments typically rely on computationally expensive MCMC sampling. This paper proposes a unified framework, KENDO (Kernel ENsemble Disagreement-aware Operator), that integrates Ensemble Gaussian Processes (EGP) with disagreement-aware acquisition strategies. The central idea is to replace hyperparameter sampling with a kernel ensemble and adaptive Bayesian weighting, combined with disagreement-aware acquisition strategies. Within this unified framework, we instantiate KENDO-BO for BO and KENDO-AL for Bayesian AL, demonstrating that both arise from a common self-correcting mechanism with task-specific acquisition objectives. We further extend the approach to multi-objective optimization via random scalarization that preserves the single-optimizer conditioning structure. Thorough numerical tests on synthetic and real-world benchmarks across single-objective optimization, multi-objective optimization, and active learning demonstrate that (i) KENDO-BO achieves competitive or superior optimization performance compared to state-of-the-art methods while reducing computational overhead by up to $5\times$ and (ii) KENDO-AL achieves superior predictive calibration over MCMC-based active learning baselines with up to $27\times$ speedup.

1 Introduction

The introduction identifies kernel-family choice and hyperparameter selection as sources of GP misspecification in BO and AL, then proposes KENDO to combine kernel ensembles with disagreement-aware, self-correcting acquisition. Its BO, active-learning, and multi-objective variants target strong performance while reducing MCMC-related computation.

  • KENDO framework: KENDO replaces SCoreBO hyperparameter sampling with an explicit kernel ensemble, preserving uncertainty quantification through kernel diversity and adaptive weighting.This removes MCMC overhead while retaining a mechanism for model uncertainty.
  • KENDO framework: Its acquisition function measures disagreement between marginal and optimizer-conditioned ensemble posteriors to balance exploration in promising regions with model selection.The construction adapts optimizer conditioning and disagreement to the ensemble setting.
  • Extensions: Random scalarization extends KENDO-BO to multi-objective optimization while preserving single-optimizer conditioning and covering the Pareto front through diverse scalarization directions.The extension avoids the computational burden of multiobjective solvers.
  • Empirical scope: KENDO-AL generalizes the ensemble mechanism to Bayesian active learning, while experiments report substantial computational savings without sacrificing optimization performance.The introduction reports savings of up to 5× for BO and 27× for active learning.

2 Related Work

Related work frames disagreement as a shared exploration signal in BO and Bayesian active learning, while highlighting the computational cost of Bayesian hyperparameter treatment and open questions in multi-objective hyperparameter learning.

  • GP hyperparameter inference: GP surrogate performance depends on kernel and hyperparameter choices; point estimation is efficient but can produce overconfident predictions, whereas MCMC and variational approaches integrate hyperparameter uncertainty.Scalable alternatives include slice sampling and related approximate treatments.
  • Disagreement-based acquisition: Disagreement-based acquisition places queries where competing posterior beliefs differ most, supporting exploration in both BO and Bayesian active learning.SCoreBO uses optimizer-conditioned posteriors in BO, while BAL methods target uncertainty reduction or prediction-focused objectives.
  • Multi-objective BO: Multi-objective BO methods include hypervolume, ϵ-indicator, information-theoretic, scalarization, and uncertainty-aware acquisitions, but hyperparameter learning remains largely underexplored.Existing approaches often fix hyperparameters or tune them independently for each objective.

3 Preliminaries

BO and BAL use probabilistic surrogate models to select informative queries, but GP performance depends strongly on kernel and hyperparameter choices. Ensemble GPs address kernel selection by combining models with different kernels and Bayesian weights, while disagreement-based acquisition incorporates optimizer-conditioned uncertainty.

  • BO and BAL: BO seeks expensive-function optimizers, whereas BAL learns the target-function mapping across the domain through sequential, uncertainty-aware querying.Both paradigms use probabilistic surrogate posteriors and acquisition functions to choose subsequent points.
  • GP surrogate models: GP surrogate performance depends on kernel choice and hyperparameters, whose point estimation can neglect uncertainty and produce overconfident predictions.Fully Bayesian GP treatments instead maintain a posterior over hyperparameters but require computationally expensive inference.
  • Ensemble GPs: Ensemble GPs automate kernel selection by assigning Bayesian weights to multiple GPs with different kernels under a Gaussian mixture prior.Each component uses hyperparameters estimated by marginal likelihood maximization, and the posterior remains a GP mixture after observing data.
  • Disagreement-based acquisition: SCoreBO uses Hellinger disagreement between marginal and optimizer-conditioned posteriors to balance promising-region exploration with hyperparameter learning.Its hyperparameters and optimizers are sampled from their posterior distributions, so it retains the computational burden of hyperparameter sampling.

4 Exploiting Kernel Diversity to Enhance BO and BAL

KENDO replaces continuous hyperparameter sampling with an explicitly weighted kernel ensemble and uses disagreement-aware acquisition functions for BO and AL. BO conditions on sampled optimizers, whereas AL measures direct ensemble disagreement to guide global function learning.

  • From hyperparameter sampling to kernel ensembles: KENDO replaces SCoreBO’s hyperparameter sampling with a discrete kernel ensemble, targeting kernel-family uncertainty while eliminating MCMC overhead.Each fixed-kernel GP fits its hyperparameters by marginal likelihood, and Bayesian weighting requires M predictive-likelihood evaluations per iteration rather than O(100)–O(1000) MCMC samples.
  • From hyperparameter sampling to kernel ensembles: Moment matching converts the ensemble’s Gaussian mixture posterior into a tractable Gaussian summary whose variance combines within-GP uncertainty and between-GP disagreement.This summary is used as input to the acquisition functions.
  • Optimizer-conditioned acquisition: KENDO-BO samples functions from each kernel GP, locates sampled optimizers, conditions on them, and aggregates disagreement between marginal and conditional posteriors.The resulting acquisition value is large where optimizer knowledge would substantially revise the ensemble prediction.
  • Active-learning acquisition: KENDO-AL removes optimizer conditioning and prioritizes locations where kernel identity has maximal impact on predictions.Its likelihood-weighted disagreement captures outputscale uncertainty for global exploration and lengthscale uncertainty for repeated noise-estimation queries.

5 KENDO for MOBO

KENDO extends to multi-objective Bayesian optimization by maintaining objective-specific kernel ensembles and replacing Pareto-front conditioning with random scalarization. This preserves single-optimizer conditioning while using diverse scalarization directions to cover the Pareto front without multi-objective solvers.

  • 5 KENDO for MOBO: MOBO optimizes multiple possibly conflicting objectives whose solutions form a Pareto set and are commonly evaluated by hypervolume relative to a reference point.The multi-objective setting therefore requires extending both per-objective modeling and optimizer conditioning.
  • 5 KENDO for MOBO: KENDO maintains independent ensemble Gaussian processes for each objective, allowing different objectives to favor different kernels.This objective-specific ensemble construction adapts kernel diversity to heterogeneous objective functions.
  • 5 KENDO for MOBO: A direct extension that conditions on sampled Pareto fronts introduces multiple phantom observations, diffuses the conditioning signal, and yields suboptimal performance.The paper identifies this as a departure from the single-optimizer mechanism underlying SCoreBO.
  • 5 KENDO for MOBO: Random scalarization samples Dirichlet weight vectors, forms scalarized objectives from RFF function paths, and finds one optimizer per kernel and scalarization.Each resulting optimizer supplies the single phantom observation used for per-objective conditioning and Hellinger-distance aggregation.
  • 5 KENDO for MOBO: Diverse scalarization directions provide Pareto-front coverage while retaining single-point conditioning and avoiding multi-objective solvers such as NSGA-II.The framework uses single-objective L-BFGS optimization on the RFF path, and its performance is robust to the Dirichlet concentration choice.

6 Experiments

Experiments evaluate KENDO across single-objective, multi-objective, and active-learning benchmarks, comparing ensemble-based methods with established baselines and ablations. KENDO matches or improves performance while adaptive weighting and ensemble inference reduce computational cost.

  • Single-Objective Optimization: KENDO-BO matches or outperforms all baselines across nine single-objective benchmarks and maintains the top average rank as budget grows.Results are averaged over 25 seeds and three kernel variants; on Borehole, KENDO-BO reaches the log10 regret floor earliest.
  • Single-Objective Optimization: KENDO-BO consistently outperforms EGP-TS, isolating disagreement-based, optimizer-conditioned querying as a source of gains beyond the shared ensemble.Both methods use the same kernel ensemble and weighting scheme.
  • Multi-Objective Optimization: KENDO-MO achieves the lowest log10 hypervolume difference on most of six multi-objective benchmarks and ranks first across the full budget.The largest margins occur on Penicillin and CarSideImpact, whose heterogeneous objectives benefit from per-objective adaptive weighting.
  • Multi-Objective Optimization: KENDO-MO outperforms the Pareto-front-conditioning variant across all six benchmarks, supporting random scalarization with focused single-point conditioning.Conditioning on entire Pareto sets introduces many phantom observations and diffuses the conditioning signal.
  • Active Learning: KENDO-AL achieves the lowest negative MLL on every active-learning benchmark, holds the best aggregate rank, and runs 7.6×–26.9× faster than SAL.The benchmarks span synthetic, simulator, HPO, and engineering-surrogate tasks.
  • Ablation Studies: Adaptive Bayesian weighting outperforms uniform averaging across all three task families, with benefits pronounced where kernel preferences vary across input regions or objectives.The ensemble becomes winner-take-all when one kernel is clearly superior and remains diversified when no kernel dominates.
  • Ablation Studies: KENDO runs 3.1×–5.4× faster than SCoreBO, while KENDO-AL runs 7.6×–26.9× faster than SAL and weight updates add less than 7 ms per iteration.The reported comparisons measure computational efficiency across the evaluated benchmarks.

7 Conclusions

The paper presents KENDO as a unified kernel-ensemble framework for Bayesian optimization and active learning, with a random-scalarization extension for multi-objective optimization. It reports improved performance and substantial computational savings, while identifying approximation and kernel-dictionary limitations.

  • Conclusions: KENDO replaces costly fully Bayesian hyperparameter sampling with an explicit kernel ensemble and adaptive Bayesian weighting for BO and AL.KENDO-BO and KENDO-AL leverage kernel-family diversity as a source of model uncertainty.
  • Conclusions: KENDO-BO and KENDO-AL achieve superior optimization and predictive performance over MCMC-based approaches with substantial computational savings.The conclusions summarize the reported cross-task performance and efficiency results.
  • Limitations: Moment matching may underestimate uncertainty when kernel predictions are highly disparate, and the kernel dictionary requires manual specification.Automatic kernel construction is identified as a possible direction for future work.

Supplementary File

The supplementary file provides the full KENDO-MO procedure, including kernel fitting, random scalarization, function-path optimization, conditioning, and posterior updates. It also specifies the algorithm’s inputs and return value.

  • KENDO-MO Procedure: Algorithm 2 requires a kernel dictionary, scalarization count, objective count, initial data, and optimization budget.The procedure returns a Pareto front from the accumulated dataset.
  • KENDO-MO Procedure: KENDO-MO fits per-objective, per-kernel models before iterating through scalarizations and objectives.Initial kernel weights are set uniformly before adaptive updates.
  • KENDO-MO Procedure: Each iteration samples Dirichlet scalarization directions and kernel-conditioned function paths, then optimizes scalarized functions to obtain candidate optimizers.The sampled optimizers are used in subsequent GP conditioning.
  • KENDO-MO Procedure: After selecting the next point and observing all objective values, the algorithm updates kernel weights and GP posteriors for every objective and kernel.The resulting dataset is used to construct the final Pareto front.

B Derivation of the KENDO-BO Acquisition Function

The derivation replaces SCoreBO’s continuous hyperparameter expectation with a discrete expectation over kernel families and retains optimizer-conditioned disagreement. Moment matching converts the predictive mixture into a Gaussian summary for a closed-form Hellinger distance.

  • From Continuous to Discrete Uncertainty: SCoreBO approximates an intractable hyperparameter expectation by MCMC samples and optimizer samples with a uniformly weighted estimator.Its joint posterior factors into optimizer uncertainty conditional on hyperparameters and the hyperparameter posterior.
  • From Continuous to Discrete Uncertainty: KENDO changes the uncertainty source from continuous hyperparameters of one kernel to a posterior over discrete kernel families.Kernel probabilities are defined through Bayesian model averaging.
  • From Continuous to Discrete Uncertainty: Factoring the joint posterior over kernel identity and optimizer yields a weighted discrete sum that recovers the KENDO-BO acquisition function.The weights multiply expectations over optimizer-conditioned disagreement for each kernel.
  • Optimizer Conditioning: KENDO obtains optimizer samples by optimizing random Fourier feature function paths and conditions each kernel GP on the resulting phantom observations.This preserves optimizer-conditioned querying while using the kernel ensemble as the uncertainty source.
  • Moment Matching: Moment matching replaces the Gaussian-mixture marginal predictive with a Gaussian having exactly matched first and second moments.The matched variance contains within-component average variance and between-component mean spread.
  • Hellinger Disagreement: The moment-matched Gaussian enables closed-form Hellinger distances between marginal and optimizer-conditioned predictive distributions.Removing optimizer conditioning gives the corresponding kernel-component disagreement used in the derivation.

C.3 Kernel Weight Convergence and Winner-Take-All Behavior

Across benchmarks, KENDO’s Bayesian kernel weighting can either rapidly select a dominant kernel or preserve diversity when evidence is ambiguous. The framework also retains competitive performance while substantially reducing per-iteration wall-clock time.

  • Kernel Weight Convergence: Kernel weights collapse to one dominant kernel within approximately 10–20 iterations on smooth, stationary benchmarks such as Branin.This winner-take-all behavior follows from multiplicative Bayesian updates that amplify repeated predictive-likelihood advantages.
  • Kernel Weight Convergence: On Branin, Matérn-2.5 wins in 5 out of 10 seeds with an average maximum weight of 0.950.The result reflects Matérn-2.5’s balance between RBF smoothness and Matérn-1.5 differentiability.
  • Winner-Take-All Behavior: Kernel preference is benchmark-dependent: RBF wins all 10 seeds on ZDT2 and Vehicle Safety, while Matérn-1.5 wins 8 of 10 seeds on Penicillin.For active learning, Matérn-2.5 is most frequently preferred, winning on 3 of 4 benchmarks.
  • Computational Cost: 3.1×–5.4× speedup is achieved by KENDO over SCoreBO, while KENDO-AL achieves 7.6×–26.9× speedup over SAL-Matérn-2.5 per iteration.The ensemble mechanism eliminates MCMC overhead while maintaining competitive optimization and learning performance.
  • Computational Cost: For BO, acquisition optimization consumes 50–80% of iteration time, whereas active-learning costs shift toward model fitting on higher-dimensional benchmarks.Weight updates remain negligible, contributing less than 7 ms per iteration.

C.5 Sensitivity to Dirichlet Concentration α

KENDO-MO uses Dirichlet concentration α to control scalarization-weight concentration and phantom-optimizer diversity. The default α = 1.0 is robust, while benchmark geometry can motivate alternative values or scalarization choices.

  • Dirichlet Concentration: α controls Dirichlet mass concentration: α →0 favors simplex vertices, α = 1 gives the uniform prior, and α →∞ concentrates on the centroid.Its per-coordinate variance decreases monotonically with α.
  • Dirichlet Concentration: α directly governs phantom-optimizer diversity through the inner arg max, with sweeps conducted at α ∈{0.2, 1.0, 5.0} on five MOO benchmarks.The phantom optimizers drive KENDO-MO’s disagreement signal.
  • Diversity and Entropy: α = 0.2 produces the highest phantom-X diversity, whereas α = 5.0 produces the lowest and approaches the entropy ceiling.For K = 2 and K = 3, mean entropy is approximately 0.65 and 1.03 at α = 5.0, respectively.
  • Hypervolume Sensitivity: α = 1.0 is never significantly outperformed across the five benchmarks, although the optimal α is benchmark-dependent and tracks Pareto-front geometry.Small α performs well on concave-front or heterogeneous benchmarks, while α = 5.0 yields a small but consistent gain on LCBench and Penicillin.
  • Scalarization Robustness: Three of four MOO benchmarks show no statistically significant difference between linear and Chebyshev scalarization, while DTLZ2 favors Chebyshev with p = 0.014.The DTLZ2 result is consistent with its concave Pareto front.
  • Takeaway: KENDO-MO’s gains arise from the kernel ensemble and disagreement-aware acquisition rather than a finely tuned Dirichlet concentration or scalarization choice.Practitioners may select scalarization based on prior knowledge of Pareto-front geometry.
  • Figure Guide: Figure 11 compares hypervolume, sampled-λ entropy, and phantom-optimizer pairwise distance across α values and five MOO benchmarks.The columns jointly show performance sensitivity, scalarization-weight dispersion, and induced optimizer diversity.
Loading 2608.24721v1…