PaperlayerView exact revision on arXiv

Source-linked AI summary

Adapting Nonstationary Multi-output Gaussian Processes to Bayesian Optimization

Zikai Xie

arXiv:2609.32464v1cs.LGcs.AI

TL;DR

MOBO relies on surrogate predictions at acquisition-selected points, which can differ from the training distribution, and standard stationary independent GPs do not explicitly share objective information or represent input-dependent covariance. MOLRN-BO adapts MO-LRN for sequential optimization through shared-spectral fitting, calibration, and Pareto-local search with periodic global rounds. Across 12 deterministic bi-objective benchmarks, it improves on original MO-LRN and attains the best average ranks for final normalized hypervolume and normalized IGD among nine evaluated algorithms.

  • Problem

    MOBO needs reliable predictions at acquisition-selected points, which may differ substantially from the training distribution, while standard independent stationary GPs do not explicitly share information across objectives or represent input-dependent covariance.

    Source: §1 · p. 1

  • Method

    MOLRN-BO combines a regularized shared-spectral surrogate with objective-specific residuals, prequential mean correction, tempered covariance scaling, and Pareto-local qLogEHVI search with periodic global rounds.

    Source: §1 · p. 2

  • Results

    Across 12 deterministic bi-objective benchmarks, MOLRN-BO achieves the best average ranks among nine algorithms for final nHV and nIGD and significantly outperforms original MO-LRN on all four primary metrics in Holm-adjusted tests.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    The results support adapting the structure and use of nonstationary multi-output surrogates to sequential optimization, with MOLRN-BO showing strong final Pareto quality and competitive anytime performance.

    Source: §5 · p. 9

    The evaluation is limited to small-budget deterministic bi-objective optimization with one candidate per round, and sensitivity to fixed regularization settings and the optimization-quality/computational-overhead trade-off remain unquantified.

    Source: §5 · pp. 9–10

Abstract

from arXiv · show

Multi-objective Bayesian optimization (MOBO) commonly relies on independent Gaussian processes (GPs) with stationary kernels, limiting its ability to represent nonstationary structure and share information between objectives. However, expressive nonstationary GPs do not necessarily make reliable BO decisions. We study this mismatch for the multi-output low-rank nonstationary (MO-LRN) GP: strong training fit can coexist with large off-design errors and optimistic acquisition predictions. We introduce MOLRN-BO, which combines a regularized shared-spectral surrogate with objective-specific residuals, prequential mean correction and tempered covariance scaling, and Pareto-local qLogEHVI optimization with periodic global search. Experiments on 12 deterministic bi-objective benchmarks show that MOLRN-BO substantially improves upon the original MO-LRN and achieves the best average problem ranks for final normalized hypervolume and normalized inverted generational distance among nine evaluated algorithms. It also achieves the strongest adverse-tail performance while remaining competitive with the leading baselines in anytime optimization. Ablation studies further show that the shared spectral construction improves off-design prediction, the local--global decision policy improves optimization performance, and hierarchical calibration reduces systematic candidate bias. These results demonstrate that nonstationary multi-output surrogates can deliver strong and robust MOBO performance when their structure and use are explicitly adapted to the demands of sequential optimization.

1 INTRODUCTION

MOBO selects experiments using probabilistic surrogates, but standard stationary independent GPs have limited capacity to model nonstationarity and share information across objectives. This paper asks how to make MO-LRN reliable for acquisition-driven optimization and introduces MOLRN-BO to address the mismatch between fit and decisions.

1 INTRODUCTION

  • MOBO's success depends on predictions at acquisition-preferred points, which may differ substantially from the observed training distribution.The paper frames this as a challenge for surrogate design as well as acquisition design.Source: §1 · p. 1
  • Increasing MO-LRN spectral capacity improved training fit in preliminary experiments but worsened off-design prediction, while acquisition-selected predictions were biased and underdispersed.These diagnostics motivate the paper's focus on reliable sequential decisions rather than training fit alone.Source: §1 · pp. 1–2
  • MOLRN-BO combines regularized shared LRN components, objective-specific residuals, prequential mean correction, bounded covariance rescaling, and Pareto-centered local search with periodic global rounds.The framework aligns surrogate fitting, predictive calibration, and acquisition optimization with sequential multi-objective decisions.Source: §1 · p. 2
  • Across 12 bi-objective benchmarks, MOLRN-BO achieves the best average ranks for final normalized hypervolume and normalized IGD among evaluated methods while maintaining competitive anytime performance.Controlled ablations examine spectral capacity and sharing, joint local–global search, mode-specific bias estimation, and hierarchical bias pooling.Source: §1 · p. 2

2 RELATED WORK

Prior MOBO work has emphasized acquisition design, while surrogate design has comparatively less attention. Nonstationary spectral multi-output models offer a relevant alternative, but their use in acquisition-driven MOBO remains underexplored.

Multi-objective surrogates and acquisitions

  • MOBO research has chiefly developed scalarization-, hypervolume-, and information-theoretic acquisitions, while standard surrogate choices are typically independent stationary GPs.A smaller surrogate-design literature considers correlated multi-task or multi-output GPs and deep GPs.Source: §2 · p. 2

Nonstationary spectral models

  • MO-LRN uses a low-rank factorization of flexible matrix-valued nonstationary spectral densities, reducing parameter growth in the number of outputs from quadratic to linear.Earlier MO-LRN evaluations focused on passive prediction rather than acquisition-driven MOBO.Source: §2 · p. 2

Calibration and localized search

  • Calibration for correlated nonstationary surrogates under hypervolume-based acquisition remains comparatively unexplored, as do nonstationary spectral multi-output surrogates in acquisition-driven MOBO.Localized BO offers a complementary approach; TuRBO uses adaptive trust regions and MORBO coordinates multiple local regions across a Pareto front.Source: §2 · pp. 2–3

3 METHOD

Diagnostics on WFG4 show that better training fit can coincide with distorted, underdispersed predictions and worse off-design and HVI estimates. MOLRN-BO addresses these issues through shared-spectral fitting, calibration, and a local–global acquisition policy.

Experimental setting

  • The experiments use deterministic objectives on a bounded box, one candidate per round, and objective-wise standardization recomputed from the observed data.The raw posterior moments are reported on the original objective scale.Source: §3 · p. 3

From surrogate fit to decision reliability

  • Increasing MO-LRN mixtures from Q = 2 to Q = 4 reduced training RMSE but worsened off-design prediction and HVI estimation, while acquisition-selected prequential predictions were severely underdispersed.The diagnostic figure also shows a posterior slice with distorted means and narrow uncertainty bands at higher spectral capacity.Source: Figure 1 · p. 3, §3 · p. 4
  • Pilot diagnostics also found spectral frequencies near imposed bounds, weak overlap between output-specific spectra, and unrestricted acquisition choices far from or near the boundary of the domain.These observations motivate controlling spectral capacity, calibrating at decision-relevant locations, and limiting acquisition overexploitation of poorly supported regions.Source: §3 · p. 4

Shared-spectral covariance

  • The shared-spectral covariance uses MO-LRN nonstationary components with spectral parameters shared across objectives, component-specific cross-objective dependence, and objective-specific ARD-RBF residuals.The stated implementation uses Q = 2 and bi-objective coregionalization matrices with unit diagonals.Source: §3 · p. 4

Regularized fitting

  • Regularized fitting constrains spectral complexity during sequential refitting of the exact GP on standardized objectives.The parameter domain bounds frequency locations and spectral scales; penalties also regularize coregionalization, residual, noise, and outputscale parameters.Source: §3 · p. 4

Prequential mean correction

  • Prequential mean correction uses standardized residuals recorded at previously selected candidates, with recent-window weighting, clipping, and partial pooling across local and global search modes.Each residual used for the next decision comes from a prediction stored before its corresponding evaluation.Source: §3 · p. 4

Covariance rescaling and decision posterior

  • Covariance rescaling distinguishes predictive from acquisition scales, tempering and capping the inflation exposed to qLogEHVI while leaving the fitted GP and its correlation structure unchanged.Predictive diagnostics instead use the raw mean and covariance scaled by cpred,n; cross-validation uses full-data hyperparameters, while stored candidate errors are strictly prequential.Source: §3 · pp. 4–5

Pareto-local and periodic-global selection

  • Most acquisition searches concentrate around diverse observed Pareto points, while periodic global rounds preserve full-domain exploration.The GP is always fitted to all observations; only acquisition optimization is localized, and global rounds are more frequent early in optimization.Source: §3 · p. 5

4 EXPERIMENTS

Across 12 deterministic bi-objective benchmarks, MOLRN-BO leads the evaluated methods on final Pareto-quality ranks, while its advantage over the original MO-LRN spans final and anytime measures. Ablations show gains in off-design prediction and optimization performance, while hierarchical bias pooling reduces prediction bias without consistent optimization gains.

4.1 EXPERIMENTAL SETTINGS

  • The evaluation covers 12 deterministic bi-objective problems, with 20 trials per algorithm–task pair, a shared 10-point initial design, and 100 sequential evaluations.Primary outcomes include final nHV and nIGD and two lower-is-better anytime AUC measures; algorithms are ranked by across-seed means within problems, then averaged across problems.Source: §4.1 · p. 6

4.2 EXPERIMENTAL RESULTS

  • 2.25 for nHV and 2.17 for nIGD are MOLRN-BO’s best average problem ranks among the nine algorithms on final Pareto quality.qLogNParEGO ranks second at the final budget; it is slightly better on both anytime AUC measures.Source: §4.2 · p. 6, Table 1 · p. 8
  • Figure 2 shows nHV and nIGD trajectories; qLogNParEGO often progresses faster initially, while MOLRN-BO is competitive or leading at the final budget on many problems.MOLRN-BO’s advantage is generally more pronounced late in optimization; the figure’s solid curves show across-trial means and shaded regions one sample standard deviation.Source: §4.2 · p. 6, Figure 2 · p. 7
  • MOLRN-BO beats original MO-LRN on 11/12 problems for final nHV and HV-regret AUC, and 12/12 for final nIGD and nIGD AUC.Cross-problem tests remain significant after Holm correction (p ≤0.00244).Source: §4.2 · p. 8
  • Across the 12 problems, qLogEHVI comparisons favor MOLRN-BO on mean final nIGD in 10/12 and final nHV in 7/12; qLogNParEGO comparisons are closer.Aggregate differences from qLogNParEGO are not statistically significant, and qLogNParEGO has slightly better anytime ranks.Source: §4.2 · p. 8
  • MOLRN-BO ranks first on all four adverse-tail measures and has the smallest mean across-seed standard deviation of final nIGD, 0.0358.Its gains are not universal: it is comparatively weak on WFG5, slower than leading baselines on WFG1, and improves nIGD slowly on DTLZ7.Source: §4.2 · p. 8

4.3 ABLATION STUDY

  • At Q = 2, the shared spectral construction lowers mean top-HVI optimism from 0.498 to 0.087 despite nearly unchanged training RMSE.Reducing output-specific capacity from Q = 4 to Q = 2 also lowers global and Pareto-region RMSE on 10/12 problems, despite increasing training RMSE; Table 2 reports controlled diagnostics with RBF residuals disabled in panel (a).Source: §4.3 · p. 9, Table 2 · p. 9
  • The local–global search and independent-bias bundle improves final nHV, final nIGD, and HV-regret AUC on 11/12 problems, and nIGD AUC on 10/12.WFG1 is the principal exception for final and hypervolume-based measures, and DTLZ4 for nIGD AUC.Source: §4.3 · p. 9

5 CONCLUSION

MOLRN-BO adapts both the surrogate and acquisition decisions to address the mismatch between training fit and sequential optimization. Its evaluation is limited to small-budget, deterministic bi-objective optimization with one candidate per round, and some computational and regularization trade-offs remain unquantified.

5 CONCLUSION

  • Across 12 deterministic bi-objective benchmarks, MOLRN-BO achieves the best average ranks for final nHV and nIGD, with second-best anytime ranks.It significantly outperforms original MO-LRN on all four primary metrics in Holm-adjusted cross-problem tests.Source: §5 · p. 9
  • The evaluation covers small-budget, deterministic bi-objective optimization with one candidate per round.Larger budgets, additional outputs, noisy observations, and batch settings are identified as settings requiring further methodological treatment.Source: §5 · p. 9
  • Sensitivity to fixed regularization settings and the trade-off between optimization quality and computational overhead remain unquantified.Source: §5 · p. 10

ETHICS STATEMENT

REPRODUCIBILITY STATEMENT

  • The supplementary repository provides MOLRN-BO, evaluated baselines, ablation code, and implementations of all 12 benchmark problems.Appendix sections detail the surrogate, calibration, and acquisition methods; Section 4.1 specifies experimental settings and tests.Source: REPRODUCIBILITY STATEMENT · p. 10

A PRELIMINARY DIAGNOSES

Pilot sequential experiments identify five BO-oriented problems with applying MO-LRN directly: three in the surrogate, one in the decision posterior, and one in acquisition search. These diagnostics motivate corresponding changes to the surrogate, decision posterior, and search strategy.

A PRELIMINARY DIAGNOSES

  • Direct MO-LRN use showed three surrogate issues: poor alignment of in-sample fit with candidate-region accuracy, frequencies near imposed bounds, and numerically negligible spectral overlap.The diagnostics were summarized on the ten-dimensional WFG4 development problem.Source: A PRELIMINARY DIAGNOSES · p. 12
  • The decision posterior was biased and underdispersed at acquisition-selected points, while unrestricted full-domain search favored distant or boundary-adjacent candidates.Source: A PRELIMINARY DIAGNOSES · p. 12
  • The diagnostics motivate a structured regularized surrogate, a decision-only error-calibrated posterior, and search combining Pareto-set local refinement with global exploration.Source: A PRELIMINARY DIAGNOSES · p. 12

B CALCULATION DETAIL OF MOLRN-BO

The setup considers maximizing an expensive vector-valued black-box function over a bounded input domain. After n evaluations, the observed input-output pairs form the available data.

B CALCULATION DETAIL OF MOLRN-BO

B.1 REGULARIZED SHARED-SPECTRAL SURROGATE

The shared-spectral surrogate separates shared nonstationary input structure and cross-objective dependence from objective-specific smooth residual variation. Its spectral and model parameters are regularized and learned through marginal-likelihood optimization.

B.1 REGULARIZED SHARED-SPECTRAL SURROGATE

  • The surrogate shares MO-LRN spectral parameters across objectives, uses component-specific cross-objective dependence, and adds independent objective-specific ARD-RBF residuals.The shared construction avoids relying on overlap between independently fitted output spectra.Source: §B.1 · pp. 12, 15
  • The objective-specific ARD-RBF residual has a learned relative scale, preventing it from becoming numerically negligible as the shared kernel’s marginal scale changes with input dimension.Source: §B.1 · p. 15
  • The composite covariance is valid because its shared components and block-diagonal residual contributions are positive semidefinite.Source: §B.1 · p. 15
  • Bounded spectral parameters and penalties on roughness, boundary proximity, mixture collapse, and model scales regularize the shared-spectral fit.The composite exact GP is fitted by minimizing regularized marginal likelihood.Source: §B.1 · pp. 15–16
  • The experiments fix the spectral mixture count at Q = 2 and specify bounds, penalty strengths, initial values, and prior widths for model parameters.Source: §B.1 · pp. 12, 16

B.2 MEAN CORRECTION AND COVARIANCE-SCALE ADJUSTMENT

The section corrects systematic mean error and uncertainty miscalibration at acquisition-selected points, where a good fit alone does not ensure reliable decisions. Its corrections use past candidate errors and covariance-scale diagnostics, with the latter treated as internal diagnostics rather than independent coverage assessment.

B.2 MEAN CORRECTION AND COVARIANCE-SCALE ADJUSTMENT

  • Acquisition-selected points show systematic mean error and posterior underdispersion, motivating mean correction and covariance rescaling for decisions.The fitted covariance regularization alone does not ensure accurate uncertainty at acquisition-preferred points.Source: §B.2 · p. 16
  • Mean corrections use only errors recorded after prior evaluations, with recent standardized residuals summarized robustly and older residuals downweighted.The method uses component-wise medians and MADs, clipping, and exponentially decaying weights.Source: §B.2 · pp. 16–17
  • Local and global mode-specific bias estimates are partially pooled toward a shared estimate, with limited mode-specific history keeping the correction near the shared trend.As mode-specific evidence accumulates, its correction receives greater weight.Source: §B.2 · p. 17
  • Spatial-fold covariance diagnostics keep full-data hyperparameters fixed, so their residuals are internal scale diagnostics rather than independent predictive-coverage assessments.The section combines leave-one-out and spatial estimates geometrically.Source: §B.2 · p. 17
  • Acquisition-selected predictive errors also inform the covariance scale, which is tempered and capped before qLogEHVI receives the decision posterior.The raw predictive view is retained for diagnostics, while the transformed view is supplied to qLogEHVI.Source: §B.2 · p. 18

B.3 PARETO-LOCAL AND PERIODIC-GLOBAL QLOGEHVI

The acquisition policy concentrates most qLogEHVI optimization around diverse observed Pareto points while retaining periodic full-domain search. The GP remains global, and the local search radius adapts to realized hypervolume improvement.

B.3 PARETO-LOCAL AND PERIODIC-GLOBAL QLOGEHVI

  • Most qLogEHVI restarts are organized around diverse observed Pareto points, while periodic global rounds preserve full-domain exploration.Localization applies only to acquisition optimization; the GP is fitted to all observations.Source: §B.3 · pp. 18–19
  • The number of local centers is capped by the restart budget and maximum region count, and restarts are distributed approximately evenly across their bounded regions.Local initializations are sampled around each center, clipped to its region, and optimized within that region.Source: §B.3 · p. 19
  • Global rounds occur every fifth evaluation during the first 40 post-initial iterations and every tenth thereafter.These schedules correspond to 20% of the first 40 post-initial iterations and 10% thereafter.Source: §B.3 · p. 19
  • The common local radius expands after three consecutive successes and contracts after five consecutive failures, while global rounds leave it unchanged.Success is defined by realized hypervolume improvement exceeding the stated threshold.Source: §B.3 · p. 20

C ADDITIONAL PERFORMANCE AND ROBUSTNESS ANALYSES

Across the additional robustness and Pareto-quality analyses, MOLRN-BO ranks strongly on adverse-tail measures and normalized IGD+, while its convergence advantage is more pronounced late in optimization. Performance varies by problem, and it is not uniformly faster across the benchmark suite.

C ADDITIONAL PERFORMANCE AND ROBUSTNESS ANALYSES

  • MOLRN-BO has the best mean problem rank for nIGD+ (2.25), ranks in the top three on 11/12 problems, and beats original MO-LRN on all 12.For Iε+, qLogNParEGO ranks first (2.83) and MOLRN-BO second (3.17); MOLRN-BO beats qLogNParEGO on 5/12 problems and original MO-LRN on 10/12.Source: C ADDITIONAL PERFORMANCE AND… · p. 20
  • MOLRN-BO's mean nIGD problem rank improves from 3.33 after 25 post-initial evaluations to 2.17 after 100, while its final nHV rank reaches 2.25.It achieves the best final nHV and nIGD ranks, but qLogNParEGO remains slightly better on both anytime AUC measures.Source: C ADDITIONAL PERFORMANCE AND… · pp. 20–21
  • MOLRN-BO ranks first on all four adverse-tail measures and has the smallest mean across-seed standard deviation of final nIGD among nine algorithms (0.0358).The authors attribute its main robustness benefit to fewer poorly performing trials, not uniformly reduced variance on every problem and metric.Source: C ADDITIONAL PERFORMANCE AND… · p. 20, Table 4 · p. 21
  • Performance is problem-dependent: MOLRN-BO is slow on WFG1, weak on WFG5 in final nHV and anytime performance, and plateaus in nIGD on DTLZ7.The trajectories support strong final quality and reduced poor-trial risk, not uniformly faster convergence.Source: C ADDITIONAL PERFORMANCE AND… · p. 21
Loading 2609.32464v1…