PaperlayerView exact revision on arXiv

Source-linked AI summary

MF-SCBO : Multi-fidelity Scalable Constrained Bayesian Optimization

Lucas Palazzolo, Mickaël Binois, Laëtitia Giraldi

arXiv:2609.29941v1cs.LGmath.OC

TL;DR

Expensive evaluations motivate multi-fidelity optimization, but existing methods do not jointly address high dimensionality, black-box constraints, arbitrary fidelity levels, and non-nested sampling. MF-SCBO extends SCBO to this setting. Across the evaluated problems, it generally converges better than SCBO and MF-MES, while MF-MES becomes computationally impractical as dimension increases.

  • Problem

    Existing methods do not simultaneously address high-dimensional Bayesian optimization with black-box constraints, arbitrary fidelity levels, and non-nested sampling, despite rising evaluation costs.

    Source: §1 · pp. 1–2

  • Method

    MF-SCBO extends SCBO to multi-fidelity optimization, using evaluated or predicted trust-region centers and a leave-one-out criterion to assess predicted centers.

    Source: §5 · p. 13

  • Results

    MF-SCBO generally achieved better convergence than SCBO and MF-MES, while MF-MES became computationally impractical as problem dimension increased.

    Source: §5 · p. 13

  • Takeaways & Limitations

    When highest-fidelity predictions are sufficiently accurate, the predicted variant generally converges faster than the evaluated variant, but this pattern did not hold for Rosenbrock and Solar.

    Source: §5 · p. 13

    The framework assumes constraints are independent of fidelity, with one constraint function shared across fidelity levels.

    Source: §3 · p. 6

Abstract

from arXiv · show

Many real-world optimization problems rely on expensive simulations or experiments, making the efficient use of available data essential. Multi-fidelity optimization of high-dimensional black-box functions subject to black-box constraints is increasingly relevant as the cost of objective evaluations continues to rise in applications such as machine learning, engineering, and control. To our knowledge, no existing method simultaneously addresses high-dimensionality, black-box constraints, an arbitrary number of fidelity levels, and non-nested sampling. In this work, we extend the Scalable Constrained Bayesian Optimization method to the multi-fidelity setting, resulting in the MF-SCBO method. The proposed approach is evaluated on standard benchmark functions as well as challenging problems. The experimental results demonstrate that MF-SCBO generally achieves better convergence than both the single-fidelity SCBO and the other multi-fidelity method considered in this high-dimensional and constrained settings.

1 Introduction

The paper addresses the challenge of combining multi-fidelity optimization with high dimensionality, black-box constraints, and non-nested sampling. It extends SCBO to this setting and evaluates the approach against SCBO and MF-MES on benchmark problems.

1 Introduction

  • Lower-fidelity approximations can reduce optimization cost by guiding search when objective evaluations are expensive, despite being less accurate.The goal is to reduce the overall budget compared with relying only on high-fidelity evaluations.Source: §1 · p. 1
  • Multi-fidelity surrogates model relationships between levels through linear or nonlinear GP-based approaches, including autoregressive models and deep Gaussian processes.Other approaches treat fidelity as an input variable and extend acquisition functions to choose both an input and fidelity.Source: §1 · pp. 1–2
  • MF-SCBO extends SCBO to address high-dimensional, constrained, multi-fidelity optimization with arbitrary fidelity levels and non-nested sampling.It introduces evaluated and predicted trust-region centers; the predicted variant uses leave-one-out assessment, and constraints are assumed fidelity-independent.Source: §1 · p. 2
  • The evaluation compares MF-SCBO with SCBO and, when computationally feasible, MF-MES on synthetic and realistic problems with dimensions from 6 to 100.Source: §1 · p. 2

2 Preliminaries

The preliminaries describe Gaussian-process Bayesian optimization, SCBO's trust-region strategy for constrained high-dimensional search, and the multi-fidelity modeling assumptions and surrogate construction used in the paper.

2.1 Gaussian Processes and Bayesian Optimization

  • Gaussian-process Bayesian optimization uses a surrogate posterior to guide iterative evaluation, and this work assumes noise-free observations.The posterior is a GP with mean and covariance functions; an acquisition function balances exploration and exploitation when selecting points.Source: §2.1 · pp. 2–3

2.2 High dimension and constraints

  • SCBO centers a local trust region on the best feasible evaluated point, or on the point with least overall constraint violation when none is feasible.Thompson sampling generates candidates from objective and constraint posteriors; the trust region expands after improvement and contracts otherwise.Source: §2.2 · p. 4

2.3 Modeling multi-fidelity

  • The multi-fidelity setup treats the highest-fidelity function as the target and assumes approximation error decreases while evaluation cost increases with fidelity.The stated objective is to use low-fidelity models to reduce high-fidelity evaluations while preserving optimization accuracy.Source: §2.3 · p. 4
  • The method adopts an autoregressive recursive GP formulation, modeling each successive fidelity with a scaled lower-fidelity surrogate plus an independent GP error process.The scaling parameter controls correlation between fidelity levels and is constrained to [0, 2] for numerical stability.Source: §2.3 · pp. 4–5
  • In non-nested sampling, higher-fidelity observations need not be included among lower-fidelity data, so missing observations are replaced by predictive means in the described extension.The nested case instead assumes higher-fidelity data are available only at a subset of lower-fidelity points.Source: §2.3 · p. 5

3 Multi-fidelity Bayesian Optimization for High-Dimensional Constrained Problems

MF-SCBO extends SCBO to constrained, high-dimensional optimization with multiple fidelities. It combines Thompson sampling with two strategies for updating the trust region.

3 Multi-fidelity Bayesian Optimization for High-Dimensional Constrained Problems

  • MF-SCBO extends SCBO to constrained, many-variable optimization while exploiting multiple fidelity levels.The method is designed for optimization problems involving many variables and multiple fidelities.Source: §3 · p. 5

3.1 Trust Region Management

  • Trust-region centers can be updated from high-fidelity evaluations alone or from high-fidelity predictions using data across fidelities.For a predicted center, a leave-one-out check assesses prediction reliability; an unreliable candidate is evaluated at the highest fidelity before it can become the center.Source: §3 · p. 6, §3.1 · pp. 7–8
  • If feasible points exist, the center minimizes the objective; otherwise, it minimizes the maximum constraint violation.This selection rule applies to both evaluation-based and prediction-based center strategies.Source: §3.1 · p. 7

3.2 Thompson Sampling

  • Thompson sampling selects a batch of q points from r designs in the trust region, while fidelity choice balances prediction error against evaluation cost.The fidelity criterion prioritizes lower-fidelity evaluations when their error remains above a threshold determined by the error-to-cost ratio for higher-fidelity evaluations.Source: §3.2 · p. 8, Figure 2 · p. 10

4 Experiments

Experiments span synthetic and realistic constrained problems with dimensions from 6 to 100. Across synthetic benchmarks, multi-fidelity methods outperform SCBO overall, while realistic results vary by problem and variant.

4.1 Problem Setup

  • The test set includes synthetic and realistic problems with dimensions from 6 to 100, including constrained Hartmann, Borehole, Ackley, and Rosenbrock functions.The realistic applications include a solar thermal power plant simulator and airfoil shape optimization.Source: §4.1 · p. 10

4.2 Protocol

  • Results use repeated-run means and percentiles, with 50 runs generally and five for the costly solar problems; MF-MES is limited to dimensions at most 8.MF-MES experiments use five independent runs because high-dimensional acquisition optimization is computationally difficult.Source: §4.2 · p. 11

4.3 Results

  • Multi-fidelity methods outperform SCBO overall on synthetic benchmarks, though Rosenbrock40D and Rosenbrock100D look weaker by total cost than by high-fidelity evaluations.The authors attribute this cost pattern to low-fidelity evaluations outweighing the savings from fewer high-fidelity evaluations; predicted MF-SCBO also consistently outperforms its evaluated variant on the other benchmarks.Source: §4.3 · p. 11, Figure 3 · p. 12
  • On realistic tasks, MF-SCBO performs well on Solar2 and converges faster than SCBO on Airfoil by high-fidelity evaluations, but predicted MF-SCBO performs poorly on Solar7.MF-MES fails to converge on Solar7; across Airfoil, early search relies mostly on low fidelity and later iterations increasingly use high fidelity.Source: §4.3 · pp. 11, 13, Figure 4 · p. 13

5 Conclusion

MF-SCBO extends SCBO to high-dimensional multi-fidelity optimization with black-box constraints and arbitrary fidelity levels. Across synthetic benchmarks and engineering applications, it generally converged better than SCBO and MF-MES, although the predicted variant’s relative speed depended on prediction accuracy.

5 Conclusion

  • MF-SCBO extends SCBO to high-dimensional problems with black-box constraints, arbitrary fidelity levels, and non-nested sampling.One variant centers trust regions using high-fidelity evaluations; another uses highest-fidelity predictions from all levels, with a leave-one-out criterion to maintain center consistency when predictions are inaccurate.Source: §5 · p. 13
  • MF-SCBO generally achieved better convergence than single-fidelity SCBO and MF-MES across synthetic benchmarks and realistic engineering applications.MF-MES became computationally impractical as problem dimension increased, while MF-SCBO remained applicable to high-dimensional problems.Source: §5 · p. 13
  • When high-fidelity predictions were sufficiently accurate, the predicted variant generally converged faster on Hartmann, Borehole, and Ackley, but not on Rosenbrock or Solar.The leave-one-out criterion adds negligible computational cost.Source: §5 · p. 13
  • Future work includes multi-source settings with non-monotone costs and errors, fidelity-dependent constraints, non-myopic budget use, and alternative Gaussian processes for higher-dimensional scalability.Sources may differ in costs, accuracies, and biases, and the most informative source may depend on input location.Source: §5 · p. 14

A Cases

The cases cover synthetic benchmarks and engineering applications with varied dimensions, fidelity levels, and constraint settings. They include Ackley and Rosenbrock benchmarks, Solar simulators, airfoil optimization, and other standard functions.

Hartmann6D

  • The Hartmann family is used as an optimization benchmark, and the six-dimensional case has six local minima.The supplied passage also gives the global-minimum location and value, but the coordinate is not readable in the excerpt.Source: A Cases · p. 14

Borehole8D

  • The Borehole function models water flow and uses two fidelity levels, with no constraints in the standard benchmark setting.Its computational domain is specified by eight parameter ranges.Source: A Cases · p. 15

Ackley20D

  • Ackley20D uses domain [−5, 10]^d with d = 20 and two fidelity levels; its low-fidelity minimum is displaced from the origin, and two constraints are imposed.The constraints are a hyperplane and a sphere centered at the origin with radius 5.Source: A Cases · p. 15

Rosenbrock40D and Rosenbrock100D

  • Rosenbrock40D and Rosenbrock100D use three fidelity levels; their flat parabolic valley and local minima make convergence difficult, particularly at high dimension.The two-dimensional illustration shows the three fidelity levels from lowest to highest.Source: A Cases · pp. 15–16, Figure 6 · p. 16

Solar2 and Solar7

  • Solar2 minimizes heliostat-field surface with 14 variables and 12 constraints, while Solar7 maximizes receiver efficiency over 24 hours with 7 variables and 6 constraints.The discrete variables are fixed to their initial Solar-provided values; hidden simulation failures return 10^20, and violations occur less often at higher fidelity.Source: A Cases · pp. 16–17

Airfoil shapes

  • Airfoil optimization maximizes lift-to-drag ratio using PARSEC parameterization; high fidelity evaluates multiple operating conditions, while low fidelity uses one condition.Four conditions support robust shapes across speed and angle-of-attack variation at fixed Reynolds number; the moment-coefficient constraint applies at both fidelities, and failed solver cases receive a default value of 10^2.Source: A Cases · p. 17

B Impact of parameters: number of fidelity levels, batch size and cost

The section examines how fidelity-level count, batch size, and evaluation cost affect convergence across benchmark functions. More fidelity levels can slow convergence, while batch-size and cost effects differ between evaluated and predicted modes.

Number of fidelity levels

Batch size

  • Larger batches increase the evaluated–predicted discrepancy because the evaluated confidence region updates only after a full highest-fidelity batch.Batch sizes 4, 8, and 16 are compared on Hartmann6D, Borehole8D, Ackley20D, and Rosenbrock100D; batch size has less effect in predicted mode.Source: B Impact of parameters: number of… · p. 19, Figure 9 · p. 20

Cost

  • Higher evaluation costs require more low-accuracy evaluations before a higher-accuracy evaluation becomes worthwhile.The comparison uses costs (1, 20) and (1, 50) for two fidelities, and (1, 10, 20) and (1, 20, 50) for three fidelities; the stated cause is criterion (16), which weights error against cost.Source: B Impact of parameters: number of… · p. 19, Figure 10 · p. 20
Loading 2609.29941v1…