A Unified Information-Theoretic Approach to Constrained Multi-Fidelity Multi-Objective Bayesian Optimization

1 Introduction

• Existing methods for the full setting either use separate mechanisms for design, fidelity, or constraints, or require more computationally involved inference.The latter approach uses multi-fidelity deep Gaussian processes and approximate joint conditioning on the Pareto set and frontier.Source: §1 · p. 1
• Queries at any input and fidelity are valued by their cost-normalized information about the target-fidelity feasible Pareto frontier.Objectives, constraints, and fidelity selection enter a single information criterion rather than separate acquisition mechanisms.Source: §1 · p. 2
• The mutual information is intractable, so the method derives a variational lower bound using over- and under-truncated Pareto-consistency regions.The resulting computation uses Gaussian region probabilities and standard cross-fidelity Gaussian conditioning.Source: §1 · p. 2
• Experiments report CMF-PFEV is effective across GP-derived synthetic problems, benchmarks, and real-world hyperparameter optimization settings.The evaluation spans different numbers of objectives, constraints, and input dimensions.Source: §1 · p. 2

2 Problem Setting

• The goal is to approximate the highest-fidelity feasible Pareto frontier under a finite evaluation budget.The problem setting treats observations at any fidelity as valuable according to their information about that frontier.Source: §2 · p. 3

2.1 Multi-fidelity GP and observation model

• Each objective and constraint function is modeled by an independent multi-fidelity Gaussian process, with fidelity represented as an additional GP input.Outputs at different fidelities are statistically coupled through the GP covariance.Source: §2.1 · p. 3
• At a query, the observation is a noisy version of the latent output, with noise independent across scalar outputs and latent functions.Evaluation cost is indexed by fidelity.Source: §2.1 · p. 3

3 A Unified Information-Theoretic Acquisition Function

• CMF-PFEV measures information about the feasible Pareto frontier through one mutual-information criterion that incorporates objectives, constraints, and fidelity.The criterion is motivated by the difficulty of selecting both where and at which fidelity to evaluate.Source: §3 · p. 3

3.1 Variational information lower bound

• CMF-PFEV applies a variational information lower bound to frontier-conditioned mutual information, extending PFEV’s mixture-based density to this setting.The direct mutual information is difficult to evaluate, and the paper notes this application had not been explored.Source: §3.1 · pp. 3–4

3.2 Local Pareto conditioning and finite-frontier approximation

• Full frontier conditioning is replaced by a local Pareto-consistency event at x; finite sampled frontiers yield over- and under-truncated regions mixed with λ > 0 to retain support.The event concerns the target-fidelity output even when the observation is made at a lower fidelity.Source: §3.2 · pp. 4–5

3.3 From the variational mixture to event probabilities

• The bound uses probabilities of the same target-fidelity Pareto-consistency event before and after a prospective observation, with the mixture weight optimized for the bound.A low-fidelity observation is useful insofar as it changes this target-fidelity event probability.Source: §3.3 · pp. 6–7

3.4 Cross-fidelity propagation and region probabilities

• Gaussian target-fidelity region probabilities are updated by conditioning on a prospective query; cross-fidelity covariance determines whether that query changes target-fidelity beliefs.Objective-space probabilities are computed by partitioning the finite-frontier dominated region into disjoint hyperrectangles.Source: §3.4 · p. 7

3.5 Monte Carlo approximation and cost-aware query selection

• Nested Monte Carlo samples posterior function paths and prospective observation noise, evaluating each query on the same path as its sampled target frontier.The shared paths preserve posterior dependence between the query output and the sampled frontier.Source: §3.5 · p. 8
• At each discrete fidelity, the method jointly optimizes input and mixture weight, then selects the query with the highest approximate information gain per unit cost.For continuous designs, input and mixture weight are jointly optimized while fidelity is held fixed.Source: §3.5 · pp. 8–9

Multi-objective BO

• PFEV builds a variational information lower bound from over- and under-truncated Pareto-consistent regions, which this paper uses as a foundation.Source: §4 · p. 9

Constrained BO

• Information-based constrained BO quantifies information about feasible optimizers or optima, and Takeno et al. (2022) provide a related variational-bound construction.Source: §4 · p. 10

Multi-fidelity BO

• Multi-fidelity BO uses inexpensive correlated approximations and includes information gain per unit cost, a perspective redirected here from scalar optima to Pareto frontiers.Source: §4 · p. 10

Multi-fidelity multi-objective BO

• MF-OSEMO and MF-HVKG combine multiple objectives with fidelity selection but do not incorporate unknown constraints.Source: §4 · p. 10

Constrained multi-fidelity multi-objective BO

• MFMO-SEGO and MF-CMOBO address constrained multi-fidelity multi-objective BO with separate mechanisms, while MF-JESMOC uses a unified joint-entropy criterion but incurs substantial computational demands.The paper contrasts these approaches with its own formulation based on standard GP inference and cross-fidelity covariance.Source: §4 · p. 10

5 Experiments

• The comparison includes MF-OSEMO, MF-HVKG, MFMO-SEGO, MF-CMOBO, and MF-JESMOC, with feasibility-weighted variants for the two methods that do not handle constraints.MF-JESMOC is omitted from some results because of its high computational cost.Source: §5 · p. 11

GP sample path functions.

• Synthetic GP sample-path problems vary the numbers of objectives, constraints, and input dimensions, with functions generated from the joint GP kernel.Source: GP sample path functions · p. 11

Benchmark functions.

• For C2-DTLZ2 and Tanaka benchmark functions, low-fidelity functions are constructed by adding GP sample paths to the original functions.Source: Benchmark functions · p. 11

5 Experiments

• The LightGBM Covertype task optimizes macro-F1 and negative log model size under a worst-class recall constraint, using 100 and 200 trees at low and high fidelities.The study also includes Waveform and Dry Bean datasets.Source: Benchmark functions · p. 11
• CMF-PFEV performs favorably across GP sample-path, benchmark, and hyperparameter optimization settings, with the strongest advantage on GP sample-path problems.Improvements are also consistent on benchmark and hyperparameter optimization problems.Source: Benchmark functions · p. 12

6 Conclusion

• CMF-PFEV expresses information about the target-fidelity feasible Pareto frontier through Pareto-consistency events, yielding a tractable variational acquisition criterion.Its computation uses Pareto-region probabilities and standard cross-fidelity GP conditioning.Source: §6 · p. 12

A Variational Lower Bound

• Maximizing the variational lower bound is equivalent to minimizing a KL divergence, using nonnegativity of KL divergence in the derivation.Source: A Variational Lower Bound · p. 16

B Overall Procedure

• At each fidelity, the procedure jointly optimizes candidate input and mixture weight, then compares cost-normalized acquisition values across fidelities.Source: B Overall Procedure · p. 16

• Each run starts with five inputs evaluated at both fidelities, shares initial data across ten seeds, and receives an additional budget of 50 highest-fidelity query costs.Coupled query costs are (L+C)κ_m, with per-output costs (1, 10) for synthetic and benchmark tasks and (1, 2) for machine-learning tasks.Source: C Experimental Detail · p. 18
• Except for MF-JESMOC’s deep GP, methods use independent augmented-input GPs; CMF-PFEV uses 10 posterior paths with four observation-noise draws per path and a nominal sampled-frontier size of 50 points.Source: C Experimental Detail · pp. 18–19
• Continuous CMF-PFEV acquisition jointly optimizes input and mixture weight per fidelity, evaluates probabilities with Gaussian CDFs, and selects the best cost-normalized pair.Optimization uses L-BFGS-B with four restarts; pool problems fix each available input and optimize the mixture weight.Source: C Experimental Detail · pp. 17, 19
• MF-OSEMO-PoF and MF-HVKG-PoF apply highest-fidelity feasibility, while MFMO-SEGO selects input by constrained WB2S-MPI and then chooses fidelity by variance reduction per squared cost.MF-HVKG uses fantasy scenarios; MFMO-SEGO has a feasibility-restoration safeguard if no mean-feasible candidate is found.Source: C Experimental Detail · pp. 19–20
• MF-CMOBO alternates weighted multi-fidelity hypervolume improvement and constraint-boundary refinement after feasibility search; MF-JESMOC is limited to two objectives, and MF-RANDOM samples fidelity uniformly.Source: C Experimental Detail · p. 20
• Recommendations are scored by noise-free highest-fidelity IRHV, using posterior-mean solutions and a shared hypervolume reference; figures report running-best averages over ten seeds with one-standard-error bands.Source: C Experimental Detail · p. 20

C.1 GP sample path functions

• Each objective and constraint is an independent fixed prior draw with mean zero, variance one, isotropic input lengthscale, and fidelity lengthscale 5.The settings use 4096 random Fourier features drawn from Matérn-5/2 and RBF spectral distributions.Source: §C.1 · p. 20
• Observation noise has standard deviation 0.01 at both fidelities, and constraint thresholds are zero.Source: §C.1 · p. 20
• The GP-generated test-function problems are fixed across BO seeds.Source: Table 1 · p. 20

C.2 Benchmark functions

• Tanaka uses (L, C, d) = (2, 2, 2), while C2-DTLZ2 uses (3, 1, 7) and (5, 1, 9).Objectives are negated for maximization.Source: §C.2 · p. 20
• Constraints are expressed as nonnegative feasible margins, with zero thresholds and observation-noise standard deviation 0.01 at both fidelities.Source: §C.2 · p. 21
• For each output, an independent RBF-GP discrepancy uses lengthscale 0.30 and 1024 Fourier features.Its mean and standard deviation, and the highest-fidelity output standard deviation, are calculated at 4096 fixed uniform calibration inputs.Source: §C.2 · p. 21

C.3 LightGBM hyperparameter selection

• All LightGBM dataset settings use 512 candidates, L = 2, and C = 1.Source: Table 2 · p. 21
• Each pool contains 512 fixed scrambled Sobol inputs, with logarithmic mappings for learning rate, leaves, and minimum samples per leaf.Dry Bean and Covertype also vary class-weight ratio ρ from 0.1 to 10; integer parameters are rounded.Source: §C.3 · p. 21
• Low and high fidelities use 100 and 200 boosting rounds, respectively, with deterministic GBDT training and precomputed candidate outputs.No row or feature subsampling, L1/L2 regularization, or added observation noise is used.Source: §C.3 · p. 21
• Waveform and Covertype maximize validation macro-F1 and negative log-base-10 model size, constrained by minimum class-wise recall.Dry Bean maximizes macro-average recall for rare and common groups, constrained by validation accuracy.Source: §C.3 · p. 21
• Feasibility thresholds use the 0.70 quantile of full-pool high-fidelity values for Waveform and Covertype, and the 0.50 quantile for Dry Bean.The fixed thresholds apply at both fidelities.Source: §C.3 · p. 21
• IRHV recommendations are posterior-mean pool recommendations pruned by crowding distance to the feasible frontier sizes listed in Table 3.This evaluation rule applies to every method, and the sizes are not used by acquisition optimization.Source: §C.3 · p. 22, Table 3 · p. 22

D Computational Time

• MFMO-SEGO and MF-CMOBO are relatively faster, while information-based and knowledge-gradient approaches are slower; MF-JESMOC takes longer.Table 4 reports next-query selection time for a GP-sampled problem with two objectives, two constraints, five dimensions, two fidelities, and 100 observations.Source: D Computational Time · p. 22, Table 4 · p. 23
• MF-JESMOC’s longer query-selection time is attributed to estimating complicated deep GPs and variational approximations.Source: D Computational Time · p. 22
• Table 4 reports median and quartile times over five seeds, measured with one CPU thread.Source: Table 4 · p. 23