Deep Gaussian Processes for Multi-fidelity Modeling

1 Introduction

• Multi-fidelity modeling combines limited high-fidelity observations with cheap low-fidelity approximations, but naive fusion can yield predictions that do not reflect the true system.Bayesian uncertainty about fidelity reliability and relationships provides implicit regularization when data are limited.Source: §1 · p. 1
• Autoregressive GP models are effective for linear fidelity mappings but are insufficient when fidelity relationships require a space-dependent nonlinear transformation.The cited AR1 formulation uses a constant scaling of the preceding fidelity and a bias term.Source: §1 · pp. 1–2

Problem Statement

• NARGP uses a disjointed hierarchy of sequentially fitted GPs, preventing lower-fidelity GPs from later updates and increasing susceptibility to overfitting.The paper contrasts this with jointly trained modeling that retains uncertainty across fidelities.Source: Problem Statement · p. 2
• Where only low-fidelity points remain, NARGP predicts an incorrect result with high confidence, despite the expected uncertainty from missing high-fidelity observations.The paper notes that this is particularly problematic for computational pipelines or active learning, where uncertainty calibration is imperative.Source: Problem Statement · p. 2
• MF-DGP is presented as an end-to-end multi-fidelity DGP that captures nonlinear relationships, propagates uncertainty across fidelities, and returns conservative uncertainty estimates.Its compositional structure also alleviates a limitation of AR1; the formulation uses a sparse DGP approximation for tractability.Source: Problem Statement · p. 2
• The paper evaluates MF-DGP against other techniques, applies it to a large-scale real-world problem, and demonstrates its use for experimental design with a determinantal point process.Its implementation is integrated into Emukit, an open-source package for emulation and decision making in a design loop.Source: Problem Statement · p. 2

2 Related Work

• Kennedy & O’Hagan (2000) modeled linear correlations across ordered fidelities with a GP, while Le Gratiet & Garnier (2014) used independent GPs to reduce complexity and match the original posterior under noiseless observations and nested inputs.The equivalence stated for the recursive model is conditional on both noiseless observations and nested inputs.Source: §2 · p. 3
• Perdikaris et al. (2017) introduced NARGP through uncertain-output propagation, but its sequentially isolated GPs cannot update lower-fidelity models after fitting.The paper says this disjointed formulation disregards nuances of jointly modeling the data across fidelities.Source: §2 · p. 3
• Raissi & Karniadakis (2016) proposed DEEP-MF, which learns deterministic input transformations but resembles a manifold GP rather than a model with actual process composition.Source: §2 · p. 3
• Other extensions target separate issues, including scalability, mismatched training and target distributions, gradient information, non-hierarchical fidelities, and methods for Bayesian optimization and bandits.Source: §2 · p. 3

• GPs model noisy observations through Bayesian inference, with covariance parameters defining the prior and posterior mean and covariance characterizing candidate functions.Source: §3 · p. 3
• Deep Gaussian processes propagate one GP’s output as the next GP’s input, and fidelity-specific layers give intermediate functions a meaningful interpretation.Source: §3 · p. 3
• DGP inference is generally intractable, while earlier multi-fidelity adaptations used structural assumptions that impaired model capacity and flexibility.Recent GP approximations make DGPs easier to specify and train; this paper adapts the Salimbeni and Deisenroth approach to avoid kernel-selection constraints.Source: §3 · p. 4

4.1 Model Specification

• Each GP layer corresponds to a fidelity and conditions on level-specific observations and preceding-fidelity evaluations; sparse variational inference uses inducing variables with higher-layer inputs combining locations and prior outputs.Source: §4.1 · p. 4, Layer-wise Sparse Approximation · pp. 4–5

Evidence Lower Bound

• The multi-fidelity objective is a variational lower bound that sums expected likelihood terms across observations and fidelities and subtracts inducing-variable KL divergences.Source: Evidence Lower Bound · p. 5

4.2 Multi-fidelity Predictions

• Predictions at a chosen fidelity are obtained recursively, augmenting each intermediate layer’s output with the original input and propagating through the model.The procedure uses Monte Carlo samples and one layer per fidelity.Source: §4.2 · p. 5

4.3 Multi-fidelity Covariance

• The intermediate-layer covariance captures potentially nonlinear fidelity mappings and input-space correlations; adding a linear kernel accommodates mappings that may be linear.Source: §4.3 · p. 5

4.4 Treatment of Inducing Inputs

• Because higher-layer inputs link each location to its preceding-layer evaluation, inducing points there are selected from prior-fidelity observations and fixed during optimization.More principled constraints on their optimization are left for future work.Source: §4.4 · pp. 5–6

4.5 Stochastic Variational Inference

• Training initially fixes variational and noise variances for stability, then jointly optimizes them; mini-batches can include scarce high-fidelity observations at every step while subsampling lower-fidelity data.Source: §4.5 · p. 6

4.6 Complexity

• When most observations are at lower fidelities, MF-DGP is expected to train faster than a regular DGP.Source: §4.6 · p. 6

4.7 Comparison to NARGP and DEEP-MF

• Unlike NARGP, MF-DGP jointly optimizes inducing points and kernel parameters under one objective, lifting nested-input and noise-free lower-fidelity constraints that can be difficult to enforce.The paper also notes that practical SVI for multi-fidelity modeling is novel to this work.Source: §4.7 · p. 6

5 Experiments

• The experiments progress from synthetic illustrations and established benchmarks to a large real-world dataset and an experimental-design demonstration.The real-world experiment uses nearly a million observations.Source: §5 · pp. 6–7

5.1 Synthetic Examples

• In synthetic cases with limited or spatially constrained high-fidelity data, MF-DGP gives conservative uncertainty estimates and superior test MNLL compared with AR1, NARGP, and DEEP-MF.The examples include linear and nonlinear mappings between fidelities.Source: §5.1 · p. 7, Figure 3 · p. 7, Figure 4 · p. 8

5.3 Large-scale Real-world Experiment

• With 800,000 low-fidelity points and 1,000 high-fidelity observations, MF-DGP achieves test RMSE 0.063 versus 0.096 for an exact GP using only high-fidelity data.The test set contains 10,000 high-fidelity points; Figure 5 visualizes the MF-DGP predictions.Source: §5.3 · p. 7, Figure 5 · p. 9

5.2 Benchmark Comparison

• On 3-level Branin, MF-DGP significantly outperforms NARGP on all metrics, while AR1 cannot properly model the high-fidelity data.The benchmark evaluation reports R-squared, RMSE, and MNLL across five multi-fidelity problems.Source: §5.2 · p. 7, Table 1 · p. 8

5.4 Experimental Design with MF-DGP

• DPP sampling uses MF-DGP posterior covariance and mean to select diverse locations that emphasize predicted high infection rates; selected points are dispersed and concentrate in those areas.The procedure samples 50 high-fidelity locations from 2,500 candidates.Source: §5.4 · pp. 8–9, Figure 5 · p. 9

6 Conclusion

• MF-DGP captures nonlinear fidelity relationships with reduced overfitting and consistently yields superior uncertainty quantification and propagation across fidelity levels.The conclusion identifies these capabilities as important for reliable decision making when unbiased observations are scarce or difficult to obtain.Source: §6 · p. 9
• The compositional structure hinders analytic acquisition-function expressions used in Bayesian optimization or quadrature, while constrained inducing-variable optimization remains an open challenge.The conclusion identifies this optimization as key to extending MF-DGP’s learning capacity.Source: §6 · p. 9

A Further Model Details

• The appendix extends the main paper’s model description, emphasizing the proposed multi-fidelity extension rather than the underlying DGP approximation.Source: A Further Model Details · p. 11

A.1 Approximating the Marginal Likelihood of MF-DGP

• Each MF-DGP layer is assumed to correspond to a realization of the process at one fidelity level, for datasets containing T fidelities.Source: §A.1 · p. 11
• The appendix derives a variational lower bound on the multi-fidelity model’s marginal likelihood, and the KL term is analytic when both distributions are Gaussian.Source: §A.1 · pp. 11–12

A.2 Reparameterization Trick

• The reparameterization trick recursively draws samples from the variational posterior during stochastic variational inference.Source: §A.2 · p. 12

B Additional Detail on Experiments

• The appendix supplies experimental-evaluation details that were excluded from the main paper due to space constraints.Source: B Additional Detail on Experiments · p. 12
• The supplied appendix context identifies Table 2 as the source of synthetic-function details but does not describe its individual entries.Source: B Additional Detail on Experiments · p. 12, Table 2 · p. 13
• Table 2 gives details of the synthetically constructed functions used in the experimental evaluation.Source: Table 2 · p. 13

B.1 Mapping Between Fidelities for Synthetic Examples

• Four synthetic examples comprise two linear and two nonlinear mappings between fidelities, with their precise definitions given in Table 2.Source: §B.1 · p. 13
• Figure 6’s top row shows the synthetic functions, while its bottom row shows mappings from low- to high-fidelity observations.Source: Figure 6 · p. 13, §B.1 · p. 13
• All methods worked best when fidelity outputs were scaled down, particularly to ensure optimization convergence, while preserving the functions’ original fidelity relationships.Source: §B.1 · p. 13

B.2 Specification of Benchmark Problems

• CURRIN is a two-dimensional computer-experiment benchmark with input domain x ∈ [0, 1]2 and specified high- and low-fidelity variants.The passage provides the high-fidelity function and describes the low-fidelity alternative.Source: §B.2 · p. 14
• PARK is a four-dimensional benchmark whose inputs all lie in [0, 1], with a specified high-fidelity function.The section also includes a low-fidelity formula fragment among the benchmark specifications.Source: §B.2 · p. 14
• The BOREHOLE benchmark is a two-level physical model of water flow with eight inputs constrained to specified ranges.The passage lists the domain for each of the eight parameters and gives high- and low-fidelity simulations.Source: §B.2 · p. 14
• The BRANIN benchmark has three fidelity levels over a two-dimensional domain [−5, 10] × [0, 15].Its specification is attributed to Perdikaris et al. (2017).Source: §B.2 · p. 14
• The HARTMANN-3D benchmark has three fidelity levels over the three-dimensional domain [0, 1]3.Its specification follows Kandasamy et al. (2016).Source: §B.2 · p. 15

AR1

• AR1 uses the Kennedy & O’Hagan (2000) formulation because it is easier to adapt to non-nested inputs than the Le Gratiet & Garnier (2014) procedure.Independent noise parameters by fidelity are jointly optimized with kernel hyperparameters and scaling factors.Source: §B.3 · p. 15

NARGP

• NARGP fits an individual GP at each fidelity sequentially, optimizing kernel parameters first with fixed noise variance and then jointly with noise freed.The strategy follows Perdikaris et al. (2017).Source: §B.3 · p. 15

DEEP-MF

• DEEP-MF uses the original two-layer sigmoid neural network transformation, shares process noise across fidelities, and had no pre-existing code available.The authors note that selecting the deterministic nonlinear transformation is challenging and lacks a straightforward configuration approach.Source: §B.3 · p. 15