Source-linked AI summary
Multi-fidelity regression using artificial neural networks: efficient approximation of parameter-dependent output quantities
Mengwu Guo, Andrea Manzoni, Maurice Amendt, Paolo Conti, Jan S. Hesthaven
TL;DR
When high-fidelity experiments are costly and sparse, multi-fidelity regression can use cheaper low-fidelity data to approximate their outputs. This paper proposes and tunes neural-network models for that setting, reports that they outperform co-kriging across numerical examples, and applies them to an acoustic-horn problem.
Problem
High-fidelity numerical or physical evaluations can be expensive and scarce, motivating regression methods that use cheaper, correlated low-fidelity data.
Source: §1 · p. 2Method
Results
Across numerical examples, multi-fidelity neural networks consistently outperform co-kriging, and in the parametrized PDE application prediction accuracy is driven more by reduced-order-model reliability than by the amount of high-fidelity data.
Source: §6 · pp. 21–22Takeaways & Limitations
For the acoustic-horn application, multi-fidelity neural networks combine reduced-basis low-fidelity evaluations with finite-element high-fidelity evaluations to predict the output functional.
Source: §6 · p. 21The current schemes leave open whether adapting the balance parameter α during training could improve accuracy as the high- and low-fidelity MSE ratio evolves.
Source: §6 · p. 22
Abstract
from arXiv · showhide
Highly accurate numerical or physical experiments are often time-consuming or expensive to obtain. When time or budget restrictions prohibit the generation of additional data, the amount of available samples may be too limited to provide satisfactory model results. Multi-fidelity methods deal with such problems by incorporating information from other sources, which are ideally well-correlated with the high-fidelity data, but can be obtained at a lower cost. By leveraging correlations between different data sets, multi-fidelity methods often yield superior generalization when compared to models based solely on a small amount of high-fidelity data. In this work, we present the use of artificial neural networks applied to multi-fidelity regression problems. By elaborating a few existing approaches, we propose new neural network architectures for multi-fidelity regression. The introduced models are compared against a traditional multi-fidelity scheme, co-kriging. A collection of artificial benchmarks are presented to measure the performance of the analyzed models. The results show that cross-validation in combination with Bayesian optimization consistently leads to neural network models that outperform the co-kriging scheme. Additionally, we show an application of multi-fidelity regression to an engineering problem. The propagation of a pressure wave into an acoustic horn with parametrized shape and frequency is considered, and the index of reflection intensity is approximated using the multi-fidelity models. A finite element model and a reduced basis model are adopted as the high- and low-fidelity, respectively. It is shown that the multi-fidelity neural network returns outputs that achieve a comparable accuracy to those from the expensive, full-order model, using only very few full-order evaluations combined with a larger amount of inaccurate but cheap evaluations of a reduced order model.
1. Introduction
Multi-fidelity regression uses correlations between scarce, costly high-fidelity data and cheaper low-fidelity data to improve approximation where high-fidelity samples are sparse. This work proposes neural-network approaches, tunes their hyperparameters, and evaluates them on benchmarks and an engineering application.
1. Introduction
- High-fidelity samples are costly and often scarce, so multi-fidelity regression uses correlated low-fidelity data to approximate the high-fidelity model where its samples are sparse.Low-fidelity data are assumed to be cheaper but less accurate, and ideally capture major trends of the problem.Source: §1 · p. 2
- Co-kriging offers uncertainty estimates but Gaussian-process regression can suffer from the curse of dimensionality, motivating methods that detect nonlinear correlations across fidelity levels.Co-kriging is a widely used multi-fidelity technique based on vector-valued Gaussian processes.Source: §1 · p. 2
- The paper proposes two all-in-one and two multilevel neural-network models, and uses cross-validation with Bayesian optimization to select hyperparameters.The all-in-one models train fidelity levels together; the multilevel models define separate networks for the fidelity hierarchy.Source: §1 · p. 2
- The proposed models are assessed on manufactured benchmarks against single-fidelity regression and co-kriging.The benchmarks are chosen to test desirable model properties.Source: §1 · p. 3
- The engineering application predicts an acoustic horn’s reflection-intensity index from shape and frequency parameters, combining many reduced-order evaluations with a few full-order evaluations.The reduced-basis model supplies low-fidelity data and the finite-element model supplies high-fidelity data.Source: §1 · p. 3
2. Artificial neural networks and Gaussian processes for regression
This section describes a fully connected neural network as a vector-valued surrogate trained on input-output data by minimizing a cost function that includes error and regularization.
2.1. Artificial neural networks (ANNs)
- A fully connected network has L hidden layers with layer-specific widths and a nonlinear activation function.The passage denotes neuron pre- and post-activation values and the network’s input and output vectors.Source: §2.1 · p. 3
- The network surrogate approximates a multivariate function from input-output training pairs, with weights and biases collected as its trainable parameters.The input x and output y are vectors in the network’s input and output spaces.Source: §2.1 · pp. 3–4
- Network training minimizes a cost function comprising mean square error and a regularization term weighted by λ ≥ 0.The passage identifies λ as the penalty coefficient.Source: §2.1 · p. 4
Single-fidelity GPR
Single-fidelity Gaussian process regression places a Gaussian-process prior with independent Gaussian noise on the regression function, then conditions on training data to predict noise-free outputs.
Single-fidelity GPR
- The regression function has a zero-mean Gaussian-process prior with covariance kernel κ and independent Gaussian noise of standard deviation χ.The kernel gives the covariance of the prior process.Source: Single-fidelity GPR · p. 4
- Given training pairs, the corresponding outputs have a joint Gaussian distribution with covariance equal to the kernel matrix plus the noise variance times the identity matrix.The training inputs and outputs are denoted by X and y, respectively.Source: Single-fidelity GPR · p. 4
- Conditioning on the training data yields a posterior predictive Gaussian process for the noise-free output at a new test input.The target is f*(s) for a new input s.Source: Single-fidelity GPR · p. 4
Multi-fidelity GPR
Multi-fidelity Gaussian-process regression combines data from fidelity levels under a correlation model, while the paper also explains how neural-network priors connect to Gaussian processes under specified assumptions.
Multi-fidelity GPR
- Co-kriging, or vector-valued Gaussian-process regression, can use many cheap low-fidelity samples and fewer high-fidelity samples, assuming linear correlation between fidelity levels.The linear model of coregionalization represents each solution level as a linear combination of independent Gaussian processes.Source: Multi-fidelity GPR · p. 4
- In the two-level setting, AR(1) co-kriging is a special case of the linear model defining the low- and high-fidelity prior.Conditioning on both levels’ training pairs yields a posterior process for the high-fidelity output.Source: Multi-fidelity GPR · p. 5
2.3. The link between ANNs and GPR
- The neural-network-to-Gaussian-process result assumes independent, identically distributed weights and biases and increasing layer widths, with zero-mean weight distributions and specified variances.The result is established for network layers under these probabilistic assumptions.Source: §2.3 · p. 5
- The proof uses induction to establish Gaussian-process pre-activations at intermediate layers, with kernels defined recursively.It considers arbitrary finite input locations and applies a multivariate central limit theorem in the second layer.Source: §2.3 · p. 5
- The network outputs consequently follow independent, identically distributed Gaussian-process priors whose kernel is formed by the recursive relation.The recursive relation is built from the layer kernels.Source: §2.3 · p. 5
3. Artificial neural networks for multi-fidelity regression
Section 3 develops neural-network regressors for bi-fidelity problems, using low-fidelity information to approximate a high-fidelity function. It contrasts simultaneous all-in-one learning with sequential multilevel models and proposes modifications to both.
3.1. All-in-one NNMFR
- The Intermediate model places the LF output in an intermediate layer of the HF surrogate, while GPmimic uses an LMC-inspired architecture to mimic multi-fidelity GPR.The two proposed all-in-one models share inputs for HF and LF data; Intermediate models their mapping directly, while GPmimic produces outputs as affine combinations of two latent variables.Source: §3.1 · p. 6, Figure 1 · p. 7
- Intermediate and GPmimic use a weighted objective for HF and LF errors, with α tuned by k-fold cross-validation and Bayesian optimization.α balances the two fidelity levels; α = 0 and α = 1 correspond to LF-only and HF-only regression, respectively.Source: §3.1 · pp. 7–8
3.2. Multilevel NNMFR
- The modified 2-step model trains an LF network first, then an HF network on LF predictions at HF locations; 3-step adds a linear-correlation network before the final HF network.The 3-step model is expected to help over 2-step only when HF and LF data have strong linear correlations.Source: §3.2 · p. 8
- Multilevel models learn the LF function and then the HF–LF correlation sequentially, leaving the degree of trust in LF data to training; the LF model can also use another technique, such as gradient boosting for discontinuities.By contrast, all-in-one models learn HF and LF functions together and can encode prior trust in LF data through α or the Intermediate model’s layer width.Source: §3.2 · p. 9
- For non-hierarchical sources with similar fidelity, GPmimic is the only proposed architecture usable without adaptation.Source: §3.2 · p. 9
4. Numerical results (I): benchmark test cases
The benchmark study evaluates proposed neural-network multi-fidelity models against co-kriging, using cross-validation and Bayesian optimization to tune neural-network hyperparameters. Results vary with the relationship between fidelity levels and the problem dimension.
4. Numerical results (I): benchmark test cases
- Neural-network hyperparameters were optimized with k-fold cross-validation and Bayesian optimization, and models were evaluated by test-set MSE and computational time.GP hyperparameters were obtained by maximizing marginal likelihood.Source: §4 · p. 9
4.1. Benchmark case 1: Linear correlation
- Co-kriging and the 3-step model outperformed other models by one or two orders of magnitude; all ANNs performed worse than classic GPR and cost more computationally.Except for the 2-step model, validation errors were very conservative estimates of test error.Source: §4.1 · p. 10
4.2. Benchmark case 2: Discontinuous function
- Both multilevel NNs achieved R2 scores above 0.99; other NNs smoothed the discontinuity, while co-kriging fluctuated around x = 0.5.HF-only models could not approximate the discontinuity, but including LF data significantly improved regression.Source: §4.2 · p. 12, Figure 3 · p. 13
4.3. Benchmark case 3: Nonlinear correlation
- Neither co-kriging nor GPmimic leveraged LF data for nonlinear correlation; GPmimic scored R2 = 0.128, and the task challenged prediction of HF local extrema.The authors attribute the model failures to mathematical structures unable to detect nonlinear correlations between fidelity levels.Source: §4.3 · p. 12, Figure 4 · p. 14
4.4. Benchmark case 4: 20-D benchmark
- In this case, co-kriging required inverting a 35,000×35,000 matrix, and the GPU-accelerated 3-step NN, tuned by 5-fold cross-validation, accurately predicted the value of yHF at one million random input locations.Storing such a matrix alone can take around 10 GB of memory; this extensive computational cost made using a GPR model impossible in this case.Source: §4.4 · p. 14
5. Numerical results (II): application to parametrized PDEs
The acoustic-horn study evaluates multi-fidelity neural networks using finite-element full-order and reduced-basis reduced-order models, first for frequency alone and then for frequency plus four shape parameters. Across both cases, prediction quality depends more on reduced-model accuracy than on increasing the number of full-order samples.
5. Numerical results (II): application to parametrized PDEs
- The engineering application predicts the horn’s reflection-intensity index from a finite-element HF model and a fast, reduced-basis LF model.The one-parameter case varies frequency over [10, 1800] without varying horn geometry.Source: §5 · pp. 14–15, §5.1 · p. 16
5.1. Case with p = 1 parameter
- For one parameter, 2-step and 3-step models performed similarly and better than the Intermediate model; multilevel models exceeded R2 = 0.8 with N = 5, while Intermediate needed at least 12 bases.The 3-step model captured smaller-amplitude peaks above 1000 better than the 2-step model.Source: §5.1 · pp. 16–17, Figure 10 · p. 20, Figure 11 · p. 21
- For the one-parameter case, each fixed basis count had an HF-data threshold beyond which additional HF samples did not improve fit; better LF models reduced the importance of HF-data quantity.The study concludes that a good LF model with fewer HF samples can be more efficient than many HF samples paired with a poor LF model.Source: §5.1 · p. 17
5.2. Case with p = 5 parameters
- The five-parameter experiment varied frequency and four geometric parameters, using 500 LF samples and testing 5–45 HF samples with ROM dimensions from 5 to 40.The reported results focus on the 3-step architecture.Source: §5.2 · pp. 17, 19
- With five parameters, R2 reached above 0.97 and MSE below 1.2 × 10−3 starting at 20 bases, for a fixed HF-data count greater than 5.At N = 15, R2 rose from 0.12768 with 5 HF data to 0.91093 with 40; at N = 35, increasing HF data from 10 to 45 improved R2 by 0.2%, from 0.98882 to 0.99107.Source: §5.2 · p. 20
5.1. Case with p = 1 parameter; 5.2. Case with p = 5 parameters
- In both parameter settings, increasing HF data and ROM basis count improved prediction, but ROM dimension mattered more as the reduced model became more accurate.In the five-parameter case, accuracy flattened with more HF data at larger basis counts, while adding bases continued to improve the indices.Source: §5.1 · p. 17, §5.2 · pp. 19–20
6. Conclusions
The paper presents four ANN architectures for multi-fidelity regression and evaluates them on manufactured benchmarks and a parametrized PDE. Its reported results show better performance than co-kriging and identify reduced-order-model reliability as the main driver of PDE prediction accuracy.
6. Conclusions
- Across numerical examples, multi-fidelity neural networks consistently outperformed co-kriging, and cross-validation with Bayesian optimization yielded very low prediction errors.The authors attribute an advantage over co-kriging to better detection of nonlinear cross-fidelity correlations and the ability to handle large data sets.Source: §6 · p. 21
- The study presents four ANN schemes and applies them to manufactured benchmarks and a parametrized PDE output-quantity problem.Source: §6 · p. 21
- For the parametrized PDE application, prediction accuracy was driven mainly by reduced-order-model reliability rather than the amount of HF data.The evaluation targets an output functional of the PDE solution, with a reduced-basis LF model and detailed finite-element HF model.Source: §6 · pp. 21–22
Appendix A: hyperparameter summary
The appendix reports selected hyperparameter patterns across artificial benchmarks and the acoustic-horn problems. The chosen optimizer was usually AdaMax, while preferred initialization differed between the benchmarks and horn application.
Appendix A: hyperparameter summary
- In all test cases, optimization selected α < 0.1, weighting LF data about one order of magnitude more than HF data.In all-in-one networks, α balances the fidelity levels’ contributions to training error.Source: Appendix A · p. 24
- Learning rates for models within the same NNMFR class were generally of the same order of magnitude.Source: Appendix A · p. 24
- AdaMax was the best-performing optimizer in most cases.Source: Appendix A · p. 24
- Glorot uniform initialization was preferred for benchmarks, while standard uniform initialization performed best for the horn’s IRI prediction.Source: Appendix A · p. 24
Appendix B: HF and LF models for the acoustic horn problem
The appendix constructs the HF acoustic model with Galerkin finite elements and the LF model by projecting the HF problem onto a reduced-basis space. Affine dependence enables efficient online assembly, while nonaffine shape dependence is handled with hyper-reduction.
Appendix B: HF and LF models for the acoustic horn problem
- The HF model discretizes the weak PDE formulation with Galerkin finite elements, yielding a linear system for the finite-element coefficients.Source: Appendix B · pp. 24–25
- The reduced-basis model approximates the HF solution as a linear combination of basis functions and projects the problem onto a reduced space of dimension N much smaller than n.The basis is built from selected HF snapshots, for example using POD, followed by a (Petrov-)Galerkin projection.Source: Appendix B · p. 25
- With affine parameter dependence, precomputed projected arrays make reduced-system assembly efficient during the online stage.Source: Appendix B · p. 26
- For shape-deformed problems with nonaffine HF arrays, DEIM and MDEIM approximate affine decompositions for the reduced model.The resulting reduced system can be solved quickly and inexpensively to generate LF training outputs.Source: Appendix B · p. 26