PaperlayerView exact revision on arXiv

Source-linked AI summary

Gradient-enhanced kriging for high-dimensional problems

Mohamed Amine Bouhlel, Joaquim R. R. A. Martins

arXiv:1708.02663v1cs.LGstat.ML

TL;DR

Accurate surrogate modeling is costly in high dimensions, and gradient-enhanced kriging faces scaling limits in its correlation matrices and hyperparameter estimation. GE-KPLS uses PLS to reduce hyperparameters and include only relevant approximation points; reported comparisons show improved accuracy or lower sampling needs, with large gains in some cases.

  • Problem

    High-dimensional kriging and gradient-enhanced kriging are difficult to construct because hyperparameter optimization and correlation-matrix growth become costly.

    Source: §2.1 · p. 6, §3.1 · p. 9

  • Method

    GE-KPLS uses PLS information and gradient-based Taylor approximations to reduce hyperparameters and add only relevant points to the correlation matrix.

    Source: §1 · pp. 4–5, §3.3.2 · p. 12

  • Results

    In some cases, GE-KPLS was over 3 times more accurate and over 3200 times faster than indirect GEK, while comparisons also found it required fewer samples or gave greater accuracy for a fixed budget.

    Source: §6 · p. 21

  • Takeaways & Limitations

    The reported comparisons indicate that GE-KPLS can provide accurate high-dimensional surrogate models with fewer sampling points than conventional methods.

    Source: §6 · pp. 20–21

    The current toolbox cannot handle problems with both many dimensions and many sampling points because correlation-matrix inversion requires substantial memory.

    Source: §5 · p. 20

Abstract

from arXiv · show

Surrogate models provide a low computational cost alternative to evaluating expensive functions. The construction of accurate surrogate models with large numbers of independent variables is currently prohibitive because it requires a large number of function evaluations. Gradient-enhanced kriging has the potential to reduce the number of function evaluations for the desired accuracy when efficient gradient computation, such as an adjoint method, is available. However, current gradient-enhanced kriging methods do not scale well with the number of sampling points due to the rapid growth in the size of the correlation matrix where new information is added for each sampling point in each direction of the design space. They do not scale well with the number of independent variables either due to the increase in the number of hyperparameters that needs to be estimated. To address this issue, we develop a new gradient-enhanced surrogate model approach that drastically reduced the number of hyperparameters through the use of the partial-least squares method that maintains accuracy. In addition, this method is able to control the size of the correlation matrix by adding only relevant points defined through the information provided by the partial-least squares method. To validate our method, we compare the global accuracy of the proposed method with conventional kriging surrogate models on two analytic functions with up to 100 dimensions, as well as engineering problems of varied complexity with up to 15 dimensions. We show that the proposed method requires fewer sampling points than conventional methods to obtain the desired accuracy, or provides more accuracy for a fixed budget of sampling points. In some cases, we get over 3 times more accurate models than a bench of surrogate models from the literature, and also over 3200 times faster than standard gradient-enhanced kriging models.

Symbols and notation

The notation defines the dimensions of the design space and sample set, then names the vectors, matrices, and kriging quantities used in the paper.

Symbols and notation

  • Bold type denotes matrices and vectors; θi denotes a covariance parameter, σ2 the process variance, and Δxj a first-order Taylor step in direction j.Source: Symbols and notation · p. 2
  • d is the number of dimensions, n the number of sampling points, h the number of principal components, and B the design hypercube.Source: Symbols and notation · p. 2
  • X contains the n sampling points, while y contains their evaluated outputs; x(i) and y(i) denote the ith sample and output.Source: Symbols and notation · p. 2
  • k is the covariance function, rxx′ the spatial correlation, R the covariance matrix, and s2(x) the predicted kriging variance.Source: Symbols and notation · p. 2

1 Introduction

The introduction motivates surrogate modeling as a lower-cost alternative to repeated expensive evaluations, while noting that gradient-enhanced kriging becomes difficult to scale. It presents GE-KPLS as a PLS-based approach to reduce hyperparameters and control correlation-matrix growth.

1 Introduction

  • GEK can improve model accuracy when gradients are available, especially with adjoint computation, whose cost is independent of the number of variables.Source: §1 · p. 3
  • GEK scales poorly as dimensions or sample counts rise because its correlation matrix grows, may become ill-conditioned, and requires dimension-proportional hyperparameter optimization.Source: §1 · p. 3
  • KPLS reduces hyperparameters to the retained principal components and showed similar accuracy to kriging with construction up to 450 times faster [Bouhlel et al., 2016b].Source: §1 · p. 4
  • KPLSK uses a reduced-space optimization followed by a full-space refinement, improving accuracy over KPLS and kriging while remaining slower than KPLS [Bouhlel et al., 2016a].Source: §1 · p. 4
  • GE-KPLS combines gradients with KPLS, using first-order Taylor approximations and PLS-selected relevant points to control correlation-matrix size.Source: §1 · p. 4

2 Kriging surrogate modeling

The section outlines conventional kriging and the KPLS family, from sample-based model construction to PLS-based reduction of hyperparameter search. It also describes KPLSK’s two-stage optimization as a refinement of KPLS.

2.1 Conventional kriging

  • Kriging constructs a surrogate from evaluated sample points and uses a Gaussian-process interpolation model with a constant mean and estimated covariance parameters.Source: §2 · p. 5, §2.1 · pp. 5–6
  • Estimating kriging hyperparameters requires numerical likelihood optimization, which becomes prohibitive above 10 dimensions because of correlation-matrix determinant costs and high-dimensional multimodal search.Source: §2.1 · p. 6

2.2 KPLS(K)—Accelerating kriging construction with partial-least squares regression

  • KPLS uses PLS information about input contributions to reduce the parameters to estimate from d to h, where h is much smaller than d.Source: §2.2 · pp. 7–8
  • PLS components are computed sequentially by regressing the input matrix and outputs on each component and updating residuals for subsequent components.Source: §2.2 · p. 7
  • KPLSK optimizes first in the reduced h-parameter space, then refines a conventional kriging model in d dimensions using that solution as its starting point.Source: §2.2 · p. 8

3 Gradient-enhanced kriging

Gradient-enhanced kriging uses derivatives to enrich surrogate models, but conventional formulations can become computationally difficult as dimensions and sample counts grow. GE-KPLS uses local PLS information to reduce hyperparameters and select added points, controlling correlation-matrix growth.

3 Gradient-enhanced kriging

  • Gradient information can enrich a surrogate when available, and adjoint-computed gradients can cost about as much as one function evaluation, independently of dimension.The section introduces indirect and direct GEK before presenting GE-KPLS.Source: §3 · p. 9

3.1 Indirect gradient-enhanced kriging

  • Indirect GEK adds one linearly extrapolated point per input direction at each sampling point, growing the matrix to n(d + 1) × n(d + 1).The resulting near-collinear columns can make the matrix ill-conditioned and model construction prohibitive in high dimensions.Source: §3.1 · p. 9

3.2 Direct gradient-enhanced kriging

  • Direct GEK includes derivative values in the data vector, producing a dense, non-symmetric correlation matrix whose size grows quadratically with variables and samples.The matrix includes correlations among data, gradients, and cross-pairs.Source: §3.2 · pp. 9–10

3.3 GE-KPLS—Gradient-enhanced kriging with partial-least squares method

  • GE-KPLS reduces the number of hyperparameters and controls correlation-matrix growth while incorporating gradient information.The approach targets scaling difficulties associated with many inputs and sampling points.Source: §3.3 · p. 10

3.3.1 Model construction

  • Locally applying PLS to FOTA-generated point sets around each sample yields coefficients that are averaged across sampling points for the model construction.The sets use a Box–Behnken design for d ≥ 3 or forward and backward variations in 2d-space.Source: §3.3.1 · p. 11

3.3.2 Controlling the size of the correlation matrix

  • For each sampling point, GE-KPLS adds only m approximating points in directions with the highest first-principal-component PLS coefficients.This raises the matrix size to n(m + 1) × n(m + 1), with m << d.Source: §3.3.2 · p. 12

4 Numerical experiments

The numerical studies compare GE-KPLS with kriging and other surrogate models on analytic functions up to 100 dimensions and engineering problems spanning 2–15 dimensions. They evaluate relative error and construction time across sampling budgets.

4 Numerical experiments

  • Gradient-based models used half as many sampling points as other models to account for gradient cost, which is roughly equal to or less than function-evaluation cost with an adjoint method.The experiments used 10 trials and 5,000 validation points, reporting mean results.Source: §4 · p. 14
  • The FOTA step was tuned for the second and third studies, but tuning time was excluded; the first study used the recommended step because GEK could be expensive.The recommended step was 10^-4 times the bound interval length in each direction.Source: §4 · p. 14

4.1 Numerical results for the analytical functions: first study

  • GE-KPLSd was more accurate than GEK except in specified low-sample cases, and for y2 at d = 80 with n ≤ 50, construction took under 45 s versus 42 min for GEK.The models use the same training and approximating points, isolating the effect of reducing hyperparameters through PLS.Source: §4.1 · p. 15
  • GE-KPLSm with m = 1–5 beat kriging on error and time; for y1 at d = 100 and n = 70, it took 7–9 s versus about 27,000 s for GEK.Adding d extra points could make matrices ill-conditioned, especially with many sampling points.Source: §4.1 · pp. 15–16

4.2 Numerical results for the analytical functions: second study

  • GE-KPLS always offered the best time–error trade-off on the analytic functions, while GE-KPLS5 had the lowest error in almost all cases.For y1 at d = 100, GE-KPLS5 with 100 samples reached 1.94% error versus 2.96% for kriging with 1,000 samples; GE-KPLS3 took 12.4 s versus kriging’s 109.6 s.Source: Figure 3 · p. 17, §4.2 · p. 17

4.3 Numerical results for the engineering functions: third study

  • Across eight engineering problems spanning 2–15 dimensions, GE-KPLS was more accurate except on P2, and every result had lower time, lower error, or both than kriging at the same sampling cost.For P5, GE-KPLS5 took 0.12 s with relative error 0.3165, versus 38.9 s and 0.4050 for kriging.Source: §4.3 · p. 18, Figure 4 · p. 19
  • For engineering applications, the authors suggest starting with GE-KPLS1 and adding points to balance time and error; iterative optimization favors fewer extras, while global accuracy may favor more.They also propose selecting approximating points using a threshold on PLS coefficients.Source: §4.3 · pp. 18, 20

5 Limitation of the GE-KPLS method

The method has a problem-dependent FOTA step size that can affect final results, and the current toolbox is limited by memory during correlation-matrix inversion.

5 Limitation of the GE-KPLS method

  • The FOTA step size can influence results and is sensitive to the problem type and sampling points.The authors recommend optimizing this parameter before using GE-KPLS.Source: §5 · p. 20
  • The current toolbox cannot handle problems with both many dimensions and many sampling points because correlation-matrix inversion requires substantial memory.The paper gives an approximate limit of n = 30550d^-0.427, fitted to the tendency between n and d.Source: §5 · p. 20

6 Conclusions

GE-KPLS uses PLS information to reduce hyperparameters and select relevant approximating points for the correlation matrix. Across analytic and engineering comparisons, it was often more accurate or faster than competing models, with results depending in part on a problem-dependent step parameter.

6 Conclusions

  • GE-KPLS applies PLS at each sampling point to select relevant approximating points and averages computed PLS information to reduce hyperparameters.The authors state this avoids dramatically increasing correlation-matrix size while using gradient information.Source: §6 · p. 20
  • The comparisons covered analytic functions and engineering problems with varied numbers of dimensions and sampling points, including comparisons with indirect GEK, kriging, KPLS, and KPLSK.The first study compared GE-KPLS with m = 1, . . . , 5 and m = d extra points against indirect gradient-enhanced kriging and ordinary kriging.Source: §6 · p. 20
  • In some cases, GE-KPLS was over 3 times more accurate and over 3200 times faster than indirect GEK; it was more accurate on 7 engineering problems, with relative-error improvement up to 9%.For the first analytic function, its accuracy was an order of magnitude smaller than kriging and KPLS.Source: §6 · p. 21
  • GE-KPLS can vary the number of approximating points to avoid ill-conditioned matrices, but effective use requires prior analysis of its problem-dependent step parameter.The paper notes that users can progressively reduce approximating points during iterative sampling; test functions and models are available for reproduction.Source: §6 · p. 21

A Definition of the engineering cases

The appendix introduces analytical expressions for the engineering cases.

A Definition of the engineering cases

A.1 P1, P2 and P3

The engineering cases include a welded-beam problem, a robot-arm position function, an aircraft-wing weight estimate, and borehole water flow.

A.1 P1, P2 and P3

  • The welded-beam problem has three responses: deflection, bending stress, and shear stress.Source: §A.1 · p. 21
  • The borehole case characterizes water flow through a borehole drilled through two aquifers.Its output is the water flow rate in m3/yr.Source: §A.1 · p. 22
  • The robot-arm function represents the position of a robot arm.Source: §A.1 · p. 22
  • The light-aircraft case estimates wing weight.Source: §A.1 · p. 22

A.5 P7 and P8

The P7 and P8 discussion identifies weight and lowest natural frequency as the quantities of interest in a torsion-vibration problem. A table lists ranges for the input variables.

A.5 P7 and P8

  • The quantities of interest are weight and the lowest natural frequency of a torsion-vibration problem.Source: §A.5 · p. 23
  • The input-variable table pairs variable names with their ranges.Source: §A.5 · p. 23
  • The listed ranges include d1 from 1.8 to 2.2 and D2 from 12.6 to 15.4.Source: §A.5 · p. 23

B Results of the analytical cases

Table 3 reports mean error and computational time for kriging, KPLS, KPLSK, and GE-KPLSm variants across m = 1 to 5, based on 10 trials.

B Results of the analytical cases

  • Table 3 reports mean error values for kriging, KPLS, KPLSK, and GE-KPLSm for m = 1 to 5.Source: Table 3 · p. 24
  • The table also reports mean computational times for the listed model variants and m values.Source: Table 3 · p. 24
  • The reported means are based on 10 trials, with best values highlighted in bold blue.Source: Table 3 · p. 24

C Results of the engineering cases

Table 4 reports mean error and computational time for kriging, KPLS, KPLSK, and GE-KPLSm variants across m = 1 to 5, based on 10 trials.

C Results of the engineering cases

  • Table 4 reports mean error values for kriging, KPLS, KPLSK, and GE-KPLSm for m = 1 to 5.Source: Table 4 · p. 25
  • The table also reports mean computational times for the listed model variants and m values.Source: Table 4 · p. 25
  • The reported means are based on 10 trials, with best values highlighted in bold blue.Source: Table 4 · p. 25
Loading 1708.02663v1…