Source-linked AI summary
Active subspace methods in theory and practice: applications to kriging surfaces
Paul G. Constantine, Eric Dow, Qiqi Wang
TL;DR
Coordinate-based sensitivity methods can miss functions whose strongest variation lies along rotated input directions. The paper detects those directions from gradient samples, constructs kriging approximations on the resulting active subspace, and reports better same-cost approximation than kriging in the full 100-dimensional space, while noting dependence on eigenvalue decay and sampling accuracy.
Problem
Sensitivity methods that rank natural coordinates perform poorly when important variation is not aligned with those coordinates.
Method
Gradient evaluations define a rotation separating strong and weak variability, after which the function is projected onto a low-dimensional active subspace and approximated there.
Results
For the same cost, the active subspace method produces a more accurate approximation than a response surface in m = 100 dimensions.
Takeaways & Limitations
Active subspace methods are appropriate for functions that vary primarily in low-dimensional subspaces and can outperform full-space response surfaces in the studied elliptic PDE application.
Takeaways & Limitations
The method performs poorly when the eigenvalues of C do not decay, and accurate active directions may require sufficient gradient samples.
Abstract
from arXiv · showhide
Many multivariate functions in engineering models vary primarily along a few directions in the space of input parameters. When these directions correspond to coordinate directions, one may apply global sensitivity measures to determine the most influential parameters. However, these methods perform poorly when the directions of variability are not aligned with the natural coordinates of the input space. We present a method to first detect the directions of the strongest variability using evaluations of the gradient and subsequently exploit these directions to construct a response surface on a low-dimensional subspace---i.e., the active subspace---of the inputs. We develop a theoretical framework with error bounds, and we link the theoretical quantities to the parameters of a kriging response surface on the active subspace. We apply the method to an elliptic PDE model with coefficients parameterized by 100 Gaussian random variables and compare it with a local sensitivity analysis method for dimension reduction.
1. Introduction & motivation.
Coordinate-based sensitivity methods rank input variables, but can miss dominant variation along rotated directions. The paper proposes detecting those directions with gradients and building response surfaces on the resulting active subspace.
- Motivation: Local and global sensitivity methods rank coordinate directions, which can fail when a function varies primarily along rotated directions.For f(x1, x2) = exp(0.7x1 + 0.3x2), both methods identify x1 as more important despite the strongest direction being [0.7, 0.3].
- Motivation: The example function is effectively univariate after rotating coordinates to align one coordinate with its strongest variation.It is flat along the orthogonal direction [−0.3, 0.7].
- Method: The proposed method evaluates gradients to rotate the input space into directions of relative variability and flatness.The resulting active subspace contains the low-dimensional directions used for approximation.
- Contributions: The paper develops error bounds for conditional-expectation, Monte Carlo, and response-surface approximations on the reduced coordinates.It also examines the effects of slightly perturbed active directions.
- Contributions: Active-subspace approximation extends subspace-based ideas from optimization and model reduction to high-dimensional scientific and engineering functions.The paper connects theoretical analysis with kriging response surfaces on the reduced domain.
2. Active subspaces and z-invariance.
Active subspaces use the eigendecomposition of a gradient covariance-like matrix to separate directions of strong and weak average variation. Functions that are nearly invariant in the weak directions can then be approximated using fewer rotated coordinates, although accurate discovery depends on gradient sampling and eigenvalue decay.
- Active-subspace construction: The matrix C is the uncentered covariance of gradients, and its eigenvectors rotate inputs into ordered directions of average variation.C is symmetric positive semidefinite with eigenvalues λ1 ≥ ··· ≥ λm ≥ 0.
- Active-subspace construction: Mean-squared directional derivatives along eigenvectors equal their corresponding eigenvalues, quantifying average variation in each rotated direction.This relationship motivates selecting directions associated with the largest eigenvalues.
- Active-subspace construction: The first n rotated coordinates y define the active subspace, while the remaining coordinates z represent directions with smaller average variation.The basis is partitioned according to the decreasing eigenvalues of C.
- z-invariance: If λn+1 through λm are zero, the function is z-invariant and varies only with y.For points sharing the same active coordinates, z-invariance gives equal function values and gradients.
- Special cases: A ridge function f(x) = h(aT x) has a rank-one C, so its active direction is a/∥a∥ and can be found from one gradient evaluation.The exponential example in Figure 1.1 is a ridge function.
- Computational procedure: The high-dimensional matrix C is approximated by Monte Carlo gradient samples, equivalently using an SVD whose directions are principal components of the gradients.The sampled matrix has size m × M, and the resulting eigendecomposition or SVD supplies approximate active directions.
- Limitations: Accurate eigenvector recovery requires an adequate number and distribution of gradient samples, especially when variability is localized or the function is nonsmooth.The paper leaves finite-sample error quantification beyond its scope and notes that eigenvalue decay is relevant to dimension reduction.
- Response-surface approximation: Response surfaces can be built in the reduced y coordinates instead of the original m-dimensional space when the function is nearly z-invariant.This avoids standard constructions that suffer from the curse of dimensionality.
3. Approximation in the active subspace.
The paper approximates functions by conditioning on active coordinates and replacing variation along inactive coordinates with conditional expectations and Monte Carlo estimates. Error bounds connect neglected eigenvalues, Monte Carlo sample count, response-surface error, and perturbed subspaces.
- The conditional expectation G(y) is the best mean-squared approximation of f given active coordinates y.
- If the trailing eigenvalues are sufficiently small, a Monte Carlo estimate using a small N, including N = 1, can approximate f accurately.
- The response surface is trained on reduced-domain data {(yk, ˆGk)} rather than directly on the original variables x.
- Theorem 3.3 combines conditional-expectation, Monte Carlo, and response-surface errors under an explicit response-surface error assumption.
- Perturbed eigenvectors introduce additional error contributions involving the eigenvalue structure and perturbation size ε.
4. Heuristics for kriging surfaces.
The kriging heuristic uses gradient-derived active directions to choose a reduced design, calibrate correlation lengths and noise, and construct a response surface on the reduced domain. Practical choices include reduced dimension, domain-specific designs, and noninterpolating regression.
- The reduced dimension n is chosen using eigenvalue decay together with practical requirements for constructing a manageable kriging design.
- For Gaussian inputs, the reduced domain is Rn and a tensor-product design can cover three standard deviations in each active coordinate.
- For uniform hypercube inputs, the reduced domain is generally a convex polytope, and computing its marginal density is challenging.
- The kriging surface uses training values ˆGk obtained from function evaluations at points corresponding to reduced design sites yk.
- Because Monte Carlo estimates ˆG contain noise, the method favors smoothing regression surfaces over exact interpolation.
- Active-subspace eigenvalues inform kriging correlation lengths, with ℓi inversely proportional to the square root of λi.
- The kriging noise parameter is motivated by variance bounds and can be estimated alongside a single hyperparameter α by maximum likelihood.
- The algorithm uses an SVD of sampled gradients, partitions the resulting directions, selects reduced-domain sites, and then applies kriging.
5. Numerical example.
The elliptic PDE experiment tests gradient-based active subspaces for a 100-parameter Gaussian coefficient field under long and short correlation lengths. Despite slower KL singular-value decay for short correlations, active-subspace singular values decay rapidly, and low-dimensional kriging outperforms coordinate reduction and full-space kriging at comparable cost.
- Model and setup: The quantity of interest is computed from a linear elliptic PDE whose coefficient field is parameterized by 100 independent standard-normal variables.The model uses a truncated KL expansion and studies correlation lengths β = 1 and β = 0.01.
- Model and setup: Adjoint-based finite elements provide f and ∇_xf at M = 300 Gaussian input samples for active-subspace construction.The PDE discretization and adjoint solve supply the gradients used to identify dominant directions.
- Active-subspace diagnostics: For β = 0.01, KL singular values decay slowly while active-subspace singular values decay rapidly, indicating effective dimension reduction for the scalar quantity of interest.The comparison concerns dimension-reduction behavior: KL values describe the spatial field, whereas active-subspace values describe a specific scalar output.
- Active-subspace diagnostics: The first eigenvector’s components are more evenly distributed for short correlation lengths, weakening the correspondence between important KL coefficients and output variability.For long correlation lengths, larger components cluster toward smaller KL indices; this relationship relaxes when the correlation length is short.
- Kriging comparison: Active-subspace kriging gives smaller testing errors than coordinate reduction for both one- and two-dimensional approximations.The error histograms shift left for the active-subspace method, indicating smaller relative error; the study therefore focuses on n = 1 and n = 2.
- Kriging comparison: At the same evaluation cost, active-subspace response surfaces are more accurate than a kriging surface built directly in the 100-dimensional input space.The cost-matched comparison uses 3M + P evaluations and an independent testing set of 500 points.
6. Summary & conclusions.
The paper applies active subspace methods to construct low-dimensional response surfaces and compares them with coordinate-based reductions. The approach performs well when functions vary primarily in low-dimensional subspaces, but can perform poorly without eigenvalue decay.
- Summary & conclusions: Active subspace methods construct response surfaces on low-dimensional subspaces of multivariate function domains.The procedure includes conditional-expectation, Monte Carlo, and kriging approximations.
- Summary & conclusions: The method detects influential directions and constructs a kriging surface for an elliptic PDE with random-field coefficients.The coefficients are represented through a series expansion whose coefficients are treated as random variables.
- Summary & conclusions: The active subspace method showed superior performance compared with a local sensitivity analysis approach for dimension reduction.The comparison was performed for the elliptic PDE application.
- Summary & conclusions: Active subspace methods are appropriate when functions vary primarily in low-dimensional input subspaces.This scope includes many high-dimensional applications with rapidly decaying eigenvalues.
- Summary & conclusions: Without decay in the eigenvalues of C, active subspace methods can perform poorly.The paper identifies this as a limitation of the method’s applicability.
- Summary & conclusions: The computational method remains open to improvements, especially reducing gradient evaluations and using sampled function evaluations more effectively.These are identified as directions for future extensions.