PaperlayerView exact revision on arXiv

Source-linked AI summary

Data-driven polynomial chaos expansion for machine learning regression

E. Torre, S. Marelli, P. Embrechts, B. Sudret

arXiv:1808.03216v2stat.MLcs.LGstat.CO

TL;DR

The paper addresses challenges in applying polynomial chaos expansion to data-driven machine-learning regression. It presents PCE as an effective regression model, with reliable predictions and response-statistics estimates, including when relatively few observations are available.

  • Problem

    Applying PCE in data-driven machine-learning settings involves major challenges beyond its classical uncertainty-quantification use.

  • Method

    The paper uses polynomial chaos expansion as a regression model for purely data-driven problems, with truncation treated as a critical design step.

  • Results

    PCE delivers reliable predictions for multivariate inputs and reliable estimates of response statistics, while requiring relatively few training observations.

  • Takeaways & Limitations

    PCE offers accurate pointwise output predictions in addition to response-statistics estimates, providing advantages over established machine-learning regression techniques.

  • Takeaways & Limitations

    PCE-based regression has difficulty handling large datasets or data with very many inputs.

Abstract

from arXiv · show

We present a regression technique for data-driven problems based on polynomial chaos expansion (PCE). PCE is a popular technique in the field of uncertainty quantification (UQ), where it is typically used to replace a runnable but expensive computational model subject to random inputs with an inexpensive-to-evaluate polynomial function. The metamodel obtained enables a reliable estimation of the statistics of the output, provided that a suitable probabilistic model of the input is available. Machine learning (ML) regression is a research field that focuses on providing purely data-driven input-output maps, with the focus on pointwise prediction accuracy. We show that a PCE metamodel purely trained on data can yield pointwise predictions whose accuracy is comparable to that of other ML regression models, such as neural networks and support vector machines. The comparisons are performed on benchmark datasets available from the literature. The methodology also enables the quantification of the output uncertainties, and is robust to noise. Furthermore, it enjoys additional desirable properties, such as good performance for small training sets and simplicity of construction, with only little parameter tuning required.

1 Introduction

The paper adapts PCE into a purely data-driven ML regression method, addressing uncertain inputs, unknown input distributions, noisy observations, and limited data. It reports accurate predictions, response-statistics estimation, noise robustness, good small-sample performance, and comparable results to NN and SVM methods with little tuning.

  • Motivation and approach: PCE provides inexpensive polynomial response evaluations and enables estimation of output moments, PDFs, and sensitivity measures.Replacing the original model with a polynomial metamodel makes statistical analysis computationally efficient.
  • Motivation and approach: Data-driven PCE is proposed as an ML regression algorithm for predicting system responses from input-output observations without an available computational model.The approach re-establishes PCE in a purely data-driven setting rather than using it only as a UQ surrogate.
  • Motivation and approach: The method infers the input joint distribution from data, using non-parametric marginal inference because parametric families may be unsupported by limited evidence.Input-density knowledge is a central requirement of classical PCE, but here it is inferred rather than assumed known.
  • Methodological scope: The proposed framework is designed to handle noisy data and input dependencies, with vine copulas used to model dependence among inputs.Noise robustness is presented as a previously uninvestigated feature for PCE in this setting.
  • Reported outcomes: Experiments report reliable pointwise predictions, response-statistics estimation, performance with small training sets, and comparable performance to NN and SVM methods.The construction also requires little fine tuning of critical hyperparameters.

2 Methods

The methods section distinguishes ML’s pointwise prediction objective from UQ’s statistical-output objective and describes the accuracy measures used for both. It emphasizes that PCE’s statistical convergence does not automatically establish equally rapid pointwise convergence, especially when the input distribution is inferred from data.

  • Accuracy objectives: ML evaluates predictions by their distance from actual responses across validation inputs, commonly using mean absolute error.The validation error averages pointwise discrepancies over a separate validation dataset.
  • Accuracy objectives: PCE evaluates output-statistics estimates such as means, standard deviations, and PDFs using relative errors and Kullback-Leibler divergence.The Kullback-Leibler divergence measures average differences between logarithms of predicted and true PDFs.
  • Evaluation procedure: Simulation experiments assess PCE estimates against reference statistics obtained by Monte Carlo simulation with 10^7 samples.The real-data experiments had too few observations to establish sufficiently accurate reference values or quantify statistical errors.
  • Convergence scope: With a suitable input joint PDF, PCE is expected to converge rapidly in mean square as training size increases, but pointwise convergence need not be equally rapid.Convergence can worsen when the input distribution is unknown and must be inferred from data.

2.2 PCE in data-driven settings

In data-driven PCE, the unknown input-output relationship is represented by a truncated orthonormal polynomial expansion whose coefficients are learned from observations. The orthonormal basis compresses the representation, supports statistical interpretation, and can reduce overfitting while retaining competitive regression accuracy with little tuning.

  • Data-driven formulation: The data-driven PCE model builds an inexpensive analytical approximation Y_PC = M_PC(X) of an unknown deterministic input-output map.Its coefficients are estimated from observed input-output pairs rather than evaluations of an available computational model.
  • Data-driven formulation: PCE represents the response with a truncated orthonormal multivariate polynomial basis and estimates its expansion coefficients from training observations.The truncation converts the infinite spectral representation into a finite regression model.
  • Basis properties: Orthonormality yields a rapidly decaying coefficient spectrum and a compressed representation, reducing regression parameters and helping avoid overfitting on finite datasets.The basis superiority over a non-orthogonal basis is demonstrated on simulated data.
  • Reported contribution: The paper’s primary claim is that purely data-trained PCE can compete with state-of-the-art ML regression models while requiring little parameter tuning.The authors identify additional advantages beyond predictive accuracy.

2.3 PCE for independent inputs

For independent inputs, the method constructs marginally orthonormal polynomial bases, infers marginals non-parametrically when needed, truncates the expansion, and estimates coefficients by regression. The resulting PCE enables pointwise and statistical response characterization, while truncation and regression choices control complexity and overfitting.

  • Orthogonalisation: For statistically independent inputs, the multivariate PCE basis is formed as a tensor product of univariate polynomials orthonormal to each input marginal.This reduces multivariate basis construction to constructing separate univariate bases.
  • Marginal specification: The study estimates unknown input marginals with kernel density estimation and directly constructs orthonormal polynomials using Stieltjes or Gram-Schmidt orthogonalisation.The direct approach avoids relying on predefined parametric distribution families.
  • Marginal specification: Transforming inputs to classical distributions is generally discouraged because the resulting nonlinear transformation can make polynomial approximation more difficult.The paper therefore applies PCE to the original inputs after estimating their marginals.
  • Truncation schemes: Truncation reduces the infinite expansion to a finite set of terms, but excessive removal causes underfitting whereas retaining too many terms causes overfitting.The paper combines hyperbolic and maximum-interaction truncation and tunes complexity by cross-validation.
  • Coefficient estimation: Sparse regression with least angle regression selects PCE coefficients using an L1-regularized objective that favors sparse solutions without explicitly optimizing the regularization parameter.Ordinary least squares is unavailable when regressors outnumber observations and can overfit when the basis is large.
  • Output characterization: The fitted PCE provides pointwise predictions and statistical characterization, including moments, PDFs, and sensitivity indices derived from coefficients or Monte Carlo sampling.Orthonormality gives direct access to the mean and variance, while higher-order statistics can be estimated numerically.

2.4 PCE for mutually dependent inputs

For dependent inputs, the paper compares aPCEonX, which ignores dependencies during basis construction, with lPCEonZ, which models dependencies through a copula and Rosenblatt transform. Although lPCEonZ provides orthonormality and can improve statistical estimates, aPCEonX typically gives better pointwise predictions because it avoids the nonlinear transformation.

  • Computational considerations: Direct orthonormalization remains possible but becomes increasingly computationally demanding with more inputs or higher expansion order because it requires higher-dimensional numerical integrals.This motivates the two alternative strategies investigated in the paper.
  • aPCEonX: aPCEonX builds a polynomial basis from marginal distributions while ignoring input dependencies, so the basis is not generally orthonormal with respect to the joint input density.Its pointwise approximation can nevertheless remain accurate.
  • Statistical estimation: Copula-based resampling improves output-statistics estimates compared with resampling from the product of marginal distributions.This provides an a-posteriori way to account for input dependencies when using aPCEonX.
  • lPCEonZ: lPCEonZ models dependencies with copulas, transforms inputs into independent variables Z, and builds the PCE between Z and Y.With standard-uniform marginals, the resulting basis uses Legendre polynomials.
  • Comparison: The transformed approach produces mutually orthonormal polynomials and retains spectral properties, apart from inference, truncation, and parameter-estimation errors.This advantage follows from capturing dependencies before constructing the basis.
  • Comparison: aPCEonX typically yields more accurate pointwise predictions, although lPCEonZ is not always worse for statistical estimates.The paper attributes this pattern to avoiding the typically strongly nonlinear Rosenblatt transform.

3 Validation on synthetic data

Synthetic experiments evaluate aPCEonX and lPCEonZ on the dependent-input Ishigami function and a 23-bar horizontal truss, measuring pointwise and statistical accuracy under varying sample sizes and noise. Across these tests, aPCEonX generally provides stronger pointwise accuracy, while PCE estimates remain effective with few observations and noisy outputs.

  • Experimental design: The validation uses two simulated datasets: a highly nonlinear three-input Ishigami function and a 23-bar horizontal truss with 10 dependent uncertain inputs.Both input-dependence structures are modeled through C-vine copulas inferred from data.
  • Input modeling: Using true copulas and marginals improves statistical estimates, whereas a Gaussian copula gives generally similar or slightly worse accuracy.The true distributions are unavailable in practical applications, limiting the direct usefulness of that comparison.
  • Ishigami function: aPCEonX achieves substantially lower rMAE than lPCEonZ on the Ishigami function, consistent with avoiding the strongly nonlinear Rosenblatt transformation.Both methods are evaluated across increasing training-set sizes using validation data.
  • Ishigami function: rMAE and Kullback-Leibler divergence fall below 1% with n′ = 100 training points for the Ishigami experiment.The study also evaluates errors in the mean, standard deviation, and full response PDF.
  • Noise robustness: For noisy Ishigami outputs with σε = 0.15 = 1.22σY, rMAE drops to 10^-2 with 100 or more training points.The errors remain smaller than the added noise and decrease as sample size increases.
  • Noise robustness: Under strong noise, PCE recovers the response standard deviation and PDF with high precision, and its standard-deviation predictor outperforms sample estimates.The output mean is only mildly affected because the added noise is unbiased.
  • Horizontal truss: For the horizontal truss, both PCEs perform well, aPCEonX has significantly lower pointwise error, and lPCEonZ estimates the standard deviation better than aPCEonX.In this case, empirical estimates from the training data are most accurate for the standard deviation.

4 Results on real data sets

On benchmark real-world datasets, data-driven PCE provides competitive pointwise regression while also supporting output-distribution estimation. Across the CCPP, housing, and wine examples, it shows low error, reduced partition sensitivity, and limited tuning requirements.

  • Evaluation: The real-data evaluation uses cross-validation against machine-learning benchmarks, with randomized k-fold procedures averaging errors across validation partitions.The study also estimates response PDFs a posteriori by resampling dependent inputs and evaluating the PCE metamodel.
  • Method: Data-driven PCE models multivariate inputs from inferred marginal distributions and expresses the scalar output as a polynomial of those inputs.The basis uses mutually orthonormal polynomials with respect to the inferred marginal PDFs.
  • Uncertainty estimation: PCE-based response PDFs are estimated for the CCPP energy output and wine ratings, extending the evaluation beyond pointwise prediction.For the CCPP, reliable energy-PDF estimates are linked to energy production planning and management.
  • CCPP results: Below 1%, the average relative error of PCE predictions on the CCPP data is reported relative to observed energy outputs.The aPCEonX average MAE is slightly smaller than the BREP regression-tree result, while the average-to-minimum error difference is significantly lower.
  • Housing results: For housing values, aPCEonX has errors comparable to the neural-network model and a smaller relative error than the instance-based learner.The passage also reports that aPCEonX does not require the fine parameter tuning affecting most neural-network models.
  • Benchmark comparison: For the other benchmark, aPCEonX has MAE comparable to the SVM regressor and always lower than the best neural network.The SVM achieved the lowest MAE but at considerably higher computational cost.

5 Discussion and conclusions

The paper establishes PCE as a data-driven regression approach that provides accurate predictions and output statistics across varied tasks, including noisy and data-scarce settings. Its main boundaries are computational cost for large datasets or many inputs and degraded convergence when the input distribution must be inferred.

  • Performance and robustness: PCE achieved low pointwise and statistical errors, even with relatively few training observations and strong response noise.The simulations also showed that PCE can be used for denoising.
  • Performance and robustness: On real datasets, PCE matched or slightly exceeded neural networks and support vector machines while requiring little parameter tuning.The comparisons used automatically calibrated PCE metamodels and previously reported ML models.
  • Statistical estimation: PCE combines pointwise prediction with output-statistics estimation by pairing its metamodel with inferred marginals and copulas and resampling the input model.The approach was tested on simulated problems with up to 10 inputs and relatively small sample sets.
  • Practical advantages: The polynomial representation makes PCE interpretable, portable to embedded devices, robust to noise, and effective with relatively few data points.For independent inputs, polynomial coefficients also directly encode output moments and Sobol sensitivity indices.
  • Limitations: PCE becomes difficult for very large datasets or many inputs because fitting parameters and, when needed, inferring copulas substantially increase computational cost.The attainable expansion order is also limited by available data, while unknown input distributions generally worsen convergence.

A Mutually dependent inputs modelled through copulas

The paper models dependent inputs by transforming variables to a standard space and representing their dependence with copulas, including flexible regular vine copulas. This dependence modelling supports more accurate output-statistics prediction, while pointwise error may still behave differently for highly nonlinear functions.

  • PCE can use transformations to map dependent inputs into a space suitable for polynomial modelling.The transformation may be based on independent components or an available mapping T.
  • Regular vine copulas provide a flexible dependence model whose transformation can be computed numerically.The paper connects this flexibility to improved accuracy when predicting output statistics.
  • Greater dependence-model complexity does not necessarily reduce pointwise error, especially when T is highly nonlinear.

A.1 Copulas and Sklar’s theorem

Copulas separate a multivariate distribution into univariate marginals and a dependence function. Sklar’s theorem formalizes this representation and, with continuous marginals, makes the copula unique.

  • A d-copula is a d-variate joint CDF on [0, 1]^d with uniform marginals.
  • Sklar’s theorem expresses any multivariate CDF through separate marginal CDFs and a copula.Conversely, any copula combined with univariate CDFs defines a multivariate CDF with those marginals.
  • With continuous marginals, the copula is unique on [0, 1]^d.
  • The probability integral transform maps each input Xi to Ui = Fi(Xi), after which the copula captures the dependence among the transformed variables.The copula can be inferred from the resulting pseudo-observations.
  • The independence copula represents mutual independence, while the study uses implemented parametric copula families for inference.

A.2 Vine copulas

Vine copulas factorize a multivariate copula density into conditional pair-copula densities organized by a sequence of linked trees. Regular vines provide the general structure, with C-vines and D-vines as special cases.

  • High-dimensional dependence modelling is challenging because simple copula families may inadequately represent pairwise and higher-order dependencies.
  • Vine copulas represent a joint copula density as a product of pair-copula densities, with the conditioning structure determining the factorization.
  • An R-vine uses d − 1 trees, where each successive tree treats the preceding tree’s edges as nodes and enforces a proximity condition.
  • Each vine edge associates two variables with a conditional pair copula whose conditioning set is determined by the vine structure.
  • C-vines and D-vines are special R-vines with distinct graphical organizations of conditioning variables.Figure 14 illustrates both structures in five dimensions.

A.3 Vine copula inference in practice

Practical vine inference selects a structure, chooses pair-copula families, and fits their parameters to transformed data. Because exhaustive structure and family search is infeasible in moderate dimensions, the paper uses heuristic structure selection and sequential family fitting.

  • Vine-copula inference consists of selecting the structure, selecting each pair-copula family, and fitting the pair-copula parameters to U.
  • Exhaustively searching all vine structures and pair-copula families is computationally infeasible even in moderate dimension.
  • The method first selects a vine structure heuristically to place the strongest dependencies in the upper trees.Kendall’s tau is used as the dependence measure.
  • For a selected structure, pair-copula families and their rotated versions are compared, then the best-fitting family is chosen using the Akaike information criterion.The study considers 62 families in total, including rotations.
  • The inference adopts the simplifying assumption that conditional pair copulas depend on conditioning variables only through conditional distribution arguments.The assumption is exact only in particular cases but is described as usually not severe.
  • After sequential inference, global likelihood maximization can optionally refine the fitted vine while retaining its structure and copula families.

A.4 Rosenblatt transform of R-vines

The Rosenblatt transform maps dependent inputs to variables with independent uniform marginals by applying conditional copula derivatives. R-vine representations make the required derivative computations tractable through recursive operations on pair copulas.

  • Transform construction: The Rosenblatt transform maps a dependent vector X to a vector Z with mutually independent components.It is expressed as a composition of the probability-integral transform and a copula-based transform.
  • Transform construction: Each transformed component is computed from a conditional copula evaluated at the corresponding uniform input and preceding components.The mapping is Zi = Ci|1,...,i−1(Ui|U1, . . . , Ui−1).
  • Transform properties: The resulting variables Zi are mutually independent and uniformly distributed on [0, 1].This independence follows after computing the required derivatives of the copula.
  • R-vine computation: Representing the copula as an R-vine enables computation of the derivatives needed for the transform.The algorithms begin with first-tree pair-copula derivatives and obtain higher-order derivatives recursively through inversion and differentiation.
  • R-vine computation: Derivatives and inverses of continuous pair copulas are generally computationally cheap to evaluate numerically when analytical solutions are unavailable.The implementation can also process n sample points in parallel.

B PCE with R-vine input models

The PCE approach first infers the joint input distribution, transforms dependent inputs into approximately independent uniform variables, and then builds a polynomial metamodel on the transformed data.

  • PCE construction: Transforming X into independent components before building the PCE is the approach denoted lPCEonZ.The metamodel is subsequently constructed from Z to Y.
  • Input modeling: The method infers the joint input CDF from marginal distributions and an input copula, with R-vine copulas compatible with this framework.The workflow first infers marginals, maps observations to U, and models the copula on U.
  • Input transformation: The Rosenblatt transform maps U to Z, whose components are approximately independent and uniformly distributed on [0, 1] when the inferred models are accurate.This transformed representation is used as the training input for the PCE metamodel.
  • PCE construction: The PCE metamodel is built on the transformed training set (Z, Y) using tensor products of univariate orthogonal-polynomial bases.The basis construction depends on the assigned distribution of each transformed variable.
  • PCE construction: Uniform transformed variables use Legendre polynomials, while variables modeled with KDE-based distributions use arbitrary orthogonal polynomials.The univariate basis is selected according to the distribution assigned to Zi.
Loading 1808.03216v2…