PaperlayerView exact revision on arXiv

Source-linked AI summary

Evading the curse of dimensionality in nonparametric density estimation with simplified vine copulas

Thomas Nagler, Claudia Czado

arXiv:1503.03305v3stat.ME

TL;DR

High-dimensional nonparametric density estimation suffers from the curse of dimensionality, motivating methods whose convergence does not deteriorate with dimension. This paper develops a nonparametric estimator based on simplified vine copulas and shows that its convergence speed is dimension-independent, with advantages also when the simplifying assumption is violated.

  • Problem

    Nonparametric density estimators converge more slowly as dimension increases, limiting their usefulness for multivariate applications.

  • Method

    The paper separates marginal and copula estimation and estimates the copula as sequentially estimated pair-copula densities under a simplified vine model.

  • Results

    The estimator achieves a convergence speed independent of dimension, and any d-dimensional PVCA can be consistently estimated at a rate equivalent to a two-dimensional problem.

  • Takeaways & Limitations

    The vine copula approach can outperform classical multivariate nonparametric estimators as dimension grows, including when the simplifying assumption is severely violated.

  • Takeaways & Limitations

    When the simplifying assumption is violated, the proposed estimator is not consistent, and formal tests for assessing the assumption have yet to be developed.

Abstract

from arXiv · show

Practical applications of nonparametric density estimators in more than three dimensions suffer a great deal from the well-known curse of dimensionality: convergence slows down as dimension increases. We show that one can evade the curse of dimensionality by assuming a simplified vine copula model for the dependence between variables. We formulate a general nonparametric estimator for such a model and show under high-level assumptions that the speed of convergence is independent of dimension. We further discuss a particular implementation for which we validate the high-level assumptions and establish its asymptotic normality. Simulation experiments illustrate a large gain in finite sample performance when the simplifying assumption is at least approximately true. But even when it is severely violated, the vine copula based approach proves advantageous as soon as more than a few variables are involved. Lastly, we give an application of the estimator to a classification problem from astrophysics.

1. Introduction

High-dimensional nonparametric density estimation is hindered by convergence rates that deteriorate with dimension. The paper proposes simplified vine densities as a flexible subclass whose estimation rate can be independent of dimension.

  • Motivation: The curse of dimensionality slows nonparametric density estimation as dimension increases, requiring excessively more observations for comparable accuracy.The cited example contrasts n = 50 observations in one dimension with more then n = 10^6 in ten dimensions.
  • Motivation: For p-times continuously differentiable d-dimensional densities, the convergence rate is bounded by n^-p/(2p+d), so higher dimensions necessarily reduce the optimal rate.The dependence on d appears in the denominator of the exponent.
  • Proposed direction: Simplified vine densities provide a richer alternative to independence by decomposing dependence into bivariate building blocks.The paper positions this class as flexible and useful even when the simplifying assumption is severely violated.
  • Contribution: The paper introduces a nonparametric estimator that separates marginal and copula densities, with the copula estimated through pair-copula components.The joint density is reconstructed from separately estimated marginal and dependence terms.
  • Proposed direction: Under the simplifying assumption, a d-dimensional copula density decomposes into d(d −1)/2 unconditional bivariate densities.This reduces estimation of the copula to multiple two-dimensional estimation tasks.

2. Simplified vine copulas and distributions

Vine copulas organize multivariate dependence through linked trees and bivariate conditional copulas. The simplifying assumption removes conditioning-variable dependence, yielding simplified vine distributions built from unconditional pair-copulas.

  • Vine structure: A regular vine is a sequence of linked trees whose first tree contains the variables and whose later trees use preceding edges as nodes.The sequence must satisfy the proximity condition: connected nodes in a later tree correspond to edges sharing a node in the preceding tree.
  • Pair-copula construction: Each vine-tree edge is associated with a bivariate copula, or pair-copula, representing dependence between two conditioned variables.The model uses d(d −1)/2 bivariate conditional copulas to construct a d-dimensional copula.
  • Conditional dependence: In a general vine, each pair-copula may depend on the values of its conditioning variables, making the model more complex to estimate.The conditional copula can have different functional forms for different conditioning-variable values.
  • Simplifying assumption: The simplifying assumption ignores that dependence and makes the conditional copula equal across conditioning-variable values.The resulting conditional density collapses to an unconditional two-dimensional object.
  • Simplified vine distributions: A distribution represented through this unconditional pair-copula construction is called a simplified vine distribution.The construction can express vine copula densities using bivariate copula densities and corresponding h-functions.

3. A nonparametric density estimator based on simplifed vine copulas

The proposed estimator builds a multivariate density from separately estimated marginals and sequentially estimated pair-copulas. Conditional quantities for higher trees are obtained recursively through h-functions and pseudo-observations.

  • Estimator design: The estimator separates marginal-density estimation from copula-density estimation, then represents the copula as a product of sequentially estimated pair-copula densities.This is implemented as a general step-wise procedure without fixing the precise component estimators.
  • Marginal estimation: The procedure first estimates each marginal density and distribution function from the observed multivariate data.The estimated marginal distributions transform observations into pseudo-observations for copula estimation.
  • Sequential estimation: Pair-copula densities and h-functions for the first tree are estimated from two-dimensional pseudo-observations.The h-functions can be obtained from the estimated pair-copula densities by integration.
  • Practical caveat: Estimating h-functions separately is optional but can produce a density estimate that does not integrate to one.The issue arises because separately estimated components need not preserve the normalization implied by the integrated pair-copula construction.
  • Sequential estimation: For subsequent trees, estimated h-functions transform pseudo-observations into inputs for conditional pair-copula estimation.The same transformation-and-estimation pattern is repeated for higher trees.
  • Output: At completion, the algorithm provides marginal distributions and densities, pair-copula densities, and h-functions needed to evaluate the simplified vine density.The resulting estimate is assembled from all components required by the vine representation.

4. Asymptotic theory

The paper establishes consistency and dimension-independent convergence for the simplified vine density estimator under high-level assumptions, then derives its asymptotic normality. The rate matches that of a two-dimensional estimator, provided the simplifying structure and regularity conditions hold.

  • Consistency and convergence: The proposed estimator is weakly consistent, with a probabilistic convergence rate that does not increase with dimension.This removes the curse of dimensionality under the simplified vine model.
  • Consistency and convergence: The analysis assumes consistent marginal, pair-copula, and h-function estimators, with h-function errors vanishing faster than n^-r.The assumptions also require controlled sensitivity to perturbed pseudo-observations and smooth pair-copula densities.
  • Consistency and convergence: Under Assumptions A1–A4, the estimator inherits the convergence rate of the bivariate copula density estimator, independent of d.The simplifying assumption permits decomposition of the multivariate estimation problem into lower-dimensional components.
  • Consistency and convergence: For p-times continuously differentiable pair-copula densities, the convergence exponent is r = p/(2p + 2), matching a classical two-dimensional estimator.A general d-dimensional estimator has exponent p/(2p + d), whereas the vine rate remains unchanged as dimension grows.
  • Scope and limitations: If the target density is not a simplified vine density, the estimator is generally inconsistent for the target and converges to a partial vine copula approximation.Finite-sample performance may nevertheless remain advantageous in that setting.
  • Scope and limitations: More generally, allowing pair-copulas to depend on at most d′ conditioning variables yields rate exponent p/(2p + 2 + d′).The simplifying assumption is necessary for the dimension-free exponent p/(2p + 2) to be feasible.
  • Asymptotic normality: The estimator’s asymptotic normality follows from joint normality of its components and an application of the multivariate delta method.The stacked vector contains marginal densities, pair-copula densities, and conditional distribution-function components.

5. On an implementation as kernel estimator

The paper instantiates the general estimator with kernel methods for marginal densities, pair-copula densities, and h-functions. It verifies the required assumptions, derives consistency rates, and establishes asymptotic normality for this implementation.

  • Implementation: The implementation uses kernel estimators for marginal densities, pair-copula densities, and h-functions within the general vine framework.The paper gives low-level conditions under which the high-level assumptions can be verified.
  • Marginal estimation: The marginal kernel estimator achieves strong uniform consistency under kernel, bandwidth, and marginal regularity conditions.With mean-square optimal bandwidth bn = O(n^-1/5), Proposition 2 gives rate Oa.s.(n^-2/5√ln n).
  • Pair-copula estimation: The transformation estimator maps copula data to standard normal margins, applies ordinary kernel estimation, and transforms the estimate back to uniform margins.This handles the unit-hypercube support of copula densities while using standard kernels on unbounded transformed margins.
  • Pair-copula estimation: The pair-copula estimator requires twice-continuously differentiable densities and an additional condition on the transformed density.The latter condition can be verified for many parametric families, including those used in the simulations.
  • H-function estimation: H-functions are estimated by integrating the pair-copula density estimate, linking conditional distribution estimation to a regression-like problem.The resulting estimators are closely related to smoothed Nadaraya–Watson estimators.
  • H-function estimation: Using the mean-square optimal pair-copula bandwidth bn = O(n^-1/6) may give the same n^-1/3 rate for pair-copula and h-function estimation, so slight undersmoothing can satisfy the theoretical requirement.The authors report that preliminary numerical experiments found no practical problems with the mean-square optimal rate.
  • Asymptotic normality: The implementation establishes joint asymptotic normality for the estimator’s components, and asymptotic normality of the full vine estimator follows by the delta method.The result is obtained under the stated kernel, marginal, copula, and bandwidth conditions.
  • Structure selection: Vine structure selection uses maximum or minimum spanning trees weighted by dependence measures such as empirical Kendall’s τ.Kernel pair-copula estimates make the structure-selection procedure fully nonparametric.

6. Simulations

The simulations compare vine-copula and classical kernel density estimators across dimensions, sample sizes, and dependence structures. The vine estimator performs especially well when the simplifying assumption holds and can remain advantageous at higher dimensions even when it is violated.

  • Experimental design: The experiments cover one simplified target density and two targets that violate the simplifying assumption.The margins are standard Gaussian so performance differences can be related directly to the simplified vine assumption.
  • Experimental design: The study compares bfvine with bfmvkde using median integrated absolute error across five sample sizes and dimensions d = 3, 5, 10.Each setting uses 250 simulated samples, and significance is assessed with Mood’s median test at the 1% level.
  • Scenario 1: Gaussian copula: Under the Gaussian copula, bfvine strictly outperforms bfmvkde, with statistically significant differences for every dimension and sample size.Its convergence rate remains independent of dimension, unlike the classical estimator.
  • Scenario 1: Gaussian copula: For d = 5, bfvine is almost two times as accurate, and for d = 10 it is almost three times as accurate as bfmvkde.The performance gap widens as dimension or sample size increases.
  • Scenario 2: Gumbel copula: With a non-simplified Gumbel copula, bfvine is slightly worse at d = 3 but significantly better at d = 5 and d = 10.At d = 10 and n = 5 000, it is almost two times as accurate despite being inconsistent under the violated assumption.
  • Scenario 3: Non-simplified Gaussian vine: In the severely non-simplified Gaussian-vine scenario, bfvine performs significantly worse for d = 3, 5 but significantly better for d = 10.The results suggest that the unrestricted nonparametric estimator also struggles to identify the dependence structure in this setting.

7. Application

The application revisits astrophysical gamma-versus-hadron classification with Bayes classifiers built from the vine-copula and classical kernel estimators. The vine-based classifier achieves higher true-positive rates across the relevant low-false-positive operating range.

  • Data and task: The astrophysical dataset contains 19 020 observations on 10 variables, with gamma rays as signal and hadron showers as background.There are 12 332 gamma observations and 6 688 hadron observations.
  • Classification method: Bayes classification estimates class-specific densities and varies the posterior threshold to trade off false-positive rate against true-positive rate.The false-positive rate concerns hadrons misclassified as gamma, while the true-positive rate measures correctly classified gamma events.
  • Experimental setup: The experiment trains the class-conditional density estimates on two-thirds of the data and evaluates predictions on the remaining third.The vine and classical estimators use equal class priors, and sparsity is induced through an independence test in structure selection.
  • Results: The ROC curve for bfvine lies above the classical estimator’s curve almost everywhere, indicating higher signal identification at a fixed false-positive rate.The application focuses particularly on false-positive rates of 0.01, 0.02, 0.05, 0.1, and 0.2.
  • Results: The bfvine classifier achieves loacc = 0.472 and highacc = 0.849 in comparison with 14 previously evaluated classification algorithms.These measures average true-positive rates over the low- and higher-false-positive operating ranges, respectively.

8. Further discussion

The simplified vine copula approach divides high-dimensional density estimation into lower-dimensional tasks, achieving dimension-independent convergence under its simplifying assumption. When that assumption fails, the estimator targets a partial vine copula approximation whose practical accuracy and interpretation have important limitations.

  • 8. Further discussion: The estimator divides a d-dimensional density estimation problem into several one- and two-dimensional tasks, yielding convergence independent of dimension.This advantage becomes more pronounced as the number of variables increases.
  • 8. Further discussion: When the assumption is violated, the estimator is not consistent, although it can still outperform a direct density estimator in most practicable situations.This advantage may disappear under extreme violations in low dimensions.
  • 8. Further discussion: Open questions concern how far simplified densities can be from arbitrary densities and how to interpret estimated components when simplification fails.The paper notes that these questions remain open in the developing vine-copula literature.
  • 8. Further discussion: A partial vine copula approximation can be consistently estimated at a rate equivalent to a two-dimensional problem.
  • 8. Further discussion: When the simplifying assumption holds, the partial vine copula approximation coincides with the true density.
  • 8. Further discussion: The partial vine copula approximation can be closer to the true density than a consistent estimate of the density itself on finite samples.The approach uses model shrinkage by restricting the set of possible solutions when the simplifying assumption is approximately true.

A. Proof of Theorem 1

The proof of Theorem 1 proceeds inductively from pseudo-observation control to pair-copula and conditional-distribution consistency, then combines these results to establish consistency of the vine estimator.

  • A. Proof of Theorem 1: The first proof step shows by induction that pseudo-observations converge sufficiently fast to the true observations.
  • A. Proof of Theorem 1: The second step establishes pointwise consistency of feasible pair-copula density and conditional distribution function estimators.
  • A. Proof of Theorem 1: The final step combines the preceding convergence results with regularity conditions and continuous differentiability to establish consistency of the vine density estimator.

B. Lemmas

The lemmas define oracle and feasible kernel-based pair-copula and h-function estimators, with feasible versions obtained by replacing latent quantities with pseudo-observations.

  • B. Lemmas: The oracle transformation pair-copula density estimator is introduced as the basis for the theoretical analysis.
  • B. Lemmas: The corresponding oracle h-function estimator is obtained by integrating the pair-copula density.
  • B. Lemmas: Feasible pair-copula and h-function estimators replace the oracle transformation variables with estimated pseudo-observations.

B.2. Results

The lemmas establish uniform approximation results for the kernel-based pair-copula and related estimators under the stated regularity conditions, supporting the consistency argument.

  • B.2. Results: Under conditions K1, K2, C1, and C2, the feasible pair-copula density estimator satisfies a uniform almost-sure approximation to the target density.
  • B.2. Results: A first-order Taylor expansion controls the effect of perturbing the estimator arguments through pseudo-observations.
  • B.2. Results: The proof combines the resulting bounds with prior lemmas to conclude the required consistency result.
  • B.2. Results: Additional lemmas establish analogous uniform bounds on the relevant interior domain under conditions K1, K2, C1, and C2.

C. Proof of Proposition 5

The proof establishes componentwise stochastic expansions for the estimated marginal and conditional copula quantities, then verifies the covariance and moment conditions needed for asymptotic normality. The limiting mean and covariance simplify because the first d components vanish, while dependence across distinct edges becomes asymptotically negligible.

  • The estimated marginal densities satisfy a uniform stochastic expansion around the true densities with a bias term of order b2n,c and a stochastic remainder.
  • The first d components of the limiting mean are zero, and the corresponding rows and columns of the covariance matrix vanish.
  • The estimated conditional copula densities inherit analogous expansions around the true conditional copula densities.
  • Covariances between contributions from distinct edges vanish asymptotically, yielding a diagonal limiting covariance matrix.
  • A stronger Lyapunov-type condition is verified instead of checking the remaining condition directly, supporting the multivariate central limit theorem.
Loading 1503.03305v3…