PaperlayerView exact revision on arXiv

Source-linked AI summary

The Nonparanormal: Semiparametric Estimation of High Dimensional Undirected Graphs

Han Liu, John Lafferty, Larry Wasserman

arXiv:0903.0649v1stat.ML

TL;DR

High-dimensional sparse graph estimation often relies on exact normality, motivating a more flexible alternative. The paper uses a semiparametric Gaussian copula with transformed marginals, develops a truncated-empirical estimator, and establishes theoretical consistency results for high-dimensional inference.

  • Problem

    The paper addresses high-dimensional sparse graph estimation when reliance on exact normality is limiting.

  • Method

    It models transformed variables as multivariate Gaussian and estimates the nonparanormal using a truncated empirical marginal distribution with graphical-lasso machinery.

  • Results

    The estimator is analyzed through consistency results for risk, model selection, and precision-matrix estimation in high-dimensional settings.

  • Takeaways & Limitations

    The nonparanormal provides a nonparametric extension of Gaussian graphical models while retaining precision-matrix encoding of independence relations.

Abstract

from arXiv · show

Recent methods for estimating sparse undirected graphs for real-valued data in high dimensional problems rely heavily on the assumption of normality. We show how to use a semiparametric Gaussian copula--or "nonparanormal"--for high dimensional inference. Just as additive models extend linear models by replacing linear functions with a set of one-dimensional smooth functions, the nonparanormal extends the normal by transforming the variables by smooth functions. We derive a method for estimating the nonparanormal, study the method's theoretical properties, and show that it works well in many examples.

I. Introduction

The paper extends sparse and nonparametric ideas from regression to high-dimensional undirected graphical models by replacing Gaussian variables with transformed variables in a semiparametric Gaussian copula. It proposes the nonparanormal and develops an estimator with theoretical analysis.

  • I. Introduction: The nonparanormal combines nonparametric marginal transformations with high-dimensional sparse graphical-model estimation.It parallels sparse additive models while regularizing the inverse covariance matrix.
  • I. Introduction: Gaussian graphical models encode conditional independences through zeros in the precision matrix, but high-dimensional estimation requires exploiting graph sparsity.Regularized likelihood and graphical-lasso methods address the p > n setting.
  • I. Introduction: The paper replaces exact normality with a semiparametric Gaussian copula whose transformed variables are jointly Gaussian.The resulting distribution is called the nonparanormal and retains a precision-matrix representation of independence relations.
  • I. Introduction: The proposed estimator uses a truncated empirical distribution because the usual empirical marginal distribution is inaccurate when dimension increases.The paper analyzes risk, model selection, and Frobenius-norm estimation of the precision matrix.
  • I. Introduction: Experiments use simulated data and gene microarray data to construct an isoprenoid biosynthetic pathway.The paper presents both theoretical analysis and empirical evaluation.

II. Estimating Undirected Graphs

This section frames graph estimation through covariance and precision-matrix criteria, then explains why high dimensions require regularization and how estimator quality is assessed. It distinguishes risk consistency from consistency under a correctly specified Gaussian model.

  • II. Estimating Undirected Graphs: For Gaussian distributions, missing graph edges correspond to zero entries of the precision matrix Ω = Σ^-1.This converts conditional-independence graph estimation into inverse-covariance estimation.
  • II. Estimating Undirected Graphs: When p > n, the sample covariance is not positive definite, so maximum-likelihood graph estimation is no longer useful.Its rank is at most n, motivating regularized estimation.
  • II. Estimating Undirected Graphs: The graphical lasso estimates the precision matrix by regularized maximum likelihood with an ℓ1 constraint and can be computed efficiently by block coordinate descent.Under appropriate sparsity conditions, the estimator has established theoretical properties.
  • II. Estimating Undirected Graphs: Persistency evaluates prediction-risk consistency without assuming the true distribution is Gaussian or belongs to the candidate covariance class.The dimension p may increase with sample size.
  • II. Estimating Undirected Graphs: Norm consistency and sparsistency instead require a correctly specified Gaussian model, with sparsistency meaning consistent recovery of the graph.The paper uses these criteria alongside persistency to evaluate covariance and inverse-covariance estimators.

III. The Nonparanormal

The nonparanormal models transformed variables as multivariate Gaussian, yielding a Gaussian-copula family with flexible monotone marginals. Its conditional-independence graph remains encoded by the inverse covariance, while monotonicity supports tractable density evaluation.

  • III. The Nonparanormal: A nonparanormal distribution has componentwise functions f_j such that f(X) is multivariate Gaussian with mean μ and covariance Σ.The model is semiparametric because only one-dimensional functions are estimated nonparametrically.
  • III. The Nonparanormal: When the functions f_j are monotone and differentiable, the nonparanormal is a Gaussian copula.Its joint distribution can be written using the Gaussian multivariate cdf and marginal distribution functions.
  • III. The Nonparanormal: Conditional independence in the nonparanormal is equivalent to zero precision-matrix entries: X_i ⫫ X_j | O\{i,j} if and only if Ω_ij = 0.Thus the model preserves the Gaussian graphical-model interpretation despite nonparametric marginals.
  • III. The Nonparanormal: The graph does not require estimating the marginal means or scales μ and σ.The zero pattern is unchanged under the relevant diagonal rescaling.
  • III. The Nonparanormal: For the illustrated two-dimensional examples, the exponent α_j controls nonlinearity, and α > 1 can produce multiple modes.The examples use component functions of the form f_j(x) = sign(x)|x|^αj.
  • III. The Nonparanormal: Monotonicity enables computational tractability, whereas more general transformation functions prevent closed-form computation of the density normalizing constant.The paper therefore restricts the transformations to a tractable class.

IV. Estimation Method

The estimator builds a transformed sample from estimated marginal distributions, using a truncated empirical distribution to control high-dimensional variance. It then estimates the precision matrix and graph with regularization, achieving the desired convergence rate under the stated choice of truncation.

  • A rank approximate Z-estimator uses estimated marginal distributions to estimate the copula parameters, which here correspond to the covariance matrix.The marginal empirical distribution is a natural candidate, but high-dimensional variance motivates truncation.
  • The truncated estimator clips marginal empirical probabilities below δ_n and above 1 − δ_n, with δ_n controlling the truncation.Choosing δ_n involves a bias-variance tradeoff.
  • The truncation choice provides the balance needed to obtain the desired convergence rate for the precision-matrix estimate and its associated graph.
  • The estimated transformation functions are applied to the observations to form a transformed sample covariance matrix.The procedure estimates each marginal transformation before constructing the covariance matrix.
  • The precision matrix is estimated from the transformed covariance matrix using either inversion or an ℓ1-regularized estimator.The regularized estimator penalizes off-diagonal precision-matrix entries, and nonzero entries define the estimated graph.

V. Theoretical Results

The paper develops theoretical guarantees for a Winsorized estimator of the nonparanormal, covering risk, model selection, and covariance or precision-matrix estimation as dimension grows with sample size.

  • The theoretical analysis targets risk consistency, model selection consistency, and covariance and inverse-covariance norm consistency.
  • The key technical result analyzes the covariance of the Winsorized estimator under appropriate conditions.
  • The paper establishes estimation consistency in both the Frobenius norm and the ℓ2-operator norm.
  • The theoretical results use assumptions involving the Fisher information matrix, support set, bounded matrix quantities, and maximum row degree.
  • The estimator's analysis yields risk consistency even when the model dimension grows exponentially with sample size.

VI. Experimental Results

Experiments compare ℓ1-regularized nonparanormal and Gaussian models using glasso on synthetic and real data. The reported conclusions depend on whether the data are Gaussian, nonparanormal, or otherwise misspecified.

  • The experiments compare ℓ1-regularized nonparanormal and Gaussian models, both computed with the graphical lasso algorithm.
  • For multivariate Gaussian data, the two methods have comparable performance.
  • When the nonparanormal model is correct, it performs much better than graphical lasso in many cases.
  • The nonparanormal can be fit by passing its Winsorized sample covariance to the existing glasso routine.

A. Neighborhood graphs

The neighborhood-graph experiments generate sparse 40-node graphs under cdf, power, and no transformations, then compare glasso with the nonparanormal across sample sizes.

  • The simulations generate a sparse graph and restrict its maximum degree to four when constructing the inverse covariance matrix.
  • The cdf and power transformations map a univariate normal distribution into highly skewed and bimodal distributions, respectively.
  • The study evaluates cdf, power, and no-transformation conditions, applying both glasso and the nonparanormal to estimate the graph.
  • The graph uses 40 nodes and 820 parameters to be estimated, with sample sizes ranging from n = 200 to n = 1000.
  • For cdf and power transformations, the nonparanormal separates relevant and irrelevant dimensions well, whereas glasso mixes relevant variables with irrelevant ones.
  • At n = 1000, the estimated transformations show that Winsorization plays a significant role for the power transformation.

A.3 Quantitative comparison

The nonparanormal generally improves graph estimation under non-Gaussian transformations while matching the glasso under exact Gaussianity. Simulations and microarray analyses compare error rates, ROC behavior, regularization paths, and selected graphs.

  • Regularization paths: For cdf and power transformations, nonparanormal regularization paths separate relevant from irrelevant dimensions more clearly than glasso paths.Without transformation, the paths for both methods are almost the same.
  • Simulation comparisons: The nonparanormal is superior to the Gaussian model in most ROC-curve comparisons, with both methods improving as sample size increases.The ROC curves use n = 1000, 500, and 200.
  • Simulation comparisons: The nonparanormal achieves significantly smaller errors than the glasso for non-Gaussian data and comparable performance when the data are exactly multivariate Gaussian.The comparison reports average false-positive and false-negative errors over 100 repetitions.
  • Microarray analysis: The microarray study analyzes associations among 40 Arabidopsis thaliana genes using data with sample size n = 118.The genes are drawn from the isoprenoid biosynthetic pathway.
  • Regularization paths: The two methods select variables in different orders, especially for λ ∈[0.2, 0.3], producing different model-selection behavior.The paths were evaluated on 50 regularization parameters over [0.16, 1.2].
  • Microarray analysis: On the microarray data, the closest glasso graphs differ from nonparanormal graphs in edges selected by either method, and several differences involve nonlinear transformations.Figure 10 compares graphs over λ in [0.16, 0.37], while Figure 11 shows the corresponding transformations.

VII. Proofs

The proof section begins by normalizing the component means and variances, then defining population and empirical transformation functions through Gaussian quantiles.

  • Normalization: The component means and variances are set to µ_j = 0 and σ_j = 1 without loss of generality.This reduction is attributed to Lemma 3.3.
  • Transformation definitions: The population transformation f_j and empirical transformation ef_j are defined by applying the inverse standard-Gaussian distribution function to F_j and eF_j.The inverse function g_j is also introduced.
  • Transformation definitions: The proof distinguishes the population and empirical component transformations through these definitions.The notation establishes the objects used in later error analysis.

A. Proof of Theorem 5.1

The proof of Theorem 5.1 controls errors in the estimated transformations by combining Gaussian tail bounds, empirical-process inequalities, and event-based decompositions. It concludes that tail observations do not affect the rate.

  • Auxiliary bounds: The proof uses Gaussian distribution-versus-quantile bounds, transformed-variable distribution bounds, and a Gaussian maximal inequality.These appear as Lemmas 7.1–7.3 and support control of transformation-estimation errors.
  • Error decomposition: The analysis is restricted to a high-probability event A_n and decomposes the error into terms that are analyzed separately.The proof introduces A_n before establishing subsequent bounds.
  • Conclusion: The tail data do not affect the rate under the analysis used in the proof.This is stated explicitly in Remark 7.8.
  • Conclusion: Theorem 5.1 follows by combining the bounds established in Lemmas 7.7 and 7.9.The proof explicitly identifies these two lemmas as sufficient for the theorem’s conclusion.

B. Proof of Theorem 5.7

The proof of Theorem 5.7 applies empirical-process bounds to a class of products of monotone component functions. These bounds support the paper’s consistency analysis for the nonparanormal estimator.

  • Empirical-process bound: The proof invokes a uniform empirical-process bound involving the envelope, population and sample risks, and a bracketing integral.The bound is stated for a function class F.
  • Entropy control: For bounded monotone one-dimensional functions, the bracketing entropy satisfies a logarithmic bound.The cited entropy result supplies the complexity control needed by the empirical-process argument.
  • Function-class construction: The proof applies the entropy calculation to products m(x) = f_j(x_j)f_k(x_k) across component pairs.The class includes j, k ∈ {1, . . . , p} and component functions in M_n.
  • Connection to estimation: The paper connects this theoretical analysis to consistency results for the nonparanormal estimator.The conclusion describes the estimator as based on thresholding empirical-distribution tails and leveraging covariance-error bounds.
Loading 0903.0649v1…