PaperlayerView exact revision on arXiv

Source-linked AI summary

Large Covariance Estimation through Elliptical Factor Models

Jianqing Fan, Han Liu, Weichen Wang

arXiv:1507.08377v1stat.ME

TL;DR

Large factor-model covariance estimation under heavy-tailed data is the problem addressed. The paper develops a generic POET framework with high-level conditions and Kendall’s tau estimators for elliptical data; its reported results include robust-estimator advantages under heavy tails and convergence results under stated conditions.

  • Problem

    The paper addresses covariance estimation for heavy-tailed data in factor models, where covariance sparsity is conditional on low-dimensional factors and earlier work focused on Gaussian or sub-Gaussian factors and noises.

    Source: §1 · p. 2

  • Method

    The paper gives POET a generic interface of sufficient conditions and, for elliptical data, uses marginal Kendall’s tau for covariance and eigenvalues and multivariate Kendall’s tau for eigenvectors.

    Source: §1.2 · p. 4

  • Results

    Under heavy-tailed t4.2 and t7 distributions, robust POET performs better than original POET, with error roughly reduced by a magnitude of two for t7; under normal or t∞ data, original POET performs better and robust POET is comparable.

    Source: §5.1 · p. 21

  • Takeaways & Limitations

    The framework provides sufficient conditions for POET convergence and allows pilot covariance, eigenvalue, and eigenvector estimators to be chosen for the data distribution.

    Source: §1.2 · p. 4, §6 · pp. 21, 23

    The elliptical model considered is continuous with P(ζ = 0) = 0; the paper assumes q = p, while stating its results also hold when q < p if q and p are of the same order.

    Source: §4.1 · p. 14

Abstract

from arXiv · show

We proposed a general Principal Orthogonal complEment Thresholding (POET) framework for large-scale covariance matrix estimation based on an approximate factor model. A set of high level sufficient conditions for the procedure to achieve optimal rates of convergence under different matrix norms were brought up to better understand how POET works. Such a framework allows us to recover the results for sub-Gaussian in a more transparent way that only depends on the concentration properties of the sample covariance matrix. As a new theoretical contribution, for the first time, such a framework allows us to exploit conditional sparsity covariance structure for the heavy-tailed data. In particular, for the elliptical data, we proposed a robust estimator based on marginal and multivariate Kendall's tau to satisfy these conditions. In addition, conditional graphical model was also studied under the same framework. The technical tools developed in this paper are of general interest to high dimensional principal component analysis. Thorough numerical results were also provided to back up the developed theory.

1 Introduction

The paper extends factor-model covariance estimation to heavy-tailed data with conditional sparsity and develops a general POET framework with conditions for its convergence. For elliptical data, it uses robust Kendall’s tau estimators for different parts of the covariance structure.

1 Introduction

  • The paper targets heavy-tailed factor-model data whose idiosyncratic-error covariance is sparse conditional on low-dimensional spiked factors.Earlier work considered Gaussian or sub-Gaussian factors and noises; this paper considers elliptical distributions.Source: §1 · pp. 2–3

1.1 Background on approximate factor model

  • The approximate factor model represents observed variables through common factors, variable-specific loadings, and idiosyncratic errors; the paper also presents its vector and matrix forms.The stated setup includes p > n, independent and identically distributed samples, and observable factors.Source: §1 · p. 2, §1.1 · p. 3

1.2 Major contributions of this paper

  • The generic POET framework uses high-level sufficient conditions to obtain convergence rates and separates deterministic analysis from the probabilistic guarantees for its pilot estimators.The conditions involve componentwise maxima.Source: §1.2 · p. 4
  • For elliptical data, marginal Kendall’s tau estimates the covariance matrix and leading eigenvalues, while multivariate Kendall’s tau estimates leading eigenvectors.The framework permits eigenvalues and eigenvectors to be estimated by different methods.Source: §1.2 · pp. 4–5

1.3 Notations

  • The notation section defines matrix and vector norms, eigenvalue notation, and the O_P and o_P conventions used in the paper.It also uses C for a generic constant that may vary between lines.Source: §1.3 · p. 5

1.4 Paper organization

  • The paper proceeds from a generic POET procedure and its theoretical interface to sub-Gaussian and elliptical results, simulations, and discussion.Technical proofs are placed in the appendix.Source: §1.4 · p. 5

2 A High-level theoretical interface

The section presents a generic POET interface for covariance estimation under a spiked factor model, then extends the framework to conditional graphical models and positive-semidefinite projection.

2.1 Spiked covariance model

  • The spiked model assumes a fixed number of leading eigenvalues proportional to p, with the remaining eigenvalues bounded; pervasive factors can produce this structure.Under the stated loading conditions, the leading eigenvalues satisfy λ_j = pλ_j(Σ_b)(1 + o(1)).Source: §2.1 · p. 6

2.2 A generic POET procedure for covariance estimation

  • Generic POET subtracts the estimated leading low-rank component, thresholds the residual, and adds the low-rank component back; its pilot estimates need not come from one sample covariance matrix.Adaptive thresholding is applied to the residual covariance, with entry-dependent thresholds corresponding to thresholding its correlation matrix.Source: §2.2 · pp. 7–8

2.3 A high level theoretical interface

  • Under the stated assumptions, pilot estimates of the covariance matrix and leading eigenstructure let generic POET attain the rates in (2.6) and (2.7).The assumptions include bounded factor loadings and idiosyncratic covariance eigenvalues, along with the pilot-estimator conditions.Source: §2.3 · pp. 8–9

2.4 Conditional graphical model

  • For conditional graphical models, POET first removes the low-rank factor component, then applies CLIME to estimate the idiosyncratic inverse covariance, whose sparsity encodes conditional relationships.The paper notes that bounded ∥Ωu∥∞ is a stronger assumption here than for covariance estimation and might be relaxed with another method.Source: §2.4 · pp. 9–10
  • With the stated pilot-estimator and boundedness conditions, the theorem gives convergence rates for the idiosyncratic and original inverse covariance estimators.The paper notes that the bounded-∥Ωu∥∞ condition could potentially be relaxed using another method instead of CLIME.Source: §2.4 · p. 10

2.5 Positive semi-definite projection under max norm

  • When the estimated residual covariance is not positive semidefinite, projection to the PSD cone can preserve the max-norm error bound required by generic POET.The paper states that choosing τ ≍ w_n yields a PSD solution satisfying the max-norm bound.Source: §2.5 · p. 11

3 Sub-Gaussian factor models

The section verifies that sample-covariance pilot estimates satisfy the POET interface for sub-Gaussian factor models. Under the stated conditions, this yields the desired POET rates and approximate recovery of the leading eigenstructure.

3 Sub-Gaussian factor models

  • The sub-Gaussian analysis assumes isotropic random vectors satisfying an additional condition that is slightly stronger than the classical sub-Gaussian condition.The paper notes that this condition holds for independent sub-Gaussian elements and may also hold for weakly dependent sub-Gaussian vectors.Source: §3 · p. 12
  • For j ≤ m, ∥ξ̂_jA − e_jA∥ = O_P(n^-1/2), and the theorems establish approximate recovery of the true leading eigenvalues and eigenvectors under pervasive spikes.The paper says this result does not require element-wise independence, only sub-Gaussian isotropic random vectors satisfying (3.1).Source: §3 · p. 13
  • Sample-covariance-based estimates of Σ, Λ, and Γ satisfy the sufficient conditions, explaining why POET achieves its desired rates for sub-Gaussian data.The argument verifies the conditions using the sample covariance, its leading eigenvalues, and its leading eigenvectors.Source: §3 · p. 13

4 Elliptical factor models

The elliptical extension replaces the sub-Gaussian assumption with elliptical factors and noises, using separate robust estimators to supply POET’s required covariance, eigenvalue, and eigenvector inputs. Together, these estimators support the framework’s covariance and precision matrix convergence results.

4 Elliptical factor models

  • The section replaces the sub-Gaussian assumption with an elliptical distribution assumption and proposes a robust factor-model estimator.Source: §4 · p. 13

4.1 Elliptical distribution

  • The elliptical model uses a scale mixture of a sphere-uniform vector, with Eζ² = q for identifiability; the paper assumes continuous distributions and bounded fourth moments.Source: §4.1 · pp. 14–15

4.2 Robust estimation of variances

  • The variance estimator applies M-estimation to squared observations; its stated guarantee requires n ≥ C log d.Source: §4.2 · p. 16

4.3 Marginal Kendall’s tau estimator

  • Marginal Kendall’s tau estimates the correlation matrix, while its resulting covariance pilot has suitable eigenvalue properties; its nonlinear sine transformation makes the eigenvector condition difficult to prove.Source: §4.3 · p. 18

4.4 Multivariate Kendall’s tau estimator

  • The multivariate Kendall’s tau kernel is distribution-free, and its diagonal population matrix preserves the covariance matrix’s eigenspaces for estimating leading eigenvectors.Source: §4.4 · p. 19
  • The third sufficient condition in (1.6) holds for Γ̂ED; with Σ̂1 and Λ̂ED from Section 4.3, it enables POET for the heavy-tail factor model to achieve all desired covariance and precision-matrix estimation convergence rates.Under Assumptions 2.1 and 4.1, Theorem 4.1 gives ∥ξ̂jA − ejA∥ = O_P(n^-1/2) for j ≤ m, where ejA is a unit vector of length m.Source: §4.4 · pp. 19–20

5 Simulations

The simulations compare robust and original POET estimators for conditional sparse covariance and graphical-model estimation under t-distributed factors and noises. Robust POET performs better in heavy-tailed settings and remains competitive in the normal setting, while original POET performs better there for covariance estimation.

5 Simulations

  • The simulations compare robust and original POET pilots within the general framework for conditional sparse covariance and conditional graphical-model estimation.They vary dimension and tail heaviness, using multivariate t distributions and a normal case.Source: §5 · p. 20, §5.1 · pp. 20–21, §5.2 · p. 21

5.1 Conditional sparse covariance estimation

  • For t4.2, original POET performs poorly while robust POET works well; for t7, rank-based estimation roughly reduces error by a magnitude of two, while original POET performs better at ν = ∞.Robust POET remains comparable to original POET in the normal setting.Source: §5.1 · p. 21

5.2 Conditional graphical model estimation

  • For t4.2 and t7, robust POET outperforms non-robust estimators for both idiosyncratic and full precision matrices, and remains competitive for the normal case.The comparison uses spectral-norm errors across 100 simulations per configuration.Source: §5.2 · p. 21

6 Discussions

The discussion presents POET as a generic procedure under sufficient conditions on its pilot covariance estimator, with transparent convergence results. It also describes robust eigen-structure estimation for elliptical factor models and notes open questions about lower bounds and model assumptions.

6 Discussions

  • POET accepts any pilot covariance estimator satisfying the stated sufficient conditions, providing a generic framework for high-dimensional factor models under pervasiveness.The discussion characterizes the resulting theoretical analysis as transparent.Source: §6 · p. 21
  • Figure 1 compares original and robust POET average errors across norms and three t-distribution degrees of freedom, using p from 100 to 1000 and n = p/2.The caption reports 100 simulations per p and m = 3.Source: Figure 1 · p. 22
  • Figure 2 compares spectral-norm average errors for original and robust POET estimates of Ωu and Ω across three t-distribution degrees of freedom.The caption reports p from 50 to 500, n = 0.6p, m = 3, and 100 simulations per p.Source: Figure 2 · p. 23
  • For heavy-tailed elliptical factor models, robust estimation of leading eigenvalues and eigenvectors is presented as an example, with possible extension to richer distribution families.The discussion contrasts this with using sample covariance for sub-Gaussian observations.Source: §6 · p. 23
  • Lower bounds for the stated convergence rates are deferred, and bounded eigenvalues alone are described as insufficient for consistent estimation.The discussion says further structural assumptions are needed and identifies comparisons between optimization and pervasiveness approaches as future work.Source: §6 · p. 23

A Proofs in Section 2

These proofs establish covariance and inverse-covariance error rates by controlling the sparse residual covariance and the estimated low-rank factor component. They then use these bounds to justify CLIME estimation of the residual precision matrix.

A Proofs in Section 2

  • The proof targets max-norm error O_P(w_n) for the estimated sparse covariance, which suffices for the stated covariance convergence rates.It decomposes the error into low-rank and residual components and chooses the threshold τ at order w_n.Source: A Proofs in Section 2 · pp. 23–25
  • The low-rank component is controlled by bounding eigenvalue and eigenvector errors, including a max-norm eigenvector bound of order O(1/√p).The proof uses Weyl’s inequality and the Davis–Kahan sin θ theorem in deriving these bounds.Source: A Proofs in Section 2 · p. 24
  • The proof applies the Sherman–Morrison–Woodbury formula to bound inverse-covariance error, then establishes CLIME residual precision error from covariance max-norm control.It obtains spectral-norm error O_P(M_p w_n^(1−q)) for both residual and full inverse covariance estimates.Source: A Proofs in Section 2 · pp. 26–27

B Proofs in Section 3

The proofs in Section 3 establish concentration rates for empirical eigenvalues and eigenvectors under the sub-Gaussian factor-model setting. They separate the spiked factor component from the remaining eigenstructure.

B Proofs in Section 3

  • For the spiked eigenvalues, the proof establishes relative error O_P(n^−1/2) by analyzing the fixed-dimensional factor component and bounding the non-spiked contribution.The latter contribution is shown to be o_P(n^−1/2) relative to a spiked eigenvalue.Source: B Proofs in Section 3 · pp. 27–28
  • For the leading eigenvectors, the factor-coordinate component, after normalization, differs from its coordinate basis vector by O_P(n^−1/2).The proof treats the one-factor and multiple-factor cases.Source: B Proofs in Section 3 · pp. 28, 30
  • The non-spiked eigenvector component has norm O_P(n^−1/2 + p^−1/2), as summarized in Lemma B.1.Source: B Proofs in Section 3 · p. 32

C Proofs in Section 4

The Section 4 proofs establish eigenvalue and eigenvector bounds for the Kendall-based estimator under the stated assumptions. They use pairwise normalized sample differences and rescaled data in the analysis.

C Proofs in Section 4

  • The proof forms a Kendall matrix from normalized pairwise sample differences and studies its eigenstructure through a sample-size-dimensional matrix with the same nonzero eigenvalues.It analyzes the spiked and non-spiked portions separately.Source: C Proofs in Section 4 · pp. 34, 36
  • For the spiked eigenvalues, the proof obtains O_P(n^−1/2) error, while the non-spiked eigenvalues are O_P(n^−1).These bounds are derived for the permutation-based Kendall matrices in Lemma C.1.Source: C Proofs in Section 4 · p. 37
  • The proof bounds the leading eigenvector’s factor-coordinate component at O_P(n^−1/2) from its coordinate basis vector.Lemma C.2 supplies the corresponding eigenvector result under Assumptions 2.1 and 4.1.Source: C Proofs in Section 4 · pp. 33, 38

D A Technical Lemma

This appendix develops concentration bounds for quadratic forms of sub-Gaussian vectors, including a lower-tail bound under condition (3.1). The bounds support a technical lemma for use in the proofs of Sections 3 and 4.

D A Technical Lemma

  • Lemma D.1 assumes iid sub-Gaussian row vectors with p ≥ n and condition (3.1), and its bound is intended for proofs in Sections 3 and 4.The proof uses a 1/4-net covering of the unit sphere in bounding a term.Source: D A Technical Lemma · pp. 38–39
  • For Lemma D.1, the proof bounds two terms separately, using a quadratic-form bound for the first and decoupling for the second.The proof states that each term is bounded by choosing C0 sufficiently large.Source: D A Technical Lemma · pp. 40–41
  • Under condition (3.1), Lemma D.2 bounds the tail probability of a quadratic functional of an isotropic sub-Gaussian vector by 2 exp(−t).The paper says this supplies the lower-tail bound not provided by Hsu et al. (2012) under their weaker assumption.Source: D A Technical Lemma · pp. 41–42

Related papers

Loading 1507.08377v1…