PaperlayerView exact revision on arXiv

Source-linked AI summary

Robust Shrinkage Estimation of High-dimensional Covariance Matrices

Yilun Chen, Ami Wiesel, Alfred O. Hero

arXiv:1009.5331v1stat.ME

TL;DR

The paper addresses covariance estimation for elliptical, potentially heavy-tailed samples in large p small n settings, where Tyler’s robust estimator is inapplicable when n < p. It regularizes Tyler’s fixed-point iterations with shrinkage, proves convergence and uniqueness, and derives a data-dependent coefficient. The resulting method achieves low estimation error, robustness to heavy tails, and strong performance in wireless-sensor intrusion detection.

  • Problem

    High-dimensional covariance estimation lacks a robust solution for non-Gaussian elliptical samples when the sample size is small relative to dimension.

  • Method

    The paper regularizes Tyler’s fixed-point covariance iterations with shrinkage and selects the coefficient using a minimum mean-squared-error plug-in approximation.

  • Results

    The iterations converge to a unique limit, while simulations and wireless-sensor experiments show low estimation error, heavy-tail robustness, and superior covariance-based detection performance.

  • Takeaways & Limitations

    The estimator provides a robust covariance-based approach for elliptical data in high-dimensional settings and activity/intrusion detection.

Abstract

from arXiv · show

We address high dimensional covariance estimation for elliptical distributed samples, which are also known as spherically invariant random vectors (SIRV) or compound-Gaussian processes. Specifically we consider shrinkage methods that are suitable for high dimensional problems with a small number of samples (large $p$ small $n$). We start from a classical robust covariance estimator [Tyler(1987)], which is distribution-free within the family of elliptical distribution but inapplicable when $n<p$. Using a shrinkage coefficient, we regularize Tyler's fixed point iterations. We prove that, for all $n$ and $p$, the proposed fixed point iterations converge to a unique limit regardless of the initial condition. Next, we propose a simple, closed-form and data dependent choice for the shrinkage coefficient, which is based on a minimum mean squared error framework. Simulations demonstrate that the proposed method achieves low estimation error and is robust to heavy-tailed samples. Finally, as a real world application we demonstrate the performance of the proposed technique in the context of activity/intrusion detection using a wireless sensor network.

I. INTRODUCTION

High-dimensional covariance estimation is difficult when sample sizes are small, because the classical sample covariance has distorted eigen-structure. Existing approaches often assume Gaussian data or require n much larger than p, motivating a robust estimator for elliptical, heavy-tailed samples.

  • Large p small n covariance problems distort the eigen-structure of the classical sample covariance, requiring improved estimators.
  • Most high-dimensional covariance estimators assume Gaussian samples, limiting their use for non-Gaussian and heavy-tailed data.
  • Ledoit-Wolf shrinkage is distribution-robust asymptotically but remains inappropriate for heavy-tailed non-Gaussian distributions because it operates on the sample covariance.
  • Traditional robust covariance estimators generally require n ≫ p and therefore do not suit large p small n problems.
  • The paper develops a shrinkage estimator for elliptical samples, establishes convergence and uniqueness, and evaluates it through simulations and wireless-sensor intrusion detection.

A. Elliptical distribution

The paper models observations with an elliptical distribution generated by a positive scale variable multiplying a Gaussian vector. This family includes several signal-processing distributions and is also called the SIRV or compound-Gaussian model.

  • An elliptical sample is generated from a positive random variable and a zero-mean Gaussian vector with positive definite covariance.
  • The scale variable and Gaussian vector are statistically independent in the model.
  • The elliptical family includes Gaussian, K, and Weibull distributions and is also known as SIRV or compound Gaussian.

B. ML estimation

Tyler’s maximum-likelihood covariance estimator normalizes samples to remove the elliptical scale variable and solves a fixed-point equation. It is robust within the elliptical family but is unsuitable when n < p.

  • The estimation problem uses n i.i.d. elliptical samples to estimate the covariance matrix Σ.
  • Trace normalization sets the covariance scale by imposing Tr(Σ) = p.
  • The sample covariance is poor for high-dimensional or heavy-tailed samples, motivating Tyler’s normalized-sample method.
  • Tyler’s maximum-likelihood estimator removes the scale variable through normalized samples and is obtained by fixed-point iterations.
  • When n ≥ p and the normalized samples are linearly independent, the iterations converge to a unique limit up to scale, followed by trace normalization.
  • The estimator has strong large-sample properties and minimizes maximum asymptotic variance within elliptical covariance estimators.

III. ROBUST SHRINKAGE COVARIANCE ESTIMATION

The paper regularizes Tyler’s fixed-point iterations with shrinkage to enable covariance estimation when n < p. For shrinkage coefficients between 0 and 1, the iterations converge to a unique limit from any positive definite initialization.

  • Shrinkage regularization extends Tyler’s method to high-dimensional elliptical covariance estimation, where the unregularized equation has no solution when n < p.
  • The shrinkage term keeps each iterate well-conditioned, while trace normalization is necessary for convergence.
  • For 0 < ρ < 1, the fixed-point iterations converge to a unique limit for every positive definite initial matrix.
  • Unlike heuristic diagonal loading, the proposed shrinkage approach supplies a systematic procedure for selecting ρ.

A. Choosing the shrinkage coefficient

The paper selects the shrinkage coefficient by minimizing mean-squared error, derives a closed-form oracle coefficient, and replaces its unknown covariance dependence with a plug-in estimate.

  • The shrinkage coefficient ρ is chosen to minimize the minimum mean-squared error rather than by computationally intensive cross-validation.
  • Theorem 2 establishes a closed-form solution to the oracle coefficient problem for i.i.d. elliptical samples under Tr(Σ) = p.
  • The oracle coefficient cannot be implemented directly because it depends on the unknown true covariance Σ.
  • A plug-in estimate replaces the oracle coefficient, using a consistent covariance estimator such as the trace-normalized Ledoit-Wolf estimator.The plug-in estimate is then used in the regularized fixed-point iterations.
  • The plug-in covariance estimator need only approximate Tr(Σ^2), rather than be well-conditioned or accurately estimate the full covariance matrix.

IV. NUMERICAL SIMULATION

Simulations evaluate the proposed estimator on heavy-tailed multivariate Student-T and Gaussian samples in high-dimensional settings. The proposed method preserves covariance structure under n < p, improves over ML when n > p, and sacrifices little performance for Gaussian data.

  • Multivariate Student-T samples: The proposed estimator outperforms other estimators on heavy-tailed multivariate Student-T samples.The Student-T simulation uses p = 100, while sample sizes range from 5 to 225 in repeated trials.
  • Multivariate Student-T samples: At p = 100 and n = 50, the proposed covariance estimate preserves the true covariance structure, whereas Ledoit-Wolf produces heavy-tail-induced block artifacts.
  • Multivariate Student-T samples: When n > p, the proposed method substantially improves over the ML covariance estimate.The comparison is presented as evidence of the value of Steinian shrinkage for reducing MSE.
  • Gaussian samples: For Gaussian samples, the proposed robust method performs even better than Ledoit-Wolf when n < 20.The Gaussian comparison uses the same simulation parameters as the preceding example except for the sampling distribution.
  • Gaussian samples: For Gaussian data, the proposed method sacrifices very little performance despite being developed for the entire elliptical family.

V. APPLICATION TO ANOMALY DETECTION IN WIRELESS

The wireless-sensor experiment evaluates robust covariance estimation for unsupervised anomaly detection on 182 RSSI measurements, whose detrended data are strongly heavy-tailed and non-Gaussian. With 200 training samples, the proposed estimator achieves the best detection performance and the narrowest 90% confidence envelope among the compared methods.

  • Preprocessing: The detection pipeline detrends each RSSI signal by subtracting a local mean computed over a window of 50 samples.The resulting detrended samples are used for anomaly detection.
  • Data characteristics: Statistical tests and visualizations establish that the detrended RSS measurements are heavy-tailed and non-Gaussian.A Student-T fit estimates degree-of-freedom d = 2 with 95% CI [1.8460, 2.2879].
  • Detection method: The anomaly statistic declares a time sample anomalous when it exceeds a specified threshold, using the proposed estimator for the unknown covariance matrix.The 182×182 sample covariance is constructed from 200 randomly subsampled RSS time slices, with training anomalies occurring at approximately 10%.
  • Detection results: The mean ROC ranking is Proposed > Ledoit-Wolf > Tyler’s ML > Sample covariance.The proposed method compensates simultaneously for heavy tails and small sample size, whereas the other methods compensate for only one of these effects or perform poorly in this setting.
  • Detection results: The proposed method achieves the smallest width among the four computed 90% confidence envelopes.The envelopes are determined by cross-validation and reflect sensitivity to variations in the training set.
  • Supervised comparison: When anomalies are excluded from the supervised training period, Ledoit-Wolf-based intrusion detection performs close to the proposed method.This comparison uses a training period selected from ground truth where no anomalies appear.

VI. CONCLUSION

The paper proposes a robust shrinkage covariance estimator for elliptically distributed samples and establishes theoretical guarantees for its fixed-point construction. Its practical relevance is demonstrated through Gaussian comparisons and wireless-sensor-network intrusion detection.

  • The proposed shrinkage covariance estimator is robust over the class of elliptically distributed samples.
  • Existence, convergence, and uniqueness are established for the estimator obtained by fixed-point iterations.
  • The optimal shrinkage coefficient follows a minimum mean-squared-error framework and has a closed-form expression involving the unknown true covariance.
  • A simple plug-in estimator approximates the unknown-covariance expression for the shrinkage coefficient.
  • For Gaussian samples, the proposed estimator performs very closely to estimators designed specifically for Gaussian data.
  • In wireless-sensor-network intrusion detection, a covariance-based algorithm using the robust estimator achieved superior performance compared with conventional covariance estimators.

A. Proof of Theorem 1

The proof applies a concave Perron–Frobenius result to the fixed-point mapping over positive semidefinite matrices. This establishes convergence to a unique normalized limit and shows that the limit is positive definite.

  • A. Proof of Theorem 1: The proof uses a concave Perron–Frobenius lemma for a normalized operator whose iteration converges to a unique point.The lemma assumes a closed convex cone and suitable positivity bounds on the operator.
  • A. Proof of Theorem 1: The operator is instantiated on symmetric matrices, with the cone defined as the set of positive semidefinite matrices.
  • A. Proof of Theorem 1: The nuclear norm used for normalization is Tr(Σ)/p on the cone and is increasing.
  • A. Proof of Theorem 1: The mapping satisfies the lemma's condition with e = I, a = ρ, and b = (1 −ρ)p^2 + ρ.
  • A. Proof of Theorem 1: Because the mapping always yields a positive definite matrix, the convergent fixed-point limit is positive definite.
  • A. Proof of Theorem 1: The proof establishes the theorem by identifying the lemma's normalized limit with the limit of the proposed iterations for every Σ ≻ 0.

B. Proof of Theorem 2

The proof derives the shrinkage estimator's coefficient by rewriting the relevant quantity through an eigendecomposition and normalized Gaussian vectors. Substitution of the resulting moments yields the stated expression.

  • B. Proof of Theorem 2: The proof introduces eC to simplify the notation used in deriving the shrinkage estimator.
  • B. Proof of Theorem 2: The shrinkage estimator is expressed in terms of eC before differentiating with respect to ρ.
  • B. Proof of Theorem 2: The derivation begins by eigendecomposing the covariance matrix Σ.
  • B. Proof of Theorem 2: Normalized Gaussian vectors zi have unit norm, are mutually independent, and are isotropically distributed.
  • B. Proof of Theorem 2: The proof rewrites eC in terms of zi and substitutes the derived relations into the final moment calculation.
Loading 1009.5331v1…