PaperlayerView exact revision on arXiv

Source-linked AI summary

Multivariate outlier detection based on a robust Mahalanobis distance with shrinkage estimators

Elisa Cabana, Rosa E. Lillo, Henry Laniado

arXiv:1904.02596v1stat.ME

TL;DR

Multivariate outliers can distort statistical procedures, while robust distances face challenges in cutoff selection and computation. This paper proposes shrinkage-based robust Mahalanobis distances using robust location and covariance estimators. Across the reported comparisons, the selected RMD-Shrinkage performs well in most cases, including skewed and heavy-tailed settings, and is reported to have inexpensive computation.

  • Problem

    Classical estimates can be distorted by outliers, while robust-distance cutoffs lack a universally applicable method and may fit poorly away from normality.

    Source: §1 · pp. 2–3

  • Method

    The paper combines robust location and comedian-based covariance estimators with shrinkage to construct robust Mahalanobis distances.

    Source: §1 · pp. 3–4, §2 · p. 4

  • Results

    RMD-Shrinkage shows the best results in the vast majority of cases, including with skewed and heavy-tailed distributions.

    Source: §3.4 · p. 13

  • Takeaways & Limitations

    The conclusions report approximately affine-equivariant behavior, high breakdown value under highly contaminated data, and inexpensive computation, with a real-data example bearing out these conclusions.

    Source: §5 · p. 22

    The MAD-to-standard-deviation relation used in the dispersion construction is stated under a normality assumption.

    Source: §2.2 · p. 7

Abstract

from arXiv · show

A collection of robust Mahalanobis distances for multivariate outlier detection is proposed, based on the notion of shrinkage. Robust intensity and scaling factors are optimally estimated to define the shrinkage. Some properties are investigated, such as affine equivariance and breakdown value. The performance of the proposal is illustrated through the comparison to other techniques from the literature, in a simulation study and with a real dataset. The behavior when the underlying distribution is heavy-tailed or skewed, shows the appropriateness of the method when we deviate from the common assumption of normality. The resulting high correct detection rates and low false detection rates in the vast majority of cases, as well as the significantly smaller computation time shows the advantages of our proposal.

1 Introduction

Multivariate outliers can distort statistical procedures, while univariate rules miss observations that are unusual only in combination across variables. Robust Mahalanobis distances address sensitivity in classical estimates, but threshold choice and computation remain challenges.

1 Introduction

  • Classical location and covariance estimates let outliers mask or swamp other observations, motivating robust estimators for Mahalanobis distance.Fast-MCD can also become computationally expensive as dimension grows because its candidate solutions increase exponentially.Source: §1 · p. 2
  • No universally applicable cutoff is established for robust distances, and the usual chi-squared threshold can fit poorly away from normality.The mismatch is documented particularly as the number of variables increases.Source: §1 · pp. 2–3
  • The paper proposes six robust Mahalanobis distances for high-dimensional outlier detection, combining robust location and comedian-based covariance estimators with shrinkage.Shrinkage is applied to component-wise and L1-median location estimates and to special comedian matrices.Source: §1 · pp. 3–4
  • The proposal is evaluated against MCD, Adj MCD, Kurtosis, OGK, and COM in simulations, including skewed or heavy-tailed data, and with a real dataset.The study also examines affine equivariance, breakdown value, and computational cost.Source: §1 · p. 4

2 A robust Mahalanobis distance based on shrinkage estimator

The section constructs robust Mahalanobis distances by replacing classical location and covariance estimates with robust alternatives. Shrinkage trades bias against variance, while yielding a well-conditioned positive-definite covariance estimate.

2 A robust Mahalanobis distance based on shrinkage estimator

  • Robust Mahalanobis distance replaces classical location and covariance estimates, which can be strongly influenced by outliers.It measures observations relative to robust estimates of centrality and covariance.Source: §2 · p. 4
  • Shrinkage combines an estimator with a target to reduce estimation error under general conditions, balancing bias and variance.For covariance matrices, shrinkage also yields a positive-definite, well-conditioned estimate.Source: §2 · pp. 4–5

2.1 Location parameter

  • The location options are the component-wise median and the multivariate L1-median, both used as robust centrality estimators.The component-wise median takes the median separately for each variable; the L1-median is described as robust and highly efficient.Source: §2.1 · p. 5
  • The L1-median is a second location option, and its shrinkage estimator likewise uses a scaled-vector target with intensity and scale chosen by expected quadratic loss.The passage presents the solution as a proposition and describes the corresponding numerator estimation.Source: §2.1 · pp. 5–7
  • For component-wise-median shrinkage, the estimator is shrunk toward a scaled vector of ones, with intensity and scale chosen to minimize expected quadratic loss.The intensity numerator is estimated using a normal-theory variance result for the sample median.Source: §2.1 · pp. 5–6

2.2 Dispersion parameter

  • The dispersion construction uses the MAD as a robust scale measure and invokes its relation to standard deviation under normality.The relation is attributed to Falk [1997].Source: §2.2 · p. 7

X and MAD2(X):

The comedian generalizes the MAD and supplies the robust dependence structure used for dispersion estimates. Shrinkage addresses the comedian matrix’s lack of guaranteed positive definiteness, and six location-dispersion combinations define the proposed distances.

X and MAD2(X):

  • The comedian generalizes the MAD, always exists, and has properties including symmetry, location invariance, and scale equivariance.Falk [1997] established consistency and asymptotic normality results for the comedian, paralleling results for the MAD.Source: X and MAD2(X) · pp. 7–8
  • The comedian matrix is formed from pairwise comedian estimates, but may fail to be positive semidefinite, preventing reliable inversion in Mahalanobis distance.Shrinkage is proposed to ensure a positive-definite, well-conditioned matrix.Source: X and MAD2(X) · p. 8
  • Covariance shrinkage uses a scaled identity target, with scale and shrinkage intensity selected to minimize expected quadratic loss.The estimator numerator is adapted from Ledoit and Wolf [2003a], Ledoit and Wolf [2003b], and Ledoit and Wolf [2004] using the robust comedian estimate.Source: X and MAD2(X) · pp. 8–9

2.3 Proposed Robust Mahalanobis Distances

  • The paper defines six robust Mahalanobis distances by pairing location and dispersion estimators, with comedian matrices centered using corresponding robust locations.The proposed outlier threshold for every combination is the chi-squared p;0.975 quantile.Source: X and MAD2(X) · pp. 9–10, §2.3 · p. 10

3 Simulation results

Across normal, heavy-tailed, skewed, and exponential simulations, the proposed distances generally perform well, with RMDv6 selected as the best overall performer. RMD-Shrinkage also performs well under correlation and affine transformations, tolerates substantial contamination, and has competitive computational times.

3.1 Normal distribution

  • For normal data, RMDv6 performed best overall, especially at high dimension; proposed methods achieved 100% correct detection for all tested dimensions and contamination levels when outliers were far from the center.In the far-outlier scenario, false detection rates were zero in most cases and very close to zero otherwise.Source: §3.1 · p. 11, Table 2 · p. 11

3.2 t3-distribution

  • With t3 data, proposed methods attained 100% correct detection in most cases, while COM and the proposals had the best false detection rates; all methods had nonzero false detection rates.Competitors' correct detection was unsatisfactory particularly at high dimension or with larger contamination.Source: §3.2 · p. 12, Table 3 · p. 12

3.3 Exponential distribution

  • For contaminated multivariate exponential data, RMDv6 performed best in every case, achieving 100% correct detection in most cases except some low-dimensional settings; false detection was similar across methods.Source: §3.3 · p. 12

3.4 Summary and selection of one of our proposed distances

  • RMDv6 ranked among the top three by F-score and performed best in the vast majority of cases, so the authors selected it as RMD-Shrinkage.Its overall classification ranking balances correct detections against false alarms.Source: §3.4 · p. 13

3.5 Correlation and affine equivariance

  • On correlated data, RMD-Shrinkage outperformed MCD, Adj MCD, Kurtosis, and OGK, while COM was similar at 10% and 20% contamination and slightly worse at 30%.Its false detection rate was also very low in these correlated settings.Source: §3.5 · p. 14, Table 4 · p. 14
  • Under affine transformations, RMD-Shrinkage detected all outliers except a few cases with 30% contamination and outliers close to the center; those cases improved as dimension increased.Source: §3.5 · p. 14, Table 5 · p. 15

3.6 Breakdown value

  • Across dimensions p=10, 30, 50, 80, 100 and contamination levels α=10%, 20%, 30%, 40%, 45%, the method achieved 100% correct detection, with maximum false detection rates of 0.55% for symmetric and 0.47% for asymmetric contamination.Source: §3.6 · pp. 16–17

3.7 Computational times

  • RMD-Shrinkage had computational times similar to COM and faster than MCD, Adj MCD, Kurtosis, and OGK.The authors characterize its computational times as competitive.Source: §3.7 · p. 18, Table 7 · p. 18

4 Real dataset

On the 357 benign WDBC samples, RMD-Shrinkage labeled fewer observations as outliers than the alternatives, and its flagged points inside the central-data box appeared distinct from the median. Its computation time was similar to COM and faster than the other competitors.

4 Real dataset

  • The analysis used only the 357 benign samples from the 569-observation WDBC dataset, with 30 features describing cell nuclei.The dataset also contains 212 malignant samples, which were not included in this analysis.Source: §4 · p. 18
  • The multivariate boxplot visualization ranks observations by L1 depth and defines its central 50% box using coordinate-wise minima and maxima.The fences use 1.5 times the interquartile range beyond those bounds.Source: §4 · p. 19, Figure 1 · p. 19
  • RMD-Shrinkage labeled the fewest observations as outliers; its detections inside the central-data box were distinct from the median, while competitor detections appeared similar or close to it.The authors report that every central-box outlier detected by RMD-Shrinkage was also detected by competitors.Source: §4 · pp. 19–20, Figure 2 · p. 20, Figure 3 · p. 21
  • RMD-Shrinkage had a computational time similar to COM and shorter than the other competitors on this dataset.Table 9 reports computational times for the WDBC comparison.Source: §4 · p. 21, Table 9 · p. 21

5 Conclusions

The proposed RMDs perform well across most simulations, including skewed and heavy-tailed settings; RMD-Shrinkage is the most competitive version. It is approximately affine equivariant, has high breakdown under highly contaminated data, and is computationally inexpensive, while adjusted-quantile and depth-based improvements remain open questions.

5 Conclusions

  • RMD-Shrinkage is the most competitive version, with high-precision outlier detection in most simulations across Gaussian, skewed, and heavy-tailed distributions.The conclusion also says competitors lack desirable overall behavior, especially in high dimensions.Source: §5 · p. 21
  • Under correlated and transformed data, RMD-Shrinkage is approximately affine equivariant, and it has high breakdown value under highly contaminated data, even in high dimensions.The paper also reports inexpensive computation time and says the real-data example bears out these conclusions.Source: §5 · p. 22
  • Whether adapting the adjusted quantile or using alternative definitions of depth could improve performance remains to be examined.The paper notes that depth is a robust measure for location.Source: §5 · p. 22

6.1 Proofs

The three proofs derive optimal shrinkage parameters by simplifying their objective functions and applying first-order optimality conditions. They use robust location or covariance estimators in practice, with the L1-median proof also invoking a normality-based asymptotic result.

6.1.1 Proof of Theorem 1.

  • Theorem 1's proof reduces the optimization by using E(ˆµCCM) = µ, then minimizes the parameter-dependent term and applies first-order conditions for scaling and shrinkage intensity.In practice, µ is estimated with ˆµCCM.Source: §6.1.1 · pp. 22–23

6.1.2 Proof of Theorem 2.

  • Theorem 2's proof uses the L1-median covariance result under normality to remove an inner-product term, then derives optimal parameters using first-order conditions.In practice, µ is estimated with ˆµMM.Source: §6.1.2 · p. 24

6.1.3 Proof of Theorem 3.

  • Theorem 3's proof uses E(ˆSCCM) = Σ to simplify the objective, then minimizes its parameter-dependent term and applies first-order conditions for scaling and shrinkage intensity.In practice, Σ is estimated with ˆSCCM.Source: §6.1.3 · pp. 25–26

6.2 Tables

The normal-distribution tables cover correct and false detection rates across several contamination settings, dimensions, outlier distances, and contamination concentrations. Separate computational-time tables report results for selected distance and concentration settings.

6.2.1 Normal distribution

  • Tables 11–14 report correct and false detection rates for multivariate Normal simulations across contamination levels, dimensions, outlier distances, and contamination concentrations.Table 10 covers false detection rates with no contamination; the supplied passages do not report the table values or identify a winning method.Source: §6.2.1 · p. 27, Table 10 · p. 27, Table 11 · p. 28

6.2.2 Computational times

  • Tables 15–17 report computational times for Normal data under selected outlier-distance and contamination-concentration settings.The captions specify δ = 5 and λ = 1, δ = 10 and λ = 0.1, and δ = 10 and λ = 1, respectively.Source: Table 15 · p. 30, Table 16 · p. 30, Table 17 · p. 31

6.3 Multivariate Student-t distribution with 3 degrees of freedom

The Student-t simulations with 3 degrees of freedom report false detection rates without contamination and correct and false detection rates across multiple contamination settings. The supplied passages identify the table scopes but do not provide the results.

6.3 Multivariate Student-t distribution with 3 degrees of freedom

  • Tables 19–22 report correct and false detection rates for Student-t data with 3 degrees of freedom across contamination levels, dimensions, outlier distances, and contamination concentrations.Table 18 reports false detection rates with no contamination; the supplied passages do not report table values or identify a winning method.Source: §6.3 · p. 31, Table 18 · p. 31, Table 19 · p. 32

6.4 Multivariate Exponential distribution

This section reports false detection rates without contamination and correct and false detection rates for simulations using a multivariate Exponential distribution across methods and specified settings.

6.4 Multivariate Exponential distribution

6.5 Figures

The figures present outlier-detection results for a real dataset, including standardized data and outputs from several methods. They also show detections among the most central half of the data by competitors and RMD-Shrinkage.

6.5 Figures

Loading 1904.02596v1…