PaperlayerView exact revision on arXiv

Source-linked AI summary

Anomaly Detection by Robust Statistics

Peter J. Rousseeuw, Mia Hubert

arXiv:1707.09752v2stat.ML

TL;DR

Anomalous cases can distort analyses or provide valuable information, so reliable detection is needed. The paper surveys robust statistical procedures that fit the majority before flagging deviations, spanning multiple data-analysis settings and cellwise anomalies. The survey illustrates these methods on real data and concludes that robust methods support anomaly detection while remaining an active area for efficiency improvements.

  • Problem

    Anomalies may damage conclusions or contain valuable information, while classical fitting can prevent reliable detection when outliers distort the fitted model.

  • Method

    The paper surveys robust procedures that fit the majority of observations and flag cases deviating from the robust fit across several statistical and machine-learning settings.

  • Results

    The methods are illustrated on real data across frameworks ranging from univariate to high-dimensional analysis, including cellwise anomalies.

  • Takeaways & Limitations

    Robust statistics provides a broad toolkit for detecting anomalous data, including robust location and scale methods and procedures for complex data settings.

  • Takeaways & Limitations

    The paper identifies improving the efficiency of robust methodologies as a direction for future research.

Abstract

from arXiv · show

Real data often contain anomalous cases, also known as outliers. These may spoil the resulting analysis but they may also contain valuable information. In either case, the ability to detect such anomalies is essential. A useful tool for this purpose is robust statistics, which aims to detect the outliers by first fitting the majority of the data and then flagging data points that deviate from it. We present an overview of several robust methods and the resulting graphical outlier detection tools. We discuss robust procedures for univariate, low-dimensional, and high-dimensional data, such as estimating location and scatter, linear regression, principal component analysis, classification, clustering, and functional data analysis. Also the challenging new topic of cellwise outliers is introduced.

INTRODUCTION

Robust statistics detects anomalous cases by fitting the majority of data and flagging observations that deviate from that fit. The paper surveys robust procedures across data types and introduces cellwise anomaly detection.

  • Anomalies may harm conclusions or contain valuable information, making their detection important.
  • Classical fitting can be distorted so severely by outliers that the fitted model fails to identify deviating observations.
  • Robust statistics fits the majority of observations and identifies outliers through large deviations such as distances or residuals.
  • The survey covers robust location and scale, multivariate data, regression, PCA, classification, clustering, and functional data analysis.
  • Cellwise anomaly detection is introduced as an emerging research topic.

ESTIMATING UNIVARIATE LOCATION AND SCALE

The section contrasts fragile classical estimators with robust location and scale procedures. Robust estimators resist contamination, while some methods trade robustness for efficiency or require distributional and tuning choices.

  • The median yields 6.28 in the contaminated example and remains reasonable, illustrating greater robustness to an outlier.
  • Replacing one observation can change the mean completely; its breakdown value is 1/n and approaches 0% as n grows.
  • The median has an approximately 50% breakdown value, meaning it can resist almost half the observations being outliers.
  • The standard deviation has breakdown value 0%; in the example, it changes from s = 0.035 on clean data to s = 25.41 with an outlier.
  • Robustness can reduce efficiency under the Gaussian model, motivating procedures such as location M-estimators that balance robustness and efficiency.
  • The conventional boxplot assumes symmetry and can flag regular observations in asymmetric distributions; skewness-adjusted fences address this issue.

MATION

Robust location, scatter, and distance methods fit the majority of multivariate data before identifying points that deviate from that fit. The examples show robust distances exposing outliers that classical distances can mask, while method choice depends on dimension and contamination.

  • Location and scatter: The classical tolerance ellipse is inflated toward three dinosaur observations, leaving their classical Mahalanobis distances near the boundary.The observations have low brain weight and high body weight.
  • Location and scatter: The MCD estimates location and scatter from h observations whose covariance determinant is lowest, then computes robust distances from those estimates.A consistency factor and optional reweighting follow the covariance estimate.
  • Location and scatter: The robust tolerance ellipse exposes the three dinosaurs while placing two other species near its upper boundary.The robust ellipse is shown in blue against the classical ellipse.
  • Distance-distance diagnostics: In the distance-distance plot, robust distances flag all outliers whereas classical Mahalanobis distances do not.Large disagreement between the two distances identifies atypical observations; flagged points should be investigated rather than automatically deleted.

LINEAR REGRESSION

Robust regression addresses the extreme sensitivity of least squares to outliers and leverage points. Least Trimmed Squares fits the majority, while outlier maps combine robust residuals and predictor distances to distinguish atypical observations.

  • Motivation: Least squares is extremely sensitive to outliers that do not follow the majority’s linear pattern.Its optimality assumes Gaussian errors.
  • Classical regression failure: In the stars example, four giant stars exert leverage that pulls the classical least-squares line to a negative slope despite the upward main sequence.Their classical residuals are not necessarily larger than those of main-sequence stars, so residuals alone can miss them.
  • Robust regression: LTS fits the majority by minimizing only the ordered subset of squared residuals, allowing outliers to retain large residuals.The robust line fits the main sequence and remains far from the four giant stars.
  • Robustness and inference: LTS has a 50% breakdown value when h ≈ n/2, while larger h gives roughly (n−h)/n.Weighted least squares using LTS weights retains robustness, improves efficiency, and supports usual inferential output.
  • Outlier maps: Standardized LTS residuals identify regression outliers, while robust distances of predictors identify leverage points regardless of their responses.The outlier map separates regular, vertical-outlier, good-leverage, and bad-leverage observations.
  • Outlier maps: For stackloss, the robust outlier map identifies point 4 as vertical, points 1, 3, and 21 as bad leverage, and point 2 as good leverage.The classical residual-distance plot flags none of these points because classical quantities mask atypical observations.
  • Follow-up analysis: Flagging atypical points starts further analysis: researchers should investigate why they differ and may revise transformations, model terms, or variable selection.Very high-dimensional regression may require sparse methods such as penalization.

PRINCIPAL COMPONENT ANALYSIS

PCA reduces dimension by representing data through principal components, but classical PCA can be distorted by anomalous observations. Robust PCA methods and outlier maps address this sensitivity while distinguishing different outlier types.

  • Classical PCA: PCA represents covariance structure through a small number of principal components, which are linear combinations of the original variables.Classically, successive components maximize projected variance subject to orthogonality constraints.
  • Classical PCA: Classical PCA is sensitive to anomalous observations because outliers can attract the first components away from regular-data variation.The sensitivity arises from the classical variance and covariance matrix used in the decomposition.
  • Robust PCA methods: Robust PCA replaces or supplements classical covariance estimation with robust covariance estimators, projection pursuit, or the hybrid ROBPCA method.ROBPCA combines projection-pursuit dimension reduction with MCD-based processing in the reduced space.
  • PCA outlier maps: Combining both distances distinguishes regular observations, good leverage points, orthogonal outliers, and bad leverage points.Good leverage points have high score distance but small orthogonal distance, whereas bad leverage points are high on both measures.
  • Examples and extensions: For glass spectra with 750 wavelengths and 180 vessels, ROBPCA revealed bad leverage points and orthogonal outliers that classical PCA did not reveal.The classical PCA components were tilted toward the outliers.

OTHER MODELS

The paper surveys robust approaches for classification, clustering, and functional data analysis. These methods adapt covariance estimation, dimension reduction, trimming, or functional visualization to handle anomalous or complex data.

  • Classification: Robust classification replaces classical covariance matrices with robust estimators such as MCD or S-estimators.For high-dimensional data, robust PCA can precede classification, or PCA can be applied separately within groups as in robust SIMCA.
  • Classification: Robust SIMCA classifies new observations using orthogonal distance to each group subspace and score distance within that subspace.This extends groupwise PCA classification with a robust PCA fit for each group.
  • Classification: Nonlinear SVM classification handles outliers relatively well when a bounded kernel is used, unlike SVMs with unbounded kernels.KROBPCA combined with LDA is presented as an alternative.
  • Clustering: Robust clustering includes medoid-based methods, trimmed k-means, and constrained maximum-likelihood approaches for non-spherical clusters.Trimmed k-means can leave n − h cases unassigned while fitting an h-subset.
  • Functional data analysis: Functional data analysis treats observations as functions rather than individual data points and includes robust PCA, classification, and graphical outlier tools.The literature discusses shape, shift, magnitude, and isolated outliers, while heatmaps and functional outlier maps help flag and type them.

DETECTING OUTLYING CELLS

Cellwise outlier detection addresses anomalous individual entries rather than only anomalous rows. DetectDeviatingCells and its cellmap identify deviations relative to column, row, and cross-variable structure, supporting interpretation of their sources.

  • Cellwise outliers: A small percentage of outlying cells can contaminate many rows in higher dimensions, undermining rowwise robust methods that require at least 50% clean rows.Cellwise outlyingness depends on a cell’s column relationship and the other cells in its row.
  • Cellwise detection: DetectDeviatingCells addresses cellwise contamination by flagging anomalous cells and producing a graphical output called a cellmap.The method is designed to provide information unavailable from rowwise outlier flags alone.
  • Mortality example: In the mortality example, rowwise analysis identified outlying years, while cellwise analysis marked values higher than predicted in red and lower values in blue.The cellwise display associated early outliers with high infant mortality and wartime periods with higher mortality among young adult men.
  • Use of cellwise results: Inspecting deviating cells can support changes in data collection, row or column removal, variable transformation, or model choice.In automated or very large analyses, flagged cells can instead be set to missing and processed with methods for incomplete data.

SOFTWARE AVAILABILITY

The surveyed robust methods are supported by software in R, MATLAB, S-PLUS, and SAS. Packages cover robust estimation, covariance, PCA, classification, clustering, functional data, and cellwise detection.

  • R: R packages provide robust estimators, covariance methods, robust PCA, classification, clustering, functional-data tools, and cellwise outlier detection.The listed packages include robustbase, rrcov, rospca, rrcovHD, cluster, tclust, mrfDepth, and cellWise.
  • R: The cellWise package includes tools for cellwise outlier detection and cellmaps.This directly supports the graphical cellwise analyses discussed in the paper.
  • MATLAB: Many of the methods are also available as MATLAB functions through the LIBRA toolbox.The toolbox is described as downloadable from the cited KU Leuven robust-statistics website.
  • S-PLUS and SAS: MCD and LTS methods are built into S-PLUS and SAS, including SAS/IML.The passage specifies SAS version 11 or higher and SAS/IML version 7 or higher.

CONCLUSIONS

The paper surveys robust statistical methods and algorithms for detecting anomalous data across multiple data-analysis frameworks, including high-dimensional settings. It also covers rowwise and cellwise outliers while identifying efficiency as a future research priority.

  • CONCLUSIONS: The survey examines robust methods and algorithms for detecting anomalous data in real-data applications.The reviewed frameworks range across covariance matrices, linear regression, principal component analysis, and other tasks.
  • CONCLUSIONS: The reviewed scope includes supervised and unsupervised classification, clustering, and functional data analysis.These methods address multiple predictive and analytical settings.
  • CONCLUSIONS: For high-dimensional data, recently developed robust methods use sparse and regularized approaches.
  • CONCLUSIONS: The paper covers both rowwise outliers and newer methods for detecting anomalous data cells.Cellwise outliers are presented as an important topic.
  • CONCLUSIONS: Further research should improve robust-method efficiency in terms of predictive accuracy and computational cost.
Loading 1707.09752v2…