PaperlayerView exact revision on arXiv

Source-linked AI summary

Robust Linear Regression: A Review and Comparison

Chun Yu, Weixin Yao, Xue Bai

arXiv:1404.6274v1stat.ME

TL;DR

OLS can be strongly affected by outliers, motivating robust alternatives. The paper reviews and compares available methods using breakdown point and efficiency, simulations, and real data; MM-estimates and REWLSE show the best overall performance among the compared methods.

  • Problem

    OLS estimators are very sensitive to unusual design values and y-outliers, with even one atypical observation potentially having a large effect on parameter estimates.

  • Method

    The paper reviews robust regression methods and compares them using breakdown point and efficiency, supported by a simulation study and real-data application.

  • Results

    MM-estimates and REWLSE had the overall best performance among the compared robust methods.

  • Takeaways & Limitations

    MM-estimates and REWLSE combine high breakdown point and high efficiency among the reviewed methods.

  • Takeaways & Limitations

    The comparison includes scenarios in which methods that work well for y-direction outliers remain sensitive to high-leverage outliers.

Abstract

from arXiv · show

Ordinary least-squares (OLS) estimators for a linear model are very sensitive to unusual values in the design space or outliers among y values. Even one single atypical value may have a large effect on the parameter estimates. This article aims to review and describe some available and popular robust techniques, including some recent developed ones, and compare them in terms of breakdown point and efficiency. In addition, we also use a simulation study and a real data application to compare the performance of existing robust methods under different scenarios.

1 Introduction

The paper reviews robust regression methods motivated by OLS sensitivity to outliers, comparing methods through breakdown point, efficiency, simulations, and a real-data application.

  • A single outlier can have a large effect on OLS parameter estimates.
  • The review compares robust estimators using breakdown point and efficiency, alongside simulation and real-data comparisons.Efficiency compares robust estimates with OLS under normal errors without outliers, while breakdown point measures tolerated contamination before divergence.
  • M-estimates resist unusual y observations but are sensitive to high-leverage points in x, giving them a breakdown point of 1/n.
  • R-estimates have relatively high efficiency, but their breakdown points are as low as those of OLS estimates.
  • LMS, LTS, and S-estimates have high breakdown points but low efficiency, whereas GS-estimates retain high breakdown points with slightly higher efficiency.
  • MM-estimates and REWLSE can combine high breakdown points with high efficiency, while GM-estimates address leverage through downweighting but may lose efficiency.The paper also reports that S1S generalized M-estimates have high breakdown points and high efficiencies.
  • The reviewed methods include regularized case-specific estimators, with Huber’s ψ-function M-estimation identified as a special case of one proposed estimator.

2 Robust Regression Methods

M-estimators replace least squares with robust loss functions, reducing sensitivity to unusual y values but not necessarily to high-leverage observations. LMS and related methods improve breakdown protection, often trading away efficiency or influence-function properties.

  • M-estimates replace the least-squares criterion with a robust loss function applied to residuals.
  • OLS has BP = 1/n, which tends to zero as n grows, so one unusual observation can strongly affect its estimate.
  • Huber’s ψ function uses c = 1.345, while Tukey’s bisquare with c = 4.685 produces 95% efficiency.
  • LAD resists unusual y values more than OLS but remains sensitive to high-leverage outliers, with BP = 1/n → 0 and efficiency 0.64 under normal errors.
  • General monotone M-estimates also have BP = 1/n → 0 because they lack immunity to high-leverage outliers.
  • LMS has breakdown point near 0.5 but at best 0.37 efficiency, lacks a well-defined influence function, and can initialize other robust methods.

2.3 LTS Estimates

LTS achieves high resistance to outliers through trimming, attaining a 0.5 breakdown point, but its efficiency is very low. It is therefore commonly used as an initial estimate for other robust procedures.

  • 2.3 LTS Estimates: LTS trims residuals using q = [n(1 − α) + 1], where α is the proportion of trimming.
  • 2.3 LTS Estimates: LTS has breakdown point BP = 0.5 and convergence rate n^-1.
  • 2.3 LTS Estimates: LTS efficiency is about 0.08 relative to OLS estimates.
  • 2.3 LTS Estimates: Despite its low efficiency, LTS is traditionally used as an initial estimate for other high-breakdown, high-efficiency robust methods.
  • 2.3 LTS Estimates: S-estimates can attain BP = 0.5 but have asymptotic efficiency 0.29 under normally distributed errors.
  • 2.3 LTS Estimates: Generalized S-estimates seek to improve S-estimator efficiency while retaining a breakdown point as high as that of S-estimates.

2.6 MM-Estimates

MM-estimates use a high-breakdown initial estimator, robust scale estimation, and a final M-estimation stage. Their tuning separates breakdown protection from normal-distribution efficiency, although larger efficiency settings increase outlier sensitivity.

  • 2.6 MM-Estimates: MM-estimates use a three-stage procedure: high-breakdown initialization, robust residual-scale estimation, and a final M-estimate.
  • 2.6 MM-Estimates: LMS or S-estimates with Huber or bisquare functions are typically used as the initial estimator β̂0.
  • 2.6 MM-Estimates: An MM-estimate’s breakdown point depends only on k0, while its asymptotic variance depends only on k1.
  • 2.6 MM-Estimates: k1 can be chosen to attain desired normal efficiency without affecting the breakdown point, subject to k1 ≥ k0.
  • 2.6 MM-Estimates: Larger k1 values yield higher normal-distribution efficiency but make MM-estimates more sensitive to outliers; efficiency 0.85 with k1 = 3.44 is recommended in practice.

2.7 Generalized M-Estimates (GM-Estimates)

GM-estimates modify M-estimation to address high-leverage points through leverage-dependent weights. Mallows and Schweppe variants can lose efficiency or consistency, whereas S1S achieves BP = 0.5 and 0.95 efficiency under the stated comparison setting.

  • 2.7.1 Mallows GM-estimate: Mallows GM-estimates assign less weight to observations with high leverage than to observations with small leverage.
  • 2.7.1 Mallows GM-estimate: Mallows weighting also downweights good leverage points that follow the bulk pattern, resulting in efficiency loss.
  • 2.7.2 Schweppe GM-estimate: Schweppe GM-estimates adjust leverage weights according to the residual size, but are not consistent under asymmetric errors.
  • 2.7 Generalized M-Estimates (GM-Estimates): Mallows and Schweppe GM-estimates have breakdown points no greater than 1/(p + 1), where p is the number of unknown parameters.
  • 2.7.3 S1S GM-estimate: S1S computes the final M-estimate in one step after initial residual and scale estimates are supplied.
  • 2.7.3 S1S GM-estimate: S1S, initialized with LTS residual estimates and LMS scale estimates, gives BP = 0.5 and 0.95 efficiency.

2.8 R-Estimates

R-estimates rank residuals and offer scale equivalence, but their score-function choice and breakdown properties vary. REWLSE adaptively weights observations and combines a 0.5 breakdown point with asymptotic efficiency under normal errors.

  • R-estimates minimize scores of ranked residuals using a monotone score function.
  • Scale equivalence is an advantage of R-estimates over M-estimates, although the optimal score function is unclear.
  • Most R-estimates have breakdown point BP = 1/n →0, while bounded-influence variants can retain high efficiency under normal errors.
  • REWLSE is a weighted least-squares estimator whose weights are adaptively calculated from an initial robust estimator.
  • With initial regression and scale estimates having BP = 0.5, REWLSE also has breakdown point 0.5.
  • Under normally distributed errors, REWLSE is asymptotically equivalent to OLS and therefore asymptotically efficient.

parameters

Mean-shift regularization identifies outliers through sparse observation-specific shifts while estimating regression parameters robustly. Penalty and thresholding choices determine the resulting estimator and its resistance to leverage points.

  • The mean-shift model assigns a nonzero γ_i to an outlying observation and zero otherwise.
  • L1-penalized least squares jointly estimates β and γ, then estimates β by OLS after replacing y with y − γ.
  • Iterative updates solve the two estimation steps, and with Huber’s ψ function the resulting estimate is equivalent to an M-estimate.
  • Replacing the L1 penalty with a general penalty produces thresholding solutions corresponding to soft, hard, and SCAD penalties.
  • Minimizing the penalized objective yields sparse γ for outlier detection and a robust estimate of β.
  • Hard or SCAD penalties correspond to redescending ψ functions and resist high-leverage outliers when initialized with a high-breakdown robust estimate.

3 Examples

Simulations across error distributions, response outliers, leverage outliers, sample sizes, and regression settings consistently favor MM and REWLSE. A cigarette-consumption example shows that these methods are less influenced by a high-leverage observation than OLS.

  • Example 1: The simulations compare parameter-estimate MSE across six error scenarios, including heavy tails, contamination, response outliers, and high-leverage outliers.
  • Example 1: MM and REWLSE have the overall best performance through most cases and remain consistent across sample sizes.
  • Example 1: Under normal errors, LSE has the smallest MSE, while MH, MT, MM, and REWLSE have similar MSE because of high efficiency.
  • Example 1: For Cauchy errors and response outliers, robust methods outperform LSE; MM and REWLSE are among the better-performing estimators.
  • Example 1: With high-leverage outliers, MT and MH perform poorly, whereas MM and REWLSE work better than other robust estimates.
  • Example 2: In the second simulation example, results similarly identify MM and REWLSE as the overall best performers.
  • Example 3: For the cigarette data, USA is a high-leverage outlier, and MM and REWLSE fitted lines better represent the majority of observations than LSE.
  • Example 3: Removing USA substantially changes LSE estimates, while REWLSE estimates remain unchanged and match LSE fitted to the data without USA.

4 Discussion

The discussion contrasts robust estimators by breakdown point and efficiency. MM-estimates and REWLSE combine strong robustness with high efficiency, whereas other methods trade one property against the other.

  • MM-estimates and REWLSE have both high breakdown point and high efficiency.
  • The paper therefore presents MM-estimates and REWLSE as methods that attain both high breakdown protection and efficiency.
  • The simulation study shows that MM-estimates and REWLSE have the overall best performance among the compared robust methods.
  • GM-estimates, bounded R-estimates, M-estimates, and LAD estimates are less attractive because of their low breakdown points.
  • LMS, LTS, S-estimates, and GS-estimates strongly resist outliers but have low efficiency.

MSE vs. Cases for intercept

Figure 1 presents MSE across cases for intercept and slope estimates from five robust regression methods in model 1 with n = 100. Figure 2 provides corresponding MSE plots for regression parameter estimates in model 2.

  • Figure 1 plots MSE against different cases for LMS, LTS, S, MM, and REWLSE under model 1 with n = 100.
  • The intercept estimates are shown on the left side of Figure 1.
  • The slope estimates are shown on the right side of Figure 1.
  • Figure 2 plots MSE of different regression parameter estimates across cases for the same five methods under model 2 with n = 100.
Loading 1404.6274v1…