The BerHu penalty and the grouped effect

1 Introduction

• Robust model selection matters when responses may have heavy-tailed errors or outliers and a sparse model is needed to identify significant predictors.The introduction notes that ordinary least squares is reputed to be inefficient in this setting.Source: §1 · p. 1
• Lasso can be biased, selects at most n variables when p exceeds a small sample size n, and may be inconsistent unless nontrivial conditions hold.Adaptive weights were proposed to penalize coefficients differently and attain oracle properties.Source: §1 · pp. 1–2
• For highly correlated predictors, lasso tends to select only one variable, while the Berhu penalty combines lasso-like treatment of small coefficients with ridge-like quadratic shrinkage of larger ones.The earlier non-adaptive Berhu work described an optimization approach but did not establish asymptotic properties.Source: §1 · p. 2
• The adaptive Berhu penalty with concomitant is reported to have oracle properties with ordinary least squares and Huber criteria and to encourage a grouping effect.The largest coefficients are penalized together in an ℓ2 way, while the smallest are treated individually with an ℓ1 penalty.Source: §1 · p. 2

2.1 The adaptive Berhu

• Adaptive weights extend the Berhu penalty, and the paper states that the resulting estimator enjoys oracle properties, unlike the earlier non-adaptive proposal.The earlier Berhu proposal was introduced for robust regression, with no asymptotic feature shown.Source: §2.1 · p. 4
• The adaptive Berhu penalty treats small coefficients with an ℓ1 penalty and penalizes the largest coefficients quadratically, encouraging retention of large coefficients and deletion of small ones.With concomitant scale, the largest coefficients form an implicit group, while smaller coefficients are treated individually.Source: §2.1 · pp. 4–5

2.2 Robust estimation

• The Huber criterion grows quadratically for small residuals and linearly for large ones, and its concomitant-scale version is combined with the Berhu penalty.The resulting criterion is minimized jointly over the intercept, coefficients, scale, and Berhu scale parameter, yielding a scale-invariant robust location procedure.Source: §2.2 · pp. 5–6

2.3 Tuning parameter estimation

• The procedures use weights based on an unpenalized estimator, fix γ = 1 for numerical results, set M = 1.345 and L = M, and select λ_n with BIC-type criteria.The authors say they lack justification for setting L = M, but observe little impact from these parameter choices; the Huber procedure uses a BIC-type criterion from.Source: §2.3 · p. 6

2.4 The Berhu penalty with concomitant induced a grouped effect

• The authors leave the grouping-effect result for Huber loss as an open question.They state that obtaining this result for Huber loss is difficult.Source: §2.4 · p. 7
• The upper bound decreases as predictor correlation increases; with γ = 0, the non-adaptive procedure has the grouping effect property exactly.Source: §2.4 · p. 7
• The adaptive grouping bound does not require coefficient estimates to have the same sign, but its useful grouping guarantee requires an initial estimator that itself has the grouping effect.The authors note that a ridge initial estimator can provide a quantitative grouping description.Source: §2.4 · p. 7

• For Huber loss, the design assumptions require both D1 and D2; least-squares loss requires only D2.Source: §3 · p. 8
• Huber-loss theory assumes symmetric errors with no mass at ±Ms* and positive probability in every interval around zero, but no error-integrability condition.Source: §3 · pp. 8–9
• The adaptive weights use a root-n-consistent unpenalized estimator, with γ > 1/3.Source: §3 · p. 9
• The Huber-Berhu estimator is consistent in variable selection, recovering exactly the active set asymptotically under the theorem’s conditions.Source: §3 · p. 9
• For least-squares loss, analogous oracle-property results hold, and the asymptotic variance matches that in Zou’s theorem [34].Source: §3 · pp. 8–9

4 Some numerical experiments

• The study compares least-squares and Huber criteria paired with adaptive lasso, ridge, adaptive elastic net, or adaptive Berhu, using adaptive weights from the corresponding unpenalized estimator.The simulation section assesses finite-sample performance across specified models and procedures.Source: §4 · p. 10, §4.1 · p. 10

4.1.1 Models used for simulations

• The simulations use 40 predictors, with 15 signals arranged in three highly correlated groups and 25 noise variables, under light-tailed and heavy-tailed error models.The least-squares and Huber losses are treated as well designed for Gaussian and heavy-tailed models, respectively.Source: §4.1.1 · pp. 10–11

4.1.2 Assessing prediction methods

• Prediction is assessed by RPE on a 10,000-design test set, while selection is assessed over 100 fits using counts of correct, overfit, and underfit models and coefficient-zero summaries.Adaptive lasso and Berhu tuning parameters are selected by BIC, while ridge-type and elastic-net parameters use cross-validation.Source: §4.1.2 · pp. 11–12

4.1.3 Comparison results

• Berhu methods compromise between over- and underfitting; with Huber loss on Model 2, they identify the right model a reasonable number of times, and their nonzero-coefficient bias is similar to ridge.By contrast, lasso and elastic-net methods almost never identify the right model and tend to underfit; Berhu has higher variability than ridge.Source: §4.1.3 · p. 12
• OLS has better RPE for Model 1 except at n = 400, but performs less well on Models 2 and 3, especially Model 2; adaptive Berhu and Huber-adaptive lasso also show extreme values and greater variability.The reported RPE comparisons are shown as boxplots.Source: §4.1.3 · p. 12

4.2 Prostate cancer data example

• In prostate-data resampling, methods have similar RPEs, while Berhu yields sparser models than elastic net and lies between lasso’s tendency to select too few variables and elastic net’s tendency to select too many.Berhu’s tuning parameter is also more stable than those for adaptive elastic-net procedures and Huber-adaptive lasso.Source: §4.2 · p. 13

5.1 Computations: software used for numerical optimization

• The optimization problems are solved with CVX, and the BerHu formulation uses fewer constraints and variables than the alternative expression in.The appendix also gives the CVX setup for computing the estimators.Source: §5.1 · pp. 13–14

5.2 Proof of Theorem 1

• For indices i and j both in G, equation (7) of Theorem 1 follows because C_i,j = 1.Source: §5.2 · p. 15

5.3 Proof of Theorem 2

• Theorem 2’s proof establishes asymptotic normality through a unique limiting argmin under assumptions (D2) and (N2), and addresses consistency in variable selection.The proof identifies the active-set inclusion as following from asymptotic normality.Source: §5.3 · p. 16

5.4.1 Proof of lemma 1

• Under λ_n/n^{γ∧1/2}→0, λ_n n^{(γ−1)/2}→∞, λ_n→∞, λ_n>1/3, and β*≠0, the supplied result gives limit u^{−d}u_{p+2}^2C(u_{p+2}) for its expression involving P_{n,j}(u_{1:p},u_{p+2}).Source: §5.4.1 · pp. 17, 21

5.4.2 Proof of lemma 2

• When β* ≠ 0, Lemma 2 states that a unique τ* > 0 satisfies equation (9).The proof uses the intermediate value theorem for existence and strict increase of I′ on each piece for uniqueness.Source: §5.4.2 · p. 21

5.4.3 Proof of lemma 3

• Lemma 3 assumes q(0)=0 and q(x)≥|x|, and its proof uses epigraph geometry to compare a truncated indicator set with the full set.Source: §5.4.3 · pp. 21–22
• The proof of Lemma 3 concludes by combining equations (28), (29), and (31) to establish the lemma’s bound.Source: §5.4.3 · p. 24

Tables 1–3

• Tables 1–3 report selection ability for Models 1–3, each based on 100 replications.The supplied passages give the tables’ scope but not their cell values.Source: Table 1 · p. 28, Table 2 · p. 29, Table 3 · p. 30

Figures 1–3

• Figures 1–3 compare RPE for eight methods at n = 100, 200, and 400; their boxplots omit the listed extreme values.The figures identify ad-lasso, ridge, ad-en, ad-Berhu, and their Huber-loss counterparts.Source: Figure 1 · p. 31, Figure 2 · p. 32, Figure 3 · p. 33

Figures 4–6

• Figures 4–6 show estimates of the first influencing coefficient, whose true value is 3, for the listed methods across Models 1–3.Figure 4 covers Model 1, Figure 5 Model 2, and Figure 6 Model 3.Source: Figure 4 · p. 34, Figure 5 · p. 35, Figure 6 · p. 36

Prostate cancer data

• The prostate-cancer visuals include a method-comparison table and a histogram of variable-selection counts in the resampling study.The supplied passages identify these visuals but provide no cell values or histogram outcomes.Source: Table 4 · p. 37, Figure 7 · p. 38