PaperlayerView exact revision on arXiv

Source-linked AI summary

Robust LassoNet: Enhancing Feature Selection in Neural Networks via Robust Loss Functions

Daniela De Canditiis, Italia De Feis, Paola Stolfi

arXiv:2609.38263v1stat.MLcs.LGmath.PR

TL;DR

Feature selection in neural networks is challenging, and LassoNet’s MSE loss is sensitive to outliers and heavy-tailed noise. Robust LassoNet integrates robust losses into LassoNet’s hierarchical framework; experiments report improved stability, prediction, and feature selection under contamination.

  • Problem

    LassoNet’s standard MSE loss is sensitive to large residuals, and outliers can affect prediction accuracy and variable-selection stability.

    Source: §1 · p. 3

  • Method

    Robust LassoNet integrates robust loss functions into LassoNet’s hierarchical sparse neural architecture.

    Source: §1 · p. 3

  • Results

    Robust losses improve LassoNet’s stability, predictive accuracy, and feature-selection performance under data contamination.

    Source: §4.3 · p. 16, §5 · p. 21

  • Takeaways & Limitations

    Nonnegative Garrote and Cauchy losses substantially improve predictive and selection performance in noisy environments, within the evaluated synthetic and real datasets.

    Source: §5.4 · p. 22

    Under strong mean contamination, Nonnegative Garrote has PE 0.640 and FNR 0.124, worse than the other robust methods, though still far better than MSE.

    Source: §4.3 · p. 14

Abstract

from arXiv · show

Feature selection in neural networks remains a challenging problem, particularly in the presence of noisy or contaminated data. LassoNet is a recent approach that addresses this issue by combining neural networks with hierarchical sparsity constraints, enabling simultaneous prediction and variable selection. However, its standard formulation relies on the mean squared error (MSE) loss, which is known to be highly sensitive to outliers. In this paper, we present Robust LassoNet, an extension of LassoNet that incorporates robust loss functions, such as Huber, Cauchy, Tukey's bisquare, and Nonnegative Garrote, to mitigate the effect of extreme observations. The proposed approach preserves the original optimization framework while improving stability under data contamination. Through experiments on synthetic and real datasets, we show that robust LassoNet significantly improves both predictive performance and feature selection accuracy in the presence of outliers or heavy tailed noise, while maintaining comparable performance in clean settings. These results highlight the importance of robustness in neural network based feature selection and suggest practical guidelines for choosing appropriate loss functions.

1. Introduction

Feature selection in neural networks supports interpretability and generalization, but standard parameter penalties do not ensure input-level sparsity. Robust LassoNet replaces LassoNet’s outlier-sensitive MSE with robust losses while retaining its hierarchical sparsity and optimization framework, and the paper evaluates it on synthetic and real data.

1. Introduction

  • Feature selection can improve interpretability, reduce computational complexity, and enhance generalization, especially when high-dimensional data contain irrelevant or redundant variables.Such variables may induce instability and overfitting.Source: §1 · p. 2
  • Standard weight penalization does not directly guarantee input-variable sparsity in neural networks because parameters do not map one-to-one to input variables.This motivates architectures that integrate feature selection into learning.Source: §1 · pp. 2–3
  • LassoNet combines a linear skip connection and nonlinear network with a hierarchical constraint, allowing a variable into the nonlinear component only when active in the linear part.This enables explicit feature selection while retaining neural-network expressiveness.Source: §1 · p. 3
  • MSE can degrade under outliers, heavy-tailed noise, or corrupted observations; Robust LassoNet replaces it with Huber, Cauchy, Tukey’s bisquare, or Nonnegative Garrote losses.The approach preserves LassoNet’s hierarchical sparsity and proximal optimization strategy; the paper evaluates prediction, variable selection, and robustness on synthetic and real datasets.Source: §1 · p. 3

2. Background and Related Work

Prior neural feature-selection work includes post-training pruning and sensitivity methods, as well as sparsity built into network training. The paper identifies a gap in combining robust losses with LassoNet’s hierarchical feature selection and proximal optimization.

2.1. Feature Selection in Neural Networks

  • LassoNet combines a linear skip connection, nonlinear network, and hierarchical constraint to select variables explicitly while retaining nonlinear flexibility.A feature contributes to the nonlinear component only if active in the linear part.Source: §2.1 · p. 4

2.2. Robustness in Neural Network

  • Many robust neural-network methods target predictive robustness, while prior work on robust nonparametric regression studies estimation error bounds and convergence properties.The cited robust regression losses include least absolute deviation, Huber, Cauchy, and Tukey’s biweight.Source: §2.2 · p. 4

2.3. Research Gap and Motivation

  • The paper reports no prior systematic integration of robust losses into LassoNet while preserving its hierarchical sparsity and proximal optimization scheme.Its proposed integration aims to improve predictive stability and feature-selection reliability in contaminated data while retaining interpretability and computational tractability.Source: §2.3 · p. 4

3. Methods

The methods retain LassoNet’s hierarchical feature-selection structure and proximal-gradient training while replacing its quadratic loss with robust alternatives. The paper describes the resulting framework and several losses, including their tuning or thresholding behavior.

3. Methods

  • An ℓ1 penalty encourages sparse coefficients in regression, but parameter sparsity alone does not guarantee feature-level sparsity in neural networks.The mismatch arises because neural-network parameters do not correspond one-to-one to input variables.Source: §3 · p. 5

3.1. LassoNet

  • LassoNet couples a linear skip connection to a feed-forward network and constrains each input’s nonlinear contribution to require an active linear coefficient.The ℓ1 penalty on the linear coefficients therefore induces variable-level sparsity across the network.Source: §3.1 · p. 5
  • LassoNet trains along a regularization path using gradient updates followed by a hierarchical proximal operator enforcing the ℓ1 penalty and constraint.The procedure repeats the two steps for each fixed regularization parameter.Source: §3.1 · p. 5

3.2. Robust LassoNet

  • Robust LassoNet replaces MSE with a robust M-estimation loss that down-weights large residuals while retaining the hierarchical constraint.The methods section introduces ψ- and ρ-functions to describe robust losses and their properties.Source: §3.2 · p. 7, Weak Redescenders · p. 8
  • The Huber tuning constant sets the transition from quadratic to linear loss behavior, controlling the efficiency–robustness trade-off.A commonly used choice is C ≈ 1.345.Source: §3.2 · p. 7

Weak Redescenders

  • Among the weak redescenders, Cauchy’s C controls loss scale, Nonnegative Garrote uses a thresholding rule, and Tukey’s bisquare is another specified loss family.The paper gives C = 2.0 as a common Nonnegative Garrote threshold and 4.685 as a common Tukey tuning constant.Source: Weak Redescenders · pp. 7–8

4. Implementation and Results

This section evaluates robust-loss LassoNet variants for prediction and variable selection on synthetic data and the Top Gear and Atherosclerosis datasets. It covers performance and selection stability, sparsity, and false-positive control across settings, including trade-offs relative to MSE and variation among robust losses.

4.1. Experimental Setup

  • The implementation used loss-specific cross-validation and hierarchical proximal gradient training; benchmarks included robust tree methods for synthetic data and robust linear selection methods for two real datasets.Tree-variable selection used normalized feature importances with a mean-positive-importance threshold; RGRPLARS or RLARS used BIC for the respective real-data experiments.Source: §4.1 · pp. 8–10

4.2. Synthetic Data Experiments

  • Synthetic benchmarks fixed n=750 observations and d=200 covariates, sampled independently from the standard uniform distribution, and comprised four increasingly challenging nonlinear regression models whose true functions depended only on active features.Models 2–4 tested feature recovery through weak linear proxies under hierarchical sparsity.Source: §4.2 · p. 10

4.3. Results on Synthetic Data

  • With Gaussian noise, robust losses generally maintained competitive PE while reducing NSR and FPR; in Model 3, MSE had the lowest PE but selected substantially larger models.The tree benchmarks behaved differently by model: Random Forest often selected more sparsely, while XGBoost consistently selected larger models.Source: §4.3 · pp. 12–13
  • Under strong mean contamination in Model 4, Tukey had the lowest PE (0.240), while MSE FNR reached 0.992; Tukey FNR was 0.000 under both contamination levels.All robust LassoNet variants outperformed the tree benchmarks on PE and variable selection under both mean-contamination levels.Source: §4.3 · p. 14, Table 5 · p. 15
  • Under strong variance contamination, MSE PE was 0.626, 0.639, 1.242, and 0.666 for Models 1–4, with FNR 0.536, 0.573, 0.833, and 0.756; robust variants were substantially more stable.The relative performance among robust estimators depended on the model.Source: §4.3 · p. 15
  • Tukey achieved the lowest PE under variance contamination for Model 1 at 0.336 moderate and 0.323 strong, and for Model 2 at 0.228 moderate and 0.220 strong.In Model 1, robust LassoNet variants had lower PE and better NSR and FPR control than tree benchmarks, which selected larger models.Source: §4.3 · p. 15
  • Across synthetic models, robust losses substantially outperform MSE and beat tree-based benchmarks in predictive error under contamination, but the best loss varies by model and scenario.Tukey, Huber, and Garrote each lead on different variable-selection or prediction measures; overall, robust losses improve stability, especially under strong contamination and in highly nonlinear settings.Source: §4.3 · p. 16

4.4. Real Data Applications

  • Real-data comparisons use 50 repeated random 80%/20% train-test splits, trimmed PE excluding the largest 5% of absolute residuals, and report variables selected in more than 70% of runs.The analysis compares distributions and 50-run averages of trimmed PE and NSR, alongside relative variable-selection frequencies.Source: §4.4 · pp. 16–17

4.5. Top Gear

  • The four robust LassoNet methods agree most on Top Gear variable selection, identifying a stable core that includes Displacement, CruiseControl, Height, Fuel, DriveWheel, and Origin.Each core variable was selected by all four methods in more than 70% of runs; benchmark methods showed more heterogeneous selections, and robust groupwise LARS recovered most of the core except Origin.Source: §4.5 · p. 18
  • All four robust losses achieve lower and more stable trimmed prediction error than MSE on Top Gear, while robust LassoNet offers a favorable balance of predictive accuracy and model complexity against benchmarks.Random Forest and XGBoost produce sparser models, while robust groupwise LARS has substantially larger trimmed prediction error than robust LassoNet and the tree-based methods.Source: §4.5 · p. 18, Table 11 · p. 19

4.6. Atherosclerosis

  • On Atherosclerosis, robust LassoNet selects a coherent core of genes tied to innate immune activation, interferon signaling, and apoptosis, while benchmark methods show limited overlap with that core.BCL2L11, IFI6, PPP3CB, MAP3K4, and TLR1 were consistently selected across methods; the robust procedures also commonly selected IL2, CR2, CASP8, IRF4, and IRF8. These biological findings align with Huang et al. (2011) and Fan et al. (2017).Source: §4.6 · pp. 20–21
  • 6.532–6.759: Robust LassoNet variants achieve the lowest trimmed prediction errors on Atherosclerosis, while RLARS is sparser but less accurate than the best robust variant.The reported trimmed PE values are Garrote 6.532, XGB 8.193, and RF 12.398; RLARS has trimmed PE 7.117 and NSR 0.103, compared with robust LassoNet NSR 0.278–0.631.Source: §4.6 · pp. 20–21, Table 12 · p. 21

5. Discussion and Conclusion

Robust LassoNet improves prediction and feature selection under contamination while retaining LassoNet’s hierarchical sparsity structure. Its robust losses have similar performance to the standard variants under ideal Gaussian conditions, with modest training-time overhead.

5. Discussion and Conclusion

  • The discussion concludes that robust losses improve model stability, predictive accuracy, and feature selection under data contamination.The authors describe the framework as a flexible, computationally tractable basis for further robust and interpretable deep-learning developments.Source: §5 · p. 21, §5.4 · p. 22

5.1. Interpretation of Results

  • Under ideal Gaussian conditions, all LassoNet variants perform comparably; under contamination or heavy-tailed noise, MSE is sensitive to outliers, while NNG and Cauchy perform best across most scenarios.Huber offers a balance of efficiency and robustness; Tukey’s bisquare suppresses extreme outliers but can introduce slight bias at low contamination.Source: §5.1 · p. 21

5.2. Computational and Methodological Insights

  • Robust extensions preserve the hierarchical proximal step, and robust versions require less than 10% longer training times while converging smoothly across configurations.The gradient component replaces the quadratic residual term with the chosen robust loss’s derivative.Source: §5.2 · p. 22

5.3. Implications and Future Work

  • Robust LassoNet may apply to settings such as biomedical analysis, sensor measurements, and econometric modeling, where high-dimensional predictors coexist with measurement errors.Future work includes consistency and oracle analyses, automatic robustness-parameter tuning, classification and multi-output regression, and scalable implementations.Source: §5.3 · p. 22

5.4. Concluding Remarks

  • Nonnegative Garrote and Cauchy losses substantially improve predictive and selection performance in noisy environments.The framework retains LassoNet’s interpretability and sparsity-inducing power.Source: §5.4 · p. 22

A. Appendix

The appendix collects boxplots from the synthetic experiments and visual summaries of performance and variable selection on two real datasets.

A. Appendix

  • Top Gear and Atherosclerosis figures show trimmed PE and variable-level NSR boxplots alongside heatmaps of variable selection across methods.The heatmaps encode selection frequency among variables selected in at least 70% of 50 runs by at least one method.Source: Figure 1 · p. 23, Figure 3 · p. 23, Figure 2 · p. 23
  • Synthetic boxplots show performance metrics for Models 1–4 under 10% mean contamination, with Gaussian, moderate-mean, and strong-mean scenarios arranged by column.The first row shows a realization of the true and contaminated training responses.Source: Figure 5 · p. 25, Figure 6 · p. 26, Figure 7 · p. 27
  • Additional synthetic boxplots cover Models 1–4 under 10% variance contamination, with Gaussian, moderate-variance, and strong-variance scenarios arranged by column.The first row shows a realization of the true and contaminated training responses.Source: Figure 9 · p. 29, Figure 10 · p. 30, Figure 11 · p. 31
  • Table 13 lists the fixed hyperparameters used by the tree baselines.Source: Table 13 · p. 33

B. Hyperparameter selection

The tree baselines use fixed, lightly regularized hyperparameters, with selected settings adjusted for the small dataset sample sizes.

B. Hyperparameter selection

CRediT authorship contribution statement

The statement attributes conceptualization, methodology, and software contributions to all three authors.

CRediT authorship contribution statement

Loading 2609.38263v1…