PaperlayerView exact revision on arXiv

Source-linked AI summary

Post-Hoc Feature Selection Layer for Neural Networks Interpretability

João Kenji Suwa, João Pedro Silveira e Silva, Bruno Iochins Grisci

arXiv:2610.31420v1cs.LGcs.AI

TL;DR

Neural networks’ black-box nature complicates interpretability, including for pre-trained models. The paper adapts FSL into a post-hoc module that learns feature weights while keeping the pre-trained network’s parameters unchanged. The authors report that it identifies relevant features in high-dimensional datasets while matching or outperforming state-of-the-art post-hoc feature-weighting methods.

  • Problem

    Neural networks’ black-box nature poses challenges for transparency, ethical accountability, and bias mitigation.

    Source: §I · p. 1

  • Method

    Post-hoc FSL adds a trainable feature-weighting layer to a pre-trained neural network while leaving the network’s learned parameters unchanged.

    Source: §I · p. 1, A. Architecture and Mechanism · p. 2

  • Results

    The method effectively isolates relevant features in high-dimensional datasets, matching or outperforming state-of-the-art post-hoc feature-weighting methods.

    Source: §VI · p. 6

  • Takeaways & Limitations

    The authors report that post-hoc FSL can be integrated into the model architecture for subsequent inferences and that zeroing irrelevant weights increases reliability.

    Source: §VI · p. 6

    The method underperforms in multi-class settings and is designed for tabular data, where features have fixed meanings.

    Source: A. Limitations and future work · p. 6

Abstract

from arXiv · show

The interpretability of complex neural networks remains a critical challenge, especially for models already deployed in high-stakes domains. To address this, we adapted the Feature Selection Layer (FSL) to optimize feature weights post-training, a variant we term post-hoc FSL. Our approach reframes the FSL as a lightweight, trainable module that integrates with already frozen pre-trained models on tabular datasets to highlight the features the original model considers most important. This post-hoc FSL learns feature relevance by fine-tuning its weights based on the original model's learned outputs. Crucially, this process is non-invasive, operating without altering the original model's architecture or its learned parameters. We conducted our experiments using both statistical and visual metrics, including accuracy, F1 score, recall, precision, weighted t-SNE and silhouette score, and also analyzed the stability of the post-hoc FSL on high-dimensional synthetic and real-world tabular datasets. We compared the post-hoc FSL feature weighting method using these metrics against the original embedded FSL and other post-hoc interpretability methods, such as Integrated Gradients, Noise Tunnel, DeepLIFT, Gradient SHAP, and Feature Ablation. Experimental results demonstrate that the post-hoc FSL feature weighting method successfully identified relevant features across the different datasets, maintaining the predictive power of the original neural network while enhancing its interpretability.

I. INTRODUCTION

The introduction frames neural-network interpretability as a challenge for transparency, accountability, and bias mitigation. It presents post-hoc FSL as a way to analyze feature relevance in pre-trained networks, addressing a constraint of the original embedded FSL.

I. INTRODUCTION

  • DNNs’ black-box nature challenges transparency, ethical accountability, and bias mitigation, making the accuracy–interpretability trade-off a concern for safe deployment.Feature weighting methods assign relevance scores to input features to provide insight into feature behavior.Source: §I · p. 1
  • Post-hoc FSL adapts the embedded FSL to optimize feature weights after initial training, enabling feature-relevance analysis on pre-trained networks without jointly training from scratch.The approach keeps the backbone frozen, and its FSL weights reportedly converge rapidly compared with training from scratch.Source: §I · p. 1

A. Feature Selection

  • Feature-selection methods include model-agnostic filters, model-specific wrappers and embedded methods, hybrids, and ensembles, with differing trade-offs in efficiency, interactions, generalization, and stability.Filters are computationally efficient but often ignore feature interactions; wrapper and embedded approaches can be costly and generalize less across architectures.Source: A. Feature Selection · p. 1

B. Feature Selection Layer

The original Feature Selection Layer is an embedded feature-weighting layer placed between a neural network’s input and first hidden layer. Its feature-aligned weights are jointly trained with the network, with a modified L1 regularization term.

B. Feature Selection Layer

  • The FSL places one neuron per input feature between the network input and first hidden layer, and its weights are jointly trained with the network.The weights are initialized to 1/n, where n is the number of features, and reflect feature relevance during learning.Source: B. Feature Selection Layer · p. 1

W FSL

  • The modified L1 regularization term is calculated to ensure interaction between features.The passage introduces the term’s calculation but does not provide a readable formula here.Source: B. Feature Selection Layer · pp. 1–2
  • The regularization term uses FSL weights W_FSL, feature count n, and λ, a value between 0 and 1 that sets regularization strength.The FSL also adds its activation function and λ as network hyperparameters.Source: W FSL · p. 2

C. Post-hoc Interpretability Methods

The paper situates post-hoc FSL among feature-attribution methods and proposes it as a trainable module for compatible pre-trained networks. It scales input features before the network while leaving the network’s learned parameters unchanged.

C. Post-hoc Interpretability Methods

  • Integrated Gradients, Gradient SHAP, and DeepLIFT estimate relevance relative to reference inputs; Feature Ablation perturbs inputs, while Noise Tunnel smooths gradient-based attributions.The experiments compare post-hoc FSL with these approaches, including Noise Tunnel applied with Integrated Gradients.Source: C. Post-hoc Interpretability Methods · p. 2

III. PROPOSED METHOD

  • Post-hoc FSL is proposed as an adaptation of FSL for interpreting pre-trained neural networks without altering their original learned parameters.It is described as a trainable module applicable to any compatible pre-trained network.Source: §III · p. 2

A. Architecture and Mechanism

  • A one-to-one, trainable feature layer scales each input before a pre-trained network, whose internal parameters remain unchanged; the resulting weights quantify feature importance.The layer can be removed after analysis so the pre-trained model returns to its original state.Source: A. Architecture and Mechanism · p. 2
  • In Figure 1, four inputs are individually multiplied by activated FSL weights before entering the pre-trained model, while only FSL weights are updated during fine-tuning.All parameters of the pre-trained model remain frozen.Source: Figure 1 · p. 2

B. Post-Hoc Training Procedure

The post-hoc training procedure integrates the feature-weighting layer with a frozen pre-trained network and optimizes only the layer’s weights. Repeated training converges to weights whose magnitudes represent feature importance.

B. Post-Hoc Training Procedure

  • The procedure starts with a pre-trained model whose internal parameters are frozen, alongside input and FSL weight vectors, an activation, loss, and learning rate.The feature layer is integrated between the input and hidden layers of the network.Source: B. Post-Hoc Training Procedure · p. 2
  • The combined FSL and frozen network process data, and the final output is used to compute the whole-model loss.Source: B. Post-Hoc Training Procedure · p. 2
  • Backpropagation computes gradients only for FSL weights, and repeating the process over epochs lets them converge to magnitudes representing feature importance.Source: B. Post-Hoc Training Procedure · p. 2

C. Layer Configuration: Regularization, Activation, and Initialization

The layer configuration uses sparsity-promoting regularization, non-negative weights, and unit initialization; the experiments cover synthetic and real-world datasets.

C. Layer Configuration: Regularization, Activation, and Initialization

  • L1 regularization promotes sparsity, ReLU makes weights non-negative, and initializing weights at 1.0 starts with unchanged feature values.The passage says zero weights indicate irrelevance and positive weights represent feature importance.Source: C. Layer Configuration:… · p. 2

IV. EXPERIMENTS

  • The experiments evaluate the proposed method using datasets, evaluation metrics, and implementation details.Source: §IV · pp. 2–3

A. Datasets

  • The dataset inventory includes synthetic data and real-world email-spam and high-dimensional microarray datasets.The datasets contain complete numerical features with no missing values.Source: A. Datasets · p. 3, Table I · p. 3

1) Synthetic Datasets:

The evaluation uses synthetic data with known feature relevance, real-world datasets from cancer microarrays and email spam, and statistical, visual, and stability metrics.

1) Synthetic Datasets:

  • Synthetic XOR and SynthA datasets vary the proportions of relevant and irrelevant features, and SynthA provides ground truth for evaluating feature-selection accuracy.SynthA informative features are coordinates of points in a hypercube with Gaussian clusters at its vertices.Source: 1) Synthetic Datasets · p. 3

2) Real-world datasets:

  • Real-world evaluation covers Liver-GSE22405 and Breast-GSE45827 microarray data plus an email-spam dataset represented by word frequencies.The microarray tasks include binary and six-class classification.Source: 2) Real-world datasets · p. 3

B. Evaluation Metrics

1) Performance Metrics:

The evaluation combines predictive-performance and synthetic feature-selection metrics with weighted t-SNE visual analysis.

1) Performance Metrics:

  • Accuracy, precision, recall, and F1 score measure predictive performance; PIFS and PSFI, each ranging from 0.0 to 1.0, measure complementary aspects of informative-feature selection.PIFS is the percentage of informative features selected, while PSFI is the percentage of selected features that are informative.Source: 1) Performance Metrics · p. 3

2) Visual Metrics:

  • Weighted t-SNE scales pairwise distances by feature importance to visualize how feature-scoring methods affect clustering in a low-dimensional space.Higher silhouette values indicate well-separated, dense clusters; values near 0 suggest boundary points, and values near −1 indicate points closer to another cluster.Source: 2) Visual Metrics · p. 3

3) Stability Metrics:

The experiments compare post-hoc FSL with embedded FSL and other post-hoc methods, assessing predictive performance, feature selection, visual clustering, and stability across synthetic datasets.

3) Stability Metrics:

  • Stability is assessed by applying feature weighting across stratified cross-validation folds and comparing selected-feature overlap, weight correlation, and rank correlation.The measures are Jaccard Index, Pearson correlation, and Spearman rank correlation.Source: 3) Stability Metrics · p. 3

C. Experiments setup

  • The evaluation trains baseline and embedded-FSL models, compares them with post-hoc methods, and also tests TabPFN.Metrics are averaged across cross-validation folds.Source: C. Experiments setup · p. 3

A. Results for Synthetic Datasets

  • On XOR, baseline predictive metrics were 0.999 versus 1.0 for FSL and post-hoc FSL, and all methods achieved perfect top-2 PIFS, PSFI, and Jaccard scores.Pearson correlations were near 1.0, while Spearman rank correlations remained near 0.0.Source: A. Results for Synthetic Datasets · p. 4
  • On SynthA, post-hoc FSL scored 0.906 on both PFSI and PIFS for the top-30 features, below embedded FSL’s 1.0 on both metrics.Other post-hoc approaches scored 0.96–0.99 on these metrics.Source: A. Results for Synthetic Datasets · p. 4
  • On SynthA, both FSL variants surpassed the baseline across F1 score, accuracy, precision, and recall; post-hoc FSL scored approximately 0.916 and embedded FSL approached 0.954.The passage reports lower average silhouette for post-hoc FSL (0.115) than for embedded FSL (0.191) and other post-hoc approaches (0.181–0.235).Source: A. Results for Synthetic Datasets · p. 4
  • On TabPFN with SynthA, post-hoc FSL and Kernel SHAP both scored 0.966 on top-30 PSFI and PIFS; silhouette scores were 0.316, 0.186, and 0.070 for post-hoc FSL, Kernel SHAP-weighted, and standard t-SNE, respectively.Source: A. Results for Synthetic Datasets · p. 4, Figure 2 · p. 4

B. Results for Real-world Datasets

Across real-world datasets, post-hoc FSL retained predictive performance comparable to baseline models while producing strong feature-separation results. Stability varied by dataset: it lagged other post-hoc methods on spam rankings but had the highest Liver Jaccard and Pearson scores.

C. Feature Erasure

Feature erasure tests whether highly ranked features matter to the baseline model by masking them and measuring subsequent predictive performance. Masking selected top-ranked features significantly degraded baseline accuracy and F1 across XOR, SynthA, and Spam, with similar effects across methods.

  • Feature Erasure zeroes the top-n features ranked by post-hoc weights, then runs inference on the baseline model.Source: C. Feature Erasure · p. 5
  • Masking the top-2 XOR, top-30 SynthA, and top-2000 Spam features significantly degraded baseline F1-Score and accuracy.Source: C. Feature Erasure · p. 6
  • All tested methods, including post-hoc FSL, produced similar performance degradation after masking, indicating that each selected relevant features.Source: C. Feature Erasure · p. 6

VI. CONCLUSION

Post-hoc FSL is presented as a feature-weighting method for interpreting pre-trained neural networks, with reported effectiveness on high-dimensional data. Its stated limits are weaker performance in multi-class settings and an assumption of fixed-meaning tabular features.

VI. CONCLUSION

  • The method effectively isolates relevant features in high-dimensional datasets, matching or outperforming state-of-the-art post-hoc feature-weighting methods.Source: §VI · p. 6
  • Post-hoc FSL can be integrated for subsequent inference and drives irrelevant feature weights toward zero; experiments also indicate a typically positive impact on predictive performance.Source: §VI · p. 6

A. Limitations and future work

Related papers

Loading 2610.31420v1…