PaperlayerView exact revision on arXiv

Source-linked AI summary

Leveraging Sparse Linear Layers for Debuggable Deep Networks

Eric Wong, Shibani Santurkar, Aleksander Mądry

arXiv:2105.04857v1cs.LGstat.ML

TL;DR

Deep networks are difficult to interpret, and localized explanations can make it challenging to assess overall decision processes. This paper fits sparse linear decision layers over learned deep features and reports high accuracy alongside greater human interpretability, with debugging applications in vision and language.

  • Problem

    Deep-network scale and the complexity of defining and assessing interpretability make it difficult to understand decisions, while localized explanations can mislead about overall decision processes.

    Source: §1 · p. 1

  • Method

    The authors fit elastic-net sparse linear decision layers to deep features and use feature-interpretation tools to inspect the features and coefficients involved in predictions.

    Source: §2.1 · p. 2, §2.2 · p. 4

  • Results

    Untrained annotators simulated sparse-layer predictions with ∼63% accuracy, versus ∼33% for models with standard dense decision layers.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The paper demonstrates sparse-feature explanations in debugging tasks including detecting spurious correlations, explaining misclassifications, and diagnosing model biases in vision and language settings.

    Source: §1 · p. 2, §6 · p. 11

    Feature visualizations for standard vision models are often hard to parse, so the main paper shows adversarially trained models and places standard-model plots in the appendix.

    Source: §3 · p. 4

Abstract

from arXiv · show

We show how fitting sparse linear models over learned deep feature representations can lead to more debuggable neural networks. These networks remain highly accurate while also being more amenable to human interpretation, as we demonstrate quantiatively via numerical and human experiments. We further illustrate how the resulting sparse explanations can help to identify spurious correlations, explain misclassifications, and diagnose model biases in vision and language tasks. The code for our toolkit can be found at https://github.com/madrylab/debuggabledeepnetworks.

1 Introduction

The paper targets model debugging: helping designers uncover unexpected behavior without seeking a complete characterization of a deep network’s decisions. It replaces dense decision layers with sparse ones and evaluates their accuracy and human interpretability.

1 Introduction

  • Because localized explanations can mislead about a model’s overall decision process, the authors focus on helping designers uncover unexpected behavior semi-automatically.They frame this as model debugging rather than a complete characterization of decisions.Source: §1 · p. 1
  • Sparse decision layers can use only 20–30 deep features per ImageNet class without sacrificing much model performance.The approach fits elastic-net-regularized linear models over deep representations.Source: §1 · pp. 1–2
  • Untrained annotators simulated sparse-layer predictions with ∼63% accuracy, versus ∼33% for models with standard dense decision layers.The authors use human experiments to assess whether people can understand the resulting models.Source: §1 · p. 2
  • The paper explores sparse decision layers for diagnosing biases and spurious correlations, generating counterfactuals, and identifying data patterns behind misclassifications.The authors design human-in-the-loop experiments to support this analysis.Source: §1 · p. 2

2 Debuggability via Sparse Linearity

The paper makes deep-network decisions easier to inspect by pairing learned features with a sparse linear decision layer. It uses elastic-net models and feature-interpretation tools to examine the small set of features and coefficients involved in predictions.

2 Debuggability via Sparse Linearity

  • Inspecting a sparse decision layer means examining the few deep features and coefficients that determine predictions, rather than a dense layer with potentially millions of parameters.The decomposition retains learned deep features while making the linear combination more tractable to inspect.Source: §2 · p. 2

2.1 Constructing sparse decision layers

  • Elastic net fits sparse linear models to normalized deep-feature representations and targets, with regularization paths supporting validation-based selection of sparsity or accuracy.The path spans models from zero weights to fully dense, and can show how feature weights change as sparsity varies.Source: §2.1 · pp. 2–3
  • A custom solver combines an elastic-net path algorithm with variance-reduced gradient methods to fit ImageNet-scale paths to numerical precision in hours on one GPU.Existing solvers struggle at deep-learning scale and cannot be directly parallelized because they rely on coordinate descent.Source: §2.1 · p. 3

2.2 Interpreting deep features

  • LIME and feature visualization interpret selected deep features, reducing explanations to a smaller set of features and their input patterns.LIME uses local surrogates around highly activating examples to identify image superpixels or sentence words; feature visualization synthesizes inputs that maximally activate a neuron.Source: §2.2 · p. 4

3 Are Sparse Decision Layers Better?

The section evaluates whether sparse decision layers preserve model performance while making decisions easier to understand. It reports sparsity–accuracy trade-offs, tests whether top-weighted features suffice, and assesses human understanding.

3.1 Sparsity vs. performance

  • Up to two orders of magnitude sparsity costs little accuracy; ImageNet uses about 20 features per class versus 2,048 dense, while toxic classifiers use fewer than 10 per class with improved generalization.Source: Figure 3 · p. 5, §3.1 · p. 5

3.2 Sparsity and feature highlighting

  • For dense layers, the top 5–10 features do not capture full performance; in language, they yield near-chance accuracy (~50%), unlike sparse layers’ largely sufficient top features.Source: §3.2 · pp. 5–6

3.3 Sparsity and human understanding

  • Sparse SST features activate for positively meaningful words, unlike a prominent dense feature; quill visualizations suggest sparse layers focus on salient class characteristics.The main-paper visualizations use adversarially trained models, whose features tend to be more human-aligned [Tsi+19] [Eng+19].Source: §3 · p. 4, §3.3 · p. 6
  • Across 1,000 tasks, annotators identified sparse-model predictions with 63.02 ± 3.02% accuracy, versus 35.61 ± 3.09% for dense layers, supporting that sparse layers reflect model reasoning.Annotators received no semantic information about the target label beyond the feature visualizations.Source: §3.3 · p. 6

4 Debugging deep networks

The section applies sparse decision layers to three debugging tasks: diagnosing biases and correlations, generating counterfactuals, and analyzing misclassifications.

4.1 Biases and (spurious) correlations

  • Sparse explanations exposed identity-word reliance in Toxic-BERT and showed Debiased-BERT using identity words as evidence against toxicity.Identity-related words comprised 27% of Toxic-BERT toxicity-feature word clouds and 43% of Debiased-BERT non-toxicity-feature word clouds.Source: §4.1 · p. 7
  • Adding “christianity” to toxic sentences flipped sparse Debiased-BERT predictions to non-toxic 74.4% of the time.The passage also reports that the standard model made the same flip 62.2% of the time.Source: §4.1 · p. 7
  • Annotators identified shared ImageNet patterns 20% more often for sparse-layer models, and judged a lower fraction of those patterns spurious.They inspect prototypical images across classes and describe shared patterns as object parts or surroundings.Source: §4.1 · pp. 7–8, Figure 5 · p. 8

4.2 Counterfactuals

  • Sparse-layer sentiment counterfactuals replace a sentence’s positively associated word with a negatively associated counterpart for a shared feature.The method uses word clouds to select candidate substitutions; ImageNet examples also show manually adding identified spurious patterns can flip sparse-layer predictions.Source: §4.2 · pp. 8–9, Figure 6 · p. 9
  • 73.1 ± 3.0% of sentiment counterfactuals flipped sparse-layer predictions, versus 52.2 ± 4% for the dense decision layer.The authors say sparse layers can make it easier to identify deep features linked to model predictions.Source: §4.2 · pp. 8–9

4.3 Misclassifications

  • Over 30% of sparse-layer ImageNet misclassifications were fixed by zeroing a single deep feature.Annotators matched the predicted-class feature to about 70% of misclassified images and judged it a better match than the ground-truth feature 60% of the time; these diagnoses can guide model or dataset changes.Source: §4.3 · p. 9, Table 8 · p. 10

5 Related Work

Prior interpretability work studies individual units, local decisions, or concept-based explanations, while this paper uses those tools to probe sparse decision layers. Related work also includes deep-feature decision trees and regularized GLM solvers.

Interpretability tools

  • Local attribution methods explain individual predictions, but cited work raises concerns that they can be easy to fool or miss global behavior [Sla+20] [STY17] [Ade+18] [Ade+20] [LM20].Examples include saliency maps [SVZ13] [Smi+17] [STY17], local surrogates [RSG16a], and influential, prototypical, or counterfactual inputs [KL17] [Goy+19a].Source: §5 · p. 10
  • Feature visualizations [Yos+15] [Ngu+16] and semantic labels [Bau+17] [Bau+20] probe hidden units, while concept-based explanations use high-level concepts [Kim+18] [Gho+19] [Yeh+20].The authors say their approach is complementary, using existing interpretation methods as primitives; they contrast their identified features with concepts that need not be causally linked to predictions [Goy+19b].Source: §5 · pp. 10–11
  • Wan et al.’s decision-tree explanations rely on heuristics and a domain-specific hierarchy such as WordNet, which the authors say limits applicability [Wan+20].Their method fits a tree on deep features and explains decisions with semantic descriptions of nodes along the path.Source: §5 · p. 11

Regularized GLMs and gradient methods

  • The glmnet solver is CPU-oriented and best suited to problems with few examples or features, whereas the authors’ solver builds on variance-reduced proximal-gradient methods [DBLJ14] [GGS19].The passage characterizes glmnet as state-of-the-art in CPU solving times, but not directly amenable to GPU acceleration.Source: §5 · p. 11

6 Conclusion

The paper presents sparse linear models over deep representations as a route to more debuggable networks and describes a GPU-accelerated solver for fitting regularized GLMs at ImageNet scale. Its solver combines minibatch variance reduction, compact gradient storage, and regularization paths, with a stated theoretical caveat.

6 Conclusion

  • Sparse linear models over deep representations provide a broadly applicable approach to more debuggable models, with usage scenarios across vision and language.The authors note that the approach applies to deep networks with a final linear layer.Source: §6 · p. 11
  • Human experiments test model simulatability, spurious-correlation detection, and misclassification validation; the authors suggest their design may be independently useful.They also hope their elastic-net solver will support large-scale sparse linear modeling beyond deep learning.Source: §6 · p. 11

A SAGA-based solver for generalized linear models

  • The solver combines variance-reduced optimization with GLM path algorithms to enable GPU-accelerated fitting at ImageNet scale.It uses a minibatch derivative of SAGA, with a proximal elastic-net step after the gradient step [GGS19].Source: A SAGA-based solver for generalized… · pp. 15–16
  • Storing one scalar per example and output instead of each full gradient reduces the gradient table to O(nk), which the authors report fits within GPU memory for ImageNet.For each output, the stored scalar is the residual-like quantity (x_i^Tβ + β_0 − y_i).Source: A SAGA-based solver for generalized… · p. 15
  • The compact-memory trick requires placing ℓ2 regularization in the proximal operator, so the smooth objective is not guaranteed strongly convex and the cited convergence analysis does not strictly apply.The authors report strong practical convergence rates in this setting despite that theoretical limitation.Source: A SAGA-based solver for generalized… · p. 15
  • Regularization paths use solutions from stronger regularization as warm starts for successively weaker values, scheduled logarithmically down to λmin = ελmax.The authors use K = 100 values and ε = 0.001 in their experiments.Source: A SAGA-based solver for generalized… · p. 16

A.1 Timing Experiments

The appendix reports solver timing and describes how elastic-net paths support feature ordering, alongside procedures for interpreting features in vision and language models. The feature visualizations and LIME-based procedures use different inputs and aggregation strategies across these settings.

A.1 Timing Experiments

  • At 50,000 CIFAR-10 examples, glm-saga takes 33 minutes versus nearly 13 hours for glmnet; it fits regularization paths for one million ImageNet examples in approximately 6 hours.glmnet is faster on smaller problems, but its runtime exceeds glm-saga’s at 3,000 datapoints.Source: Table 36 · p. 17, §A.1 · p. 17

A.2 Elastic net, ℓ1, and ℓ2 regularization

  • Elastic net combines ℓ1-driven sparsity with ℓ2 regularization’s performance, uniqueness, and grouping effects; correlated features can become nonzero together along the path.The passage describes ℓ1 as primarily providing sparsity and ℓ2 as providing a grouping effect for similar neurons.Source: §A.2 · p. 17

A.3 Feature ordering

  • Features that enter the sparse decision layer earlier as regularization decreases are treated as more useful under sparsity and appear more semantically aligned with their class.The appendix illustrates regularization paths and feature orderings for standard and robust ResNet-50 classifiers on ImageNet and Places-10.Source: §A.3 · p. 18, Figure 12 · p. 18, Figure 13 · p. 19

B Feature interpretations

B.1 Feature visualization

  • For standard networks, feature visualization uses Lucid regularizers and Fourier-space optimization to discourage high-frequency artifacts in synthesized inputs.The passage notes that vanilla pixel-space gradient descent often yields artifacts and high-frequency patterns for standard networks; robust models can produce meaningful visualizations with pixel-space gradient descent.Source: §B.1 · p. 21

B.2 LIME

  • Vision LIME aggregates superpixel explanations from the top-activated images for a feature, while language LIME averages each word’s weights across validation sentences and visualizes the top 30 words.The figures use k = 1 prototypical image, which the authors found representative of results with higher k.Source: §B.2 · p. 21

C.1 Datasets

The experiments cover ImageNet-1k, Places-10, sentiment, and toxic-comment classification, using ResNet-50 and BERT models. Toxic-comment test sets are balanced to make the reported accuracy baseline interpretable.

C.1 Datasets

  • The vision datasets are ImageNet-1k and a ten-class Places-10 subset, while the language datasets are SST sentiment and Toxic Comments.Places-10 contains ten specified scene classes; SST uses positive and negative labels, and Toxic Comments includes six toxicity categories.Source: §C.1 · p. 22
  • Toxic-comment test sets are randomly subsampled to 50% toxic and 50% non-toxic examples, making random-chance accuracy 50% for those experiments.The passage notes that unbalanced test sets can yield baseline accuracy above 90% by always predicting non-toxic.Source: §C.1 · p. 22

C.2 Models

  • The experiments use ResNet-50 classifiers for vision and BERT models for language; vision includes both standard and robust models.The BERT models are pretrained, use the base architecture, and are sourced from HuggingFace.Source: §C.2 · p. 22

D.1 Trade-offs for all datasets

Appendix D traces sparsity–accuracy paths across vision and language tasks, then describes validation-based selection and visual comparisons of sparse and dense features.

D.1 Trade-offs for all datasets

  • Each point in the vision and language sparsity–accuracy curves represents a linear classifier from the regularization path.The SST curve is repeated in the Toxic-BERT and Debiased-BERT language plots.Source: §D.1 · p. 23, Figure 16 · p. 23

D.1.1 Selecting a single sparse model

  • The selected model is the sparsest decision layer whose validation accuracy is no more than 5% below the best layer’s.The validation set is 10% of the training set, and the threshold is illustrative and application-dependent.Source: §D.1.1 · p. 23, Figure 17 · p. 23

D.2 Feature highlighting

  • The extended feature-highlighting comparison covers accuracy using only the top-k features, overall accuracy, and accuracy gained from remaining features.Table 37 adds a wide ImageNet representation and three toxic-comment categories for each classifier.Source: Table 37 · p. 24, §D.2 · p. 24

D.3.1 Language models

  • SST word clouds show clearer positive and negative semantic patterns for sparse-layer features than for dense-layer or randomly selected features.For the comparison, dense-layer weights determine whether the randomly selected features are positive or negative.Source: Figure 16 · p. 25

D.3.2 Vision models

  • Vision figures pair dense- and sparse-layer coefficients with feature visualizations and LIME superpixels for randomly chosen classes.The additional feature interpretations are limited to at most five randomly selected features per model and class.Source: §D.3 · p. 24, Figure 17 · p. 26, Figure 18 · p. 27

D.4 Human evaluation

The MTurk study tests whether annotators can simulate predictions from dense versus sparse decision layers using selected features and candidate images; sparse-layer predictions were substantially easier to simulate.

Study setup

  • The study used an adversarially trained ImageNet ResNet-50 sparse layer with 57.65% test accuracy and 39.18 features per class on average.The sparse layer was selected using a validation set.Source: §D.4 · p. 32

Task setup

  • For each of 100 target classes and each layer type, annotators saw five weighted features and chose among three images sampled from target-probability percentiles.Features had weights greater than 5% of the model’s maximum; image candidates came from the (90, 95), (98, 99), and (99.99, 100) percentile ranges.Source: §D.4 · pp. 32–33

Study setup

  • The study comprised 2,000 tasks, with five annotators per task; reported choices used confidence-weighted majority voting after quality-control exclusions.Tasks were retained only when at least two of five annotators made selections.Source: §D.4 · pp. 33–34

Results

  • Annotators simulated sparse-layer predictions at nearly 30% higher accuracy, while dense-layer performance was close to chance at 33%.Annotators could also simulate sparse-layer predictions when the correct image belonged to a different class.Source: §D.4 · p. 34, Table 25 · p. 34

E.1 Toxic comments

Sparse-layer word clouds expose identity-related patterns in toxic-comment classifiers, while an MTurk study collects descriptions of shared ImageNet features and their associated correlations.

E.1 Toxic comments

  • Toxic-BERT’s sparse-layer top features include more identity-group words than the dense layer’s top features, whose top ten neurons contain 7.5% such words.The paper notes that this distinction is less clear in the dense-layer word clouds.Source: §E.1 · p. 34, Figure 26 · p. 35
  • Debiased-BERT’s sparse-layer word clouds show identity words less prevalently among toxic-comment features but often among non-toxic features.The paper interprets this pattern as identity words serving as strong evidence for non-toxicity, also visible to a lesser degree in dense-layer word clouds.Source: §E.1 · p. 34, Figure 27 · p. 35

E.2.1 Human study

  • The ImageNet MTurk study sampled class pairs sharing a substantially weighted feature, then asked annotators to describe shared patterns and judge whether they belonged to the object or its surroundings.Annotators saw three maximally activating images per class and provided confidence ratings; tasks required selections from at least three annotators.Source: §E.2.1 · p. 36, Figure 28 · p. 37

E.2.2 Additional visualizations of spurious correlations

F Counterfactual experiments

The counterfactual experiments generate language or image inputs by modifying patterns associated with a model’s prediction, then test the model on the resulting inputs.

F.1 Language counterfactuals

  • Language counterfactuals replace a sentence word used as evidence for the predicted label with a word from the opposite activation cloud of its deep feature.The procedure selects a relevant feature-word pair and substitutes a word from the feature’s positive or negative word cloud, as appropriate.Source: §F.1 · p. 39
  • Some sentences have no candidate substitutions because no deep feature uses their words as evidence for the prediction.In the sentiment experiments, analysis was restricted to sentences with counterfactuals in both dense and sparse layers, though similar results were reported when considering each model’s possible counterfactuals separately.Source: §F.1 · pp. 39–40

F.2 ImageNet counterfactuals

  • Image counterfactuals add a discovered spurious pattern to images from other ImageNet classes, using annotated regions and masks to combine the images.The resulting inputs are tested to see whether the added pattern leads the model to assign them to the target class; the described example associates water with snorkel.Source: Figure 31 · p. 40, §F.2 · p. 41

G.1 Human study

The study tests whether sparse decision-layer features help identify patterns behind misclassifications and relate shared features to class confusion. Annotators compare feature visualizations, while additional examples and feature-overlap analysis illustrate these debugging uses.

G.1 Human study

  • 26% of misclassifications (5,673 examples) were flipped to the ground-truth class by zeroing the top predicted-class feature, termed a “problematic” feature.The score ranks features by their relative contribution to the predicted versus ground-truth class.Source: §G.1 · p. 41
  • Annotators evaluated feature visualizations for 1,330 sampled misclassifications, comparing the top ground-truth and predicted-class features or a random-feature control.Five annotators saw each task without the ground-truth or predicted labels; results retained tasks with at least two annotator selections and used a confidence-weighted majority vote.Source: §G.1 · pp. 41, 43

G.2 Additional error visualizations

  • Additional examples show annotators judging the top feature for the predicted class a better image match than the top feature for the ground-truth class.Figure 33 presents these examples.Source: §G.2 · p. 43, Figure 33 · p. 43

G.3 Model confusion

  • For models with sparse decision layers, feature overlap between class pairs is significantly correlated with model errors; inspecting corresponding shared features can help understand underlying causes of inter-class confusion.Source: §G.3 · p. 44
Loading 2105.04857v1…