PaperlayerView exact revision on arXiv

Source-linked AI summary

Understanding Failures of Deep Networks via Robust Feature Extraction

Sahil Singla, Besmira Nushi, Shital Shah, Ece Kamar, Eric Horvitz

arXiv:2012.01750v3cs.CV

TL;DR

Aggregate metrics do not expose which visual attributes or instance groups drive model failures, and collecting attribute metadata can be costly. Barlow uses interpretable features from robust-model representations to discover and visualize failure modes. Human studies provide evidence that the workflow helps people interpret features and identify important error clusters.

  • Problem

    Aggregate scores can hide non-uniform errors across instance groups, while human-collected visual metadata is costly and may not reflect the attributes models use.

    Source: §1 · p. 1

  • Method

    Barlow extracts features from robust-model representations and visualizes them to help identify dataset-wide image failure modes.

    Source: §1 · p. 1, §4 · p. 4

  • Results

    Human studies provide evidence that practitioners can interpret features and identify important error clusters, and that visualizations help explain many extracted features.

    Source: §8 · p. 7, §10 · p. 8

  • Takeaways & Limitations

    The identified modes and visualizations can inform iterative model debugging, including adding diverse-background examples or masking overemphasized features.

    Source: §1 · p. 2, §10 · p. 8

    Determining whether a failure reflects a spurious or overemphasized feature, and formulating mitigation, still depends on human expertise.

    Source: §1 · p. 1

Abstract

from arXiv · show

Traditional evaluation metrics for learned models that report aggregate scores over a test set are insufficient for surfacing important and informative patterns of failure over features and instances. We introduce and study a method aimed at characterizing and explaining failures by identifying visual attributes whose presence or absence results in poor performance. In distinction to previous work that relies upon crowdsourced labels for visual attributes, we leverage the representation of a separate robust model to extract interpretable features and then harness these features to identify failure modes. We further propose a visualization method aimed at enabling humans to understand the meaning encoded in such features and we test the comprehensibility of the features. An evaluation of the methods on the ImageNet dataset demonstrates that: (i) the proposed workflow is effective for discovering important failure modes, (ii) the visualization techniques help humans to understand the extracted features, and (iii) the extracted insights can assist engineers with error analysis and debugging.

1. Introduction

Aggregate metrics can obscure uneven model failures, while collecting visual-attribute metadata is costly and may miss cues models use. Barlow uses robust-model representations to identify and visualize failure modes, with ImageNet examples including spurious correlations and overemphasized features.

1. Introduction

  • Aggregate metrics such as AUC or accuracy do not reveal failures concentrated in particular instance clusters, while human metadata collection is costly and may miss model-focused attributes.Prior examples include systematic errors for young children and higher error rates for women with darker skin tone.Source: §1 · p. 1
  • Barlow leverages robust-model representations to generate metadata and identify two failure types: spurious correlations and causally relevant but overemphasized features.The method is demonstrated on ImageNet.Source: §1 · p. 1
  • Human expertise remains necessary to determine the failure type and formulate mitigation steps.Source: §1 · p. 1
  • Barlow visualizes failure modes and suggests distinct interventions, including adding diverse-background images or masking overemphasized features in training data.Examples include background variation for maillot and monastery images, and masking purse buckles or syringe markings.Source: §1 · p. 2
  • The work contributes a discovery framework, robust-feature extraction and visualization, a crowdsourcing study of interpretability, and an engineer study of debugging effectiveness.Source: §1 · p. 2

2. Related work

Prior explanation methods often inspect individual predictions or depend on features collected through other means. This work targets dataset-wide image failure modes using features automatically extracted from robust representations.

2. Related work

  • Robust-model visualizations are reported as more interpretable than standard-model visualizations, while activation maps alone may not identify a feature’s specific image attribute.The paper combines heatmaps with feature-activation visualizations to help identify relevant attributes.Source: §2 · p. 2
  • Disentangled VAE representations are another feature-extraction option, but the paper notes difficulty scaling them to rich datasets such as ImageNet.Source: §2 · p. 2
  • Local explanations inspect individual images, whereas this work seeks major failure modes across a dataset to guide model improvement.Prior approaches use tabular features, query-definable text operators, or crowdsourced image metadata; this work automatically extracts image features from learned robust representations.Source: §2 · p. 3

3. Background and method overview

Barlow describes failures with interpretable rules that identify image clusters with elevated error rates and meaningful coverage of benchmark errors. It uses cluster error rate, error coverage, and benchmark base error rate to quantify those failures.

3. Background and method overview

  • Error rate measures the fraction of images in a cluster that are misclassified, while error coverage measures the fraction of benchmark errors in that cluster.Base error rate is the error rate for the whole benchmark, treated as one cluster.Source: §3 · p. 3
  • Barlow seeks low-dimensional, human-interpretable rules that identify clusters with both high error rates and substantial coverage of benchmark errors.The explanations are intended as actionable guidance for engineers, not as failure predictors.Source: §3 · p. 3
  • Figure 2 depicts the end-to-end Barlow workflow, with feature extraction and visualization followed by feature selection and failure-mode generation.Source: Figure 2 · p. 3

4. Feature extraction and visualization

Barlow extracts image features from a robust model’s representation and visualizes those features with activating images, heatmaps, and feature attacks. These views help distinguish the visual attribute a neuron responds to.

4. Feature extraction and visualization

  • For each image, Barlow uses the penultimate layer of a pretrained robust neural network as a feature vector whose elements encode visual attributes.The robust model is trained for l2 robustness.Source: §4 · p. 3
  • A robust ResNet-50 pretrained on ImageNet can supply features even when the model being analyzed is not robust.In that setting, the features are treated as data attributes rather than necessarily as part of the inspected model’s representation.Source: §4 · p. 3
  • Heatmaps clarify which image region activates a neuron; in Figure 3, they indicate focus on the ground where top-activating images leave sky, water, or ground ambiguous.Source: Figure 3 · p. 4, §4 · p. 4
  • Robust-model features are visualized using top-activating images, CAM-based heatmaps, and feature attacks that maximize a neuron with respect to the original image.Heatmaps can resolve ambiguity in activating images; feature attacks are used when heatmaps remain ambiguous.Source: §4 · p. 4

5. Failure mode generation

Barlow generates failure explanations by fitting shallow decision trees to robust-model features and selecting rules that capture high-error clusters. It evaluates trees by the error concentration and coverage represented by their leaves, rather than relying on classification metrics alone.

5. Failure mode generation

  • Figure 4 shows feature attack resolving ambiguity left by heatmaps, with the neuron appearing to focus on limbs and tail.Source: Figure 4 · p. 4
  • Failure-mode generation fits decision trees to features and model failures, with analyses organized around prediction groupings for false positives and label groupings for false negatives.The inspected model need not be the robust model used to extract features.Source: §5 · p. 4
  • Barlow selects the top 20 features by mutual information and truncates trees to depth 3 or less, retaining paths above an error-coverage threshold and at least BER + δ error rate.The thresholds δ and τ are adjustable.Source: §5 · pp. 4–5

5.1. Automatic evaluation of decision tree

  • Traditional accuracy, precision, and recall are insufficient for explanatory evaluation because feature incompleteness, generalization difficulty, and rare failures complicate finding all failure modes.The paper therefore focuses on the explanatory properties of failure analyses.Source: §5.1 · p. 5
  • ALER−BER measures the increase in error and discrepancy explained by a decision tree relative to the benchmark’s base error rate.ALER is the leaf error rate averaged with each leaf weighted by its error coverage.Source: §5.1 · p. 5
  • Importance value ranks a leaf by the product of its error rate and error coverage, so leaves with larger values contribute more to ALER.Source: §5.1 · p. 5

6. Failure modes discovered by Barlow

Barlow uses robust-model features to identify failure modes in ImageNet classifiers and examines whether those features correlate with errors. Examples show errors rising when a purse lacks a buckle or a predicted Rhodesian Ridgeback lacks a collar.

6. Failure modes discovered by Barlow

  • Barlow analyzes a standard ImageNet ResNet-50 using features from a robust ResNet-50, the training set, and depth-one trees, selecting the leaf with highest IV(Cl).The training set provides 1,300 images per class versus 50 in the validation set; discovered failure modes may vary by benchmark.Source: §6 · p. 5
  • For purse images, error rises from BER 0.3085 to 0.4179 (+10.94%) when feature[1456] < 0.3641; this node contains 0.6409 of all failures.The reported ALER is 0.3433.Source: Figure 6 · p. 6
  • For images the model predicts as Rhodesian Ridgebacks that lack a dog collar, predictions are more likely to be false positives, with error increasing by 10.91%.Source: §6 · p. 6

7. Experiments with automated evaluation

Automated evaluation compares failure-explanation trees across grouping strategies, model robustness, and tree depths. Ground-truth-label grouping and robust-model explanations score better under the reported ALER − BER criterion.

7. Experiments with automated evaluation

  • Ground-truth-label grouping produces better ALER − BER scores than prediction grouping for both standard and robust models and across tree depths.The comparison is shown for depth-three trees in Figure 8.Source: §7 · p. 6, Figure 8 · p. 6
  • Robust-model failure explanations score significantly better than standard-model explanations across both grouping strategies and tree depths.The cited comparisons are in Appendix Figures 16 and 19.Source: §7 · pp. 6–7
  • Increasing tree depth significantly improves the score for both models; at least 0.787 of classes have a valid leaf for every robust-model tree.For standard-model trees, the fraction is at least 0.596 except for prediction grouping at depth 1, where it is 0.211.Source: §7 · p. 7

8. Crowd study on feature interpretability

A crowdsourcing study assessed whether people could understand Barlow feature visualizations. Workers reported ease and confidence for many features, and most found the heatmap, feature attack, or both useful.

8. Crowd study on feature interpretability

  • The study presented most-activating images, heatmaps, and feature attacks to elicit descriptions and assess how easily workers could understand the features.The appendix includes agreement analysis and examples workers found easy or difficult to describe.Source: §8 · p. 7
  • Workers reported average ease-of-understanding and confidence scores above 3 for more than 800 features.The study evaluated 1,200 visualizations using 5,971 answers from 312 workers; five workers assessed each visualization except 29 assessed by four.Source: §8 · p. 7
  • Among responses, 17.67% reported feature-attack visualizations, 32.14% reported both heatmaps and feature attacks, and 7% reported none; authors say heatmaps were most valuable for understanding feature meaning.Only 0.92% of the 1,200 features received None as the majority vote from all five workers; authors describe crowd-worker results as likely a lower bound on effectiveness for engineers.Source: §8 · p. 7

9. Study with machine learning practitioners

Fourteen experienced vision-ML practitioners used Barlow to inspect ImageNet classification failures and suggest mitigation actions. They reported that the explanations prompted ideas, while some features were difficult to describe.

9. Study with machine learning practitioners

  • The study involved 14 Microsoft practitioners with prior vision-ML experience, asked to imagine deploying an ImageNet robust ResNet-50 classifier.Each one-hour session began with an explanation of Barlow; participants inspected two groupings and saw decision trees, node error and coverage, instances, and feature visualizations.Source: §9 · p. 7
  • In a study of 14 ML practitioners at Microsoft, all participants described features in at least one of two inspected groupings, while 10 of 14 described features in both.Source: §9 · p. 7
  • Practitioners identified spurious correlations and necessary-but-insufficient features, and said explanations suggested mitigation ideas given sufficient time and resources.Data collection was the most popular action item; model-architecture suggestions often concerned increasing capacity.Source: §9 · p. 8

10. Conclusion and future work

Barlow uses robust-model features and visualizations to identify failure modes and provide practitioners with insights for debugging. The conclusion reports evidence that practitioners can interpret these features and identify significant error clusters, while causal analysis and richer tools remain future directions.

10. Conclusion and future work

  • Human studies provide evidence that practitioners can interpret features and identify significant error clusters, including those associated with systematic spurious correlations.The authors see the analyses and visualizations as promising for iterative improvement without expensive manual metadata collection.Source: §10 · p. 8

A. Feature extraction

  • Feature extraction uses activations from the penultimate layer of an adversarially trained ResNet-50 with an l2 threat radius of 3.Each benchmark image is run through the model at inference time; the layer is also the output of global average pooling.Source: A. Feature extraction · p. 11

B. Heatmap generation

  • Heatmaps are generated by selecting a feature map before global average pooling, normalizing it to 0–1, and resizing it to the image dimensions.Source: B. Heatmap generation · p. 11

C. Feature attack

  • Feature attacks optimize an image to maximize a selected feature’s value, with epsilon controlling the allowed image change.The reported optimization uses gradient ascent with step size 1, epsilon 500, and 500 iterations.Source: C. Feature attack · p. 11

D. Failure mode generation

  • Failure-mode generation accepts any image cluster, selects features by mutual information with error, and trains a decision tree to find leaves with concentrated errors.Experiments group images by label or model prediction, but practitioners may define clusters differently.Source: D. Failure mode generation · p. 11

E. Automatic evaluation of decision tree

The evaluation compares decision-tree error analysis across tree depths, model robustness, and instance-grouping strategies. Ground-truth-label grouping and robust-model features yield better ALER−BER scores than their respective alternatives, even when baseline error rates are similar.

E. Automatic evaluation of decision tree

F. Failure modes discovered by Barlow

  • Failure-mode examples analyze standard and robust ResNet-50 models using robust-model features, ImageNet training images, and depth-one decision trees.The highest-importance leaf is visualized; at depth one, a single feature defines that leaf.Source: F. Failure modes discovered by Barlow · p. 14

F.1.1 Grouping by label

F.1.2 Grouping by prediction

F.2.1 Grouping by label

F.2.2 Grouping by prediction

  • Figure 25 visualizes feature[544] for images with label tiger cat; when feature[544] < 0.2036, error rate increases to 0.8754 (+37.85%).Source: Figure 25 · p. 17

G. Examples from Crowd study

The crowd study assesses how workers describe robust-model feature visualizations, while a practitioner study and qualitative model comparison address their use in interpretation and debugging. Worker agreement is higher for longer descriptions and for visualizations with fewer, salient objects; the authors report robust-model visualizations as more interpretable qualitatively.

G. Examples from Crowd study

  • Worker-description agreement increases with description length and is at least 0.45 when visualizations contain fewer objects and salient objects.Agreement uses pairwise average cosine similarity of Word2Vec description vectors, capturing shared themes across different wording.Source: G. Examples from Crowd study · pp. 39–40

G.1. Easy examples (Table 6) / G.2. Difficult examples (Table 7)

  • Tables 6 and 7 contrast visualizations workers found easy versus difficult to describe, with example descriptions for each.Source: Table 6 · p. 40, Table 7 · p. 43

G.3. Examples with most votes for Section C, Feature Attacks (Question 5 in Figure 65) / G.4. Examples with most votes for Both (Question 5 in Figure 65)

H. User study with ML practitioners

  • Tables 8 and 9 describe practitioner-study participants and the first class groupings they received; five examples were selected as easy to describe for onboarding.The second class grouping was randomly assigned from 120 groupings in the MTurk study.Source: Table 8 · p. 51, Table 9 · p. 51

I. Comparison between the interpretations of a robust and non-robust model

  • Robust-model feature visualizations were qualitatively more interpretable than non-robust-model visualizations, though the comparison had no quantitative human-study evaluation.The appendix illustrates paired visualizations for water jug, horned viper, tiger cat, tape player, and overskirt.Source: §I · p. 51, Figure 88 · p. 52, Figure 89 · p. 52
Loading 2012.01750v3…