PaperlayerView exact revision on arXiv

Source-linked AI summary

Salient ImageNet: How to discover spurious features in Deep Learning?

Sahil Singla, Soheil Feizi

arXiv:2110.04301v4cs.LGcs.CV

TL;DR

The paper addresses the difficulty of discovering spurious visual features without extensive human annotation or model-specific procedures. It uses minimally supervised labels for robust-model penultimate-layer neural features, generalizes them to many images, and builds Salient ImageNet with core and spurious masks. Evaluations show that several standard ImageNet models rely on spurious regions, so standard accuracy alone does not fully assess their behavior.

  • Problem

    Spurious features can undermine deep-model reliability, while existing discovery methods require extensive human annotation or apply only to specific models.

  • Method

    The method labels robust-model penultimate-layer neural features as core or spurious from a few activating images, generalizes the labels automatically, and uses activation maps as soft masks.

  • Results

    Several standard ImageNet models show similar accuracy drops when spurious or core regions are corrupted, indicating reliance on both types of regions; the resulting Salient ImageNet dataset contains 52,521 images.

  • Takeaways & Limitations

    Salient ImageNet supports model debugging and evaluation of sensitivity to core and spurious visual attributes using annotated masks.

  • Takeaways & Limitations

    The defined core accuracy is incomplete because the discovered spurious attributes may not cover all spurious attributes in the dataset.

Abstract

from arXiv · show

Deep neural networks can be unreliable in the real world especially when they heavily use {\it spurious} features for their predictions. Focusing on image classifications, we define {\it core features} as the set of visual features that are always a part of the object definition while {\it spurious features} are the ones that are likely to {\it co-occur} with the object but not a part of it (e.g., attribute "fingers" for class "band aid"). Traditional methods for discovering spurious features either require extensive human annotations (thus, not scalable), or are useful on specific models. In this work, we introduce a {\it general} framework to discover a subset of spurious and core visual features used in inferences of a general model and localize them on a large number of images with minimal human supervision. Our methodology is based on this key idea: to identify spurious or core \textit{visual features} used in model predictions, we identify spurious or core \textit{neural features} (penultimate layer neurons of a robust model) via limited human supervision (e.g., using top 5 activating images per feature). We then show that these neural feature annotations {\it generalize} extremely well to many more images {\it without} any human supervision. We use the activation maps for these neural features as the soft masks to highlight spurious or core visual features. Using this methodology, we introduce the {\it Salient Imagenet} dataset containing core and spurious masks for a large set of samples from Imagenet. Using this dataset, we show that several popular Imagenet models rely heavily on various spurious features in their predictions, indicating the standard accuracy alone is not sufficient to fully assess model performance. Code and dataset for reproducing all experiments in the paper is available at \url{https://github.com/singlasahil14/salient_imagenet}.

1 INTRODUCTION

The paper presents a scalable framework that uses limited human supervision to discover and localize core and spurious features through neural features. Applied to ImageNet, it produces Salient ImageNet and reveals that standard models often rely on spurious regions.

  • Dataset: The study constructs Salient ImageNet from 52,521 images across 232 ImageNet classes, with approximately 226 images per class on average.Workers identified 160 spurious features among 1,160 annotated neural features, with at least one discovered for 93 classes.
  • Dataset: Salient ImageNet contains core and spurious masks generated from neural activation maps, enabling feature-specific model evaluation through image corruption.Each instance includes an image, ground-truth label, and core and spurious masks.
  • Results: Across ResNet-50, Wide-ResNet-50-2, EfficientNet-B4, and EfficientNet-B7, spurious-region and core-region corruptions produced no significant difference in accuracy drops.For the drake class, however, spurious-feature corruption caused a drop greater than 40%, compared with less than 5% for core-feature corruption.
  • Contribution: The work claims a general, minimally supervised method for discovering core and spurious features and releases an ImageNet version annotated with both types of masks.The authors position the framework as a first general methodology using robust-model penultimate-layer neurons as interpretable visual feature detectors.

2 RELATED WORK

Prior approaches to failure explanation and spurious-feature discovery often depend on domain-specific interpretable features, crowdsourced attributes, simulations, or manually proposed hypotheses. Neural features from robust models offer an alternative, but earlier work had important applicability constraints.

  • Interpretability: Post-hoc interpretability research has largely focused on local explanations that inspect individual image decisions.The related work includes saliency maps and other techniques for explaining single-image predictions.
  • Spurious-feature discovery: Earlier spurious-feature methods required humans to propose candidate features and label visual attributes, making them difficult to scale to datasets such as ImageNet.Some approaches also required constructing new datasets with unusual visual attributes to test hypotheses.
  • Failure explanation: Existing failure-explanation methods use different sources of interpretability, including tabular features, textual queries, crowdsourced image attributes, and photorealistic simulation.These approaches are tied to the structure or resources available in each domain.
  • Neural feature detectors: Recent methods use neurons of robust models as visual-attribute detectors to avoid crowdsourcing, but one cited approach cannot analyze standard non-robust models.The cited work also asks workers to identify common visual patterns, retaining a human-supervision burden.

3 A GENERAL FRAMEWORK FOR DISCOVERING SPURIOUS FEATURES

The framework treats penultimate-layer neurons of a robust model as visual feature detectors, selects class-relevant neurons, and uses limited crowd judgments to identify core or spurious attributes. Their activation maps then localize those attributes across many images.

  • 3.1 NOTATION AND DEFINITIONS: Core attributes are always part of an object definition, whereas spurious attributes co-occur with the object without being part of it.The framework seeks to identify and localize these attributes in image-classification training data.
  • 3.1 NOTATION AND DEFINITIONS: A neural feature is one penultimate-layer activation, and its Neural Activation Map provides a soft mask for highly activating image pixels.Heatmaps overlay these maps to show where the feature activates.
  • 3.1 NOTATION AND DEFINITIONS: The method visualizes five maximally activating images, uses heatmaps to disambiguate spatial focus, and applies feature attacks when heatmaps cover multiple attributes.In the cited example, the feature attack distinguishes wires from the cougar when the heatmap highlights both.
  • 3.2 EXTRACTING, VISUALIZING AND SELECTING NEURAL FEATURES: For class i and feature j, Neural Feature Importance is IV_i,j = (r(i) ⊙ w_i,:)j, and the five highest-importance features are selected per class.The mean neural-feature vector r(i) is combined elementwise with the class-logit weight row w_i,:.
  • 3.3 MECHANICAL TURK STUDY FOR DISCOVERING SPURIOUS AND CORE FEATURES: Mechanical Turk workers classify each selected feature as core or spurious using top-5 images, heatmaps, feature attacks, and class descriptions, with majority responses determining the label.The study covered 232 classes and 1,160 HITs; 160 features were judged spurious.
  • 3.4 GENERALIZATION: The annotations generalize from five inspected images to top-k images with k much larger than five, allowing automatic localization of the corresponding visual attribute.The resulting image sets are validated by comparing heatmaps on highly and weakly activating examples.

4 THE SALIENT IMAGENET DATASET

Salient Imagenet packages neural-feature-derived soft masks for core and spurious attributes across a large ImageNet sample, enabling systematic analysis of model sensitivity.

  • Figure 4 summarizes class coverage and the distribution of images and discovered spurious features across the dataset.Its top row counts images for classes with at least one spurious feature, while the bottom row presents additional dataset plots.
  • 52,521 images with about 226 images per class form the Salient Imagenet dataset, whose instances include labels and core/spurious masks.The dataset is constructed from per-class, per-feature image sets and neural activation maps used as soft segmentation masks.
  • Each dataset instance stores an image, ground-truth label, and sets of spurious and core masks derived from neural activation maps.These masks support sensitivity tests by selectively corrupting image regions associated with the corresponding attributes.
  • The dataset contains 160 spurious and 1,000 core features, with at least one spurious feature discovered for 93 of 232 evaluated classes.Three classes—space bar, miniskirt, and seatbelt—had all five examined features classified as spurious.

5 EVALUATING DEEP MODELS USING THE SALIENT IMAGENET DATASET

The evaluation corrupts core or spurious regions with Gaussian noise and measures accuracy changes, revealing that standard models often depend strongly on spurious regions and fail to distinguish feature types.

  • The evaluation uses soft masks with Gaussian noise so corruption preserves image content while varying in strength through σ.Mask values near one receive noise with the target standard deviation, while values near zero are changed little.
  • More than 40% accuracy drops for spurious-region corruption versus less than 5% for core-region corruption show strong spurious reliance for drake.For ResNet-50, the reported clean accuracy is 95.4%, with σ = 0.25 Gaussian noise used in the example.
  • Core accuracy is intended to remain high when spurious regions are corrupted, whereas spurious-region accuracy should fall when core regions are corrupted.Core accuracy is computed after aggregating relevant spurious masks; spurious accuracy is defined analogously using core masks.
  • For triumphal arch, core accuracy remains about 40% at σ = 2.0, whereas for ostrich it is almost 0%, despite both having 100% standard accuracy.These class-level curves illustrate that standard accuracy can conceal different dependence on core features.
  • Across four standard pretrained models, accuracy drops similarly when noise is added to spurious or core regions, suggesting they do not differentiate between them.The comparison uses ResNet-50, Wide ResNet-50-2, EfficientNet-b4, and EfficientNet-b7 across noise levels σ.

ETHICS STATEMENT

The paper presents a scalable framework and dataset for discovering spurious features, while reporting no foreseeable negative consequences. Its implementation uses neural activation maps and feature attacks to visualize features and selects ImageNet classes to reduce Mechanical Turk costs.

  • The authors report no foreseeable negative consequences and introduce Salient ImageNet without collecting new samples.
  • Neural activation maps are normalized and resized to image dimensions to highlight feature-responsive regions.
  • Heatmaps overlay jet-colored neural activation maps on the original image to visualize feature importance.
  • Feature attacks optimize images to maximize a selected feature using gradient ascent with step size 40, 25 iterations, and ρ = 500.
  • To reduce Mechanical Turk expense, the study selects a smaller ImageNet class subset using pretrained-model label and prediction groupings.

C MECHANICAL TURK STUDY FOR DISCOVERING SPURIOUS FEATURES

The Mechanical Turk study asks workers to classify neural features as parts of the main object, separate objects, or background. Workers inspect top activations, heatmaps, feature attacks, and object information before providing reasons and confidence ratings.

  • Each study item paired a robust ResNet-50 neuron visualization with an ImageNet object-class description.
  • Neuron visualizations included five highest-activation images, heatmaps, and amplified feature-attack visualizations.
  • Class descriptions included the WordNet category, supercategory, object definition, relevant Wikipedia links, and three validation images.
  • Workers judged whether each visual attribute belonged to the main object, a separate object, or its background, with majority background or separate-object judgments indicating spuriousness.
  • The study restricted participation to workers with at least 95% approval and 1,000 completed HITs, rejecting short generic explanations.

D MECHANICAL TURK STUDY FOR VALIDATING HEATMAPS

A second Mechanical Turk study validates whether neural activation maps consistently focus on the same visual attribute across images. Workers compare heatmaps from highly and weakly activating examples and mark the feature as validated when most select “same.”

  • The study’s stated goal was to test whether heatmaps focused on the same visual attribute throughout the subset.
  • The validation subset was built for each discovered spurious feature by selecting images labeled with its object class and then choosing the top-65 activations.
  • Workers compared five highest-activation images and five lowest-activation images, each paired with heatmaps of the same neuron.
  • The validation questionnaire offered same, different, or unclear judgments for the two visualization sections.
  • A feature was validated when most workers judged its heatmaps to focus on the same visual attribute across the 65-image subset.

E DISCOVERING SPURIOUS VISUAL ATTRIBUTES IN OTHER IMAGE DATASETS

The framework transfers to other image-classification datasets by combining robust models, selected penultimate-layer neural features, human annotation, and heatmap-based localization. It defines core and spurious attributes through object definitions and recommends validation on new samples.

  • DISCOVERING SPURIOUS VISUAL ATTRIBUTES IN OTHER IMAGE DATASETS: The proposed methodology is intended to discover core and spurious features in image-classification datasets beyond ImageNet.
  • DISCOVERING SPURIOUS VISUAL ATTRIBUTES IN OTHER IMAGE DATASETS: The workflow trains a robust model, selects highly important penultimate-layer neural features, visualizes them, and obtains Mechanical Turk annotations for each class-feature pair.
  • DISCOVERING SPURIOUS VISUAL ATTRIBUTES IN OTHER IMAGE DATASETS: The method uses k = 5 neural features and m = 5 highly activating images, while allowing both values to vary by application.
  • DISCOVERING SPURIOUS VISUAL ATTRIBUTES IN OTHER IMAGE DATASETS: After workers identify spurious neural features, heatmap methods can automatically produce core and spurious masks for many additional samples, followed by recommended validation.
  • FORMAL DEFINITIONS OF CORE AND SPURIOUS VISUAL ATTRIBUTES: Core attributes are always part of an object’s definition, whereas spurious attributes co-occur with it without being part of that definition.
  • FORMAL DEFINITIONS OF CORE AND SPURIOUS VISUAL ATTRIBUTES: The formal graphical model represents label, core, and spurious variables as conditionally independent given the core variable.

H ADDITIONAL RESULTS FROM EVALUATING DEEP MODELS ON THE SALIENT IMAGENET DATASET

Core accuracy reveals how model performance changes when spurious regions are corrupted, while sensitivity varies across classes as corruption increases.

  • Core accuracy is compared with usual accuracy across different ImageNet-trained models using σ = 0.25.
  • Core accuracy generally decreases as noise level σ increases, although some classes remain robust at larger σ.Triumphal arch and jellyfish on Efficientnet-b7 are cited as examples of differing behavior.
  • The class-level curves show that robustness to corruption is not uniform across ImageNet categories.

I EXAMPLES OF IMAGES FROM THE SALIENT IMAGENET DATASET

The dataset illustrates spurious-region corruption and provides sampled image groups organized by class and neural-feature index.

  • The examples depict images corrupted by adding Gaussian noise to highlighted spurious regions.
  • Randomly sampled image sets are shown for space bar, lighter, lakeside, soup bowl, missile, and dock classes.
  • Each sampled set is indexed by a class identifier and feature identifier, linking the examples to particular neural features.

J.1 BACKGROUND SPURIOUS FEATURES

Across many classes, corrupting highlighted regions produces varied accuracy drops, with some features causing substantially larger declines than others.

  • Feature visualizations report training accuracy and the accuracy drop after Gaussian noise is added to highlighted regions across many ImageNet classes.
  • Other examples show substantial but smaller drops, including -32.308% for rock beauty, -35.385% for dock, and -36.923% for space bar.
  • The largest reported drops include -46.154% for wok, -46.154% for space bar, and -46.154% for soup bowl.
  • Some highlighted features have minimal effects, such as -1.538% for bullet train and gondola.
  • The examples span classes with training accuracies ranging from 59.043% for space bar to 99.385% for sorrel.

L.1 CLASS NAME: OSTRICH, TRAIN ACCURACY (STANDARD RESNET-50): 98.615%

Across the displayed classes, corrupting annotated spurious features generally causes larger accuracy drops than corrupting annotated core features, especially at higher noise levels.

  • Spurious features consistently show larger drops than paired core features across ostrich, brambling, house finch, bulbul, coucal, jacamar, and drake.
  • At higher noise levels, spurious-feature drops reach -93.846% for brambling, -92.308% for coucal, and -86.154% for drake.
  • Core features can also become influential at larger σ, with drops of -60.0% for house finch and -41.538% for coucal.
  • For drake, spurious feature 341 causes a -46.154% accuracy drop at σ = 0.25, whereas core feature 736 causes a 0.0% drop.
  • For oystercatcher, ringlet, lycaenid, and proboscis monkey, the annotated spurious feature likewise produces a larger drop than the corresponding core feature.
Loading 2110.04301v4…