PaperlayerView exact revision on arXiv

Source-linked AI summary

Distilling Model Failures as Directions in Latent Space

Saachi Jain, Hannah Lawrence, Ankur Moitra, Aleksander Madry

arXiv:2206.14754v2cs.LG

TL;DR

Identifying hard subpopulations and spurious correlations can require labor-intensive human exploration, and automatic methods may not capture interpretable error patterns. This paper learns latent-space directions for model failures and uses them to find and caption challenging groups and guide interventions; on CelebA, most flagged errors came from challenging groups that formed a minority of the dataset.

  • Problem

    Identifying biases and difficult subpopulations often requires labor-intensive human intervention, while automatic methods may not capture simple, human-understandable error patterns.

    Source: §1 · p. 1

  • Method

    Per-class linear SVMs predict model mistakes from embeddings, defining failure directions that support detection, captioning, and targeted intervention.

    Source: §2 · p. 2, Our Contributions · p. 2

  • Results

    On CelebA, 89% of images flagged as incorrect came from challenging groups that made up 33% of the dataset, and the SVM more consistently selected those groups than model confidences.

    Source: §3.1 · p. 5

  • Takeaways & Limitations

    The framework identifies and supports intervention on hard subpopulations in image datasets without direct human intervention or pre-annotated groups.

    Source: Our Contributions · p. 2

    The method identifies only the most prominent failure mode for a class; disentangling multiple important modes remains future work.

    Source: §6 · p. 11

Abstract

from arXiv · show

Existing methods for isolating hard subpopulations and spurious correlations in datasets often require human intervention. This can make these methods labor-intensive and dataset-specific. To address these shortcomings, we present a scalable method for automatically distilling a model's failure modes. Specifically, we harness linear classifiers to identify consistent error patterns, and, in turn, induce a natural representation of these failure modes as directions within the feature space. We demonstrate that this framework allows us to discover and automatically caption challenging subpopulations within the training dataset. Moreover, by combining our framework with off-the-shelf diffusion models, we can generate images that are especially challenging for the analyzed model, and thus can be used to perform synthetic data augmentation that helps remedy the model's failure modes. Code available at https://github.com/MadryLab/failure-directions

1 Introduction

Identifying dataset biases and difficult subpopulations often requires labor-intensive human exploration, while existing automatic approaches may not reveal consistent, interpretable error patterns. The paper introduces a scalable framework that represents model failures as latent-space directions and uses them to detect, interpret, and intervene on hard subpopulations.

1 Introduction

  • Existing identification methods often require labor-intensive human exploration, while automatic methods may not capture simple, human-understandable patterns.Feature-attribution methods can help debug individual examples but do not provide a global understanding of dataset biases without examining many attributions.Source: §1 · p. 1

Our Contributions

  • Linear classifiers on normalized feature embeddings identify consistent prediction mistakes, with their decision boundaries defining directions of hard examples.Alignment with a direction measures an example’s relevance to the failure mode.Source: Our Contributions · p. 2
  • The framework detects and quantifies reliability failures, assigns meaningful captions to error patterns, and supports interventions aimed at improving reliability along failure axes.Source: Our Contributions · p. 2
  • It automatically identifies and enables intervention on hard subpopulations in datasets including CIFAR-10, ImageNet, and ChestX-ray14, without direct human intervention or pre-annotated groups.The authors frame this as a scalable way to identify important subpopulations relative to downstream tasks.Source: Our Contributions · p. 2

2 Capturing Failure Modes as Directions Within a Latent Space

The framework learns failure directions by training per-class linear classifiers to separate correct from incorrect predictions in an embedding space. It uses these directions to measure failure strength, surface and caption hard subpopulations, and guide data interventions.

2 Capturing Failure Modes as Directions Within a Latent Space

  • Per-class linear SVMs trained on held-out validation examples predict the base model’s mistakes, and each SVM’s hyperplane normal defines a failure direction.The method uses shared vision-language embeddings; the CelebA example depicts gender separating correct and incorrect predictions for the “old” class.Source: §2 · p. 2, Figure 1 · p. 3

2.1 Detection

  • SVM cross-validation accuracy measures failure-mode strength because consistent errors are more easily separated in the embedding space.The score can identify classes with a clear bias for subsequent interpretation and intervention.Source: §2.1 · p. 3

2.2 Interpretation

  • For interpretation, extreme SVM decision values surface prototypically correct or incorrect examples, while aligned captions identify attributes associated with the failure direction.Caption scoring contrasts each candidate’s embedding with a generic class-reference caption, directly matching the failure direction rather than averaging hard examples.Source: §2.2 · p. 4
  • When diffusion models can sample directly from CLIP space, spherical interpolation between a reference-caption embedding and the SVM direction can generate harder or easier images.The rotation amount is expected to determine image difficulty.Source: §2.2 · p. 4

2.3 Intervention

  • Interventions select low-decision-value examples from an external pool or use generated images from automatically captioned failure modes; the SVM more reliably captures CelebA’s spurious correlation than the listed baselines.Fine-tuning on synthetic images is reported to improve reliability on the corresponding subpopulations.Source: §2.3 · pp. 4–5, Figure 2 · p. 5

3 Validating Our Framework in the Presence of Known Subpopulations

The experiments validate the framework on known spurious-correlation and underrepresented-subtype settings. Across CelebA and CIFAR-100, it isolates challenging groups, makes them interpretable, and supports targeted data selection.

3.1 Using Our Framework to Isolate and Label Spurious Correlations

  • Although challenging age-gender groups make up 33% of the dataset, 89% of images flagged as incorrect come from those groups.The SVM more consistently selects these hard subpopulations than model confidences and outperforms Domino, Learning from Failure, and early-stopped confidences.Source: §3.1 · pp. 5–6
  • Extreme SVM decision values surface the hard CelebA groups, and captions aligned with the extracted direction capture the gender correlation.The most negative examples are from hard subpopulations, while the most positive are from majority subpopulations.Source: §3.1 · p. 6, Figure 3 · p. 6
  • SVM cross-validation scores strongly correlate with the planted spurious-correlation strength, making them a measure of shift strength.The experiment varies the degree of the planted correlation in CelebA training data.Source: §3.1 · p. 6, Figure 4 · p. 6

3.2 Identifying Underrepresented Sub-types

  • The minority-subclass caption ranks as the top negative caption for 80% of CIFAR-100 superclasses and among the top two for 95%.For each superclass, candidate captions correspond to its possible subclasses.Source: §3.2 · p. 6
  • On CIFAR-100, the framework isolates the underrepresented subclass as most incorrect more consistently than model confidences.Figure 5 compares the minority-subclass share among the top K flagged test images selected by SVM decision value or model confidence.Source: §3.2 · p. 6, Figure 5 · p. 7
  • SVM-based selection of extra CIFAR-100 images improves minority-subclass accuracy more than confidence-based or random selection while maintaining approximately the same overall accuracy.The comparison adds K images per superclass from an external pool; Figure 6 reports minority-subclass accuracy over five runs.Source: Figure 6 · p. 7, §3.2 · p. 7

4 Discovering New Subpopulations in Image Datasets

The framework is applied to CIFAR-10, ImageNet, and ChestX-ray14 without pre-annotated difficult groups to discover model failure subpopulations. CIFAR-10 experiments also test diffusion-generated examples and targeted augmentation.

4.1 CIFAR-10

  • On CIFAR-10, white cats on grass and black dogs are hard; negative-caption neighbors have lower accuracy than positive-caption neighbors, validating the captions as failure modes.The evaluation ranks test images by CLIP-space similarity to SVM-selected captions because minority-group annotations are unavailable.Source: §4.1 · p. 7, Figure 7 · p. 8
  • Spherical interpolation between a reference caption embedding and the SVM direction generates hard or easy images, on which the original model performs worst or best, respectively.The rotated embedding is passed to a diffusion model.Source: §4.1 · pp. 7–8, Figure 8 · p. 8
  • Fine-tuning on 100 synthetic images per class prompted by negative SVM captions improves accuracy on corresponding hard CLIP-nearest test images more than generic images; easy-caption augmentation does not.The hard subpopulation is defined by similarity to the negative caption in CLIP space.Source: §4.1 · pp. 8–9, Figure 9 · p. 9

4.2 ImageNet

  • On ImageNet, SVM captions expose color and co-occurring-object biases, and CLIP-nearest negative-caption images have lower accuracy than positive-caption images.Examples include red wolves and tench; the identified subpopulations have a significant difference in overall accuracy.Source: Figure 10 · p. 9, §4.2 · p. 9

4.3 ChestXray-14

  • For no-effusion X-rays, the SVM flags blurrier, less bright images and many AP radiographs, selecting AP examples more consistently than model confidence.This suggests the model might rely on radiograph position when predicting whether a patient is healthy.Source: §4.3 · p. 10, Figure 13 · p. 10

5 Related Work

Related work covers mitigation of known biases, discovery of hard subpopulations, and model interpretability. The paper distinguishes its framework by targeting consistent failure modes and using latent-space geometry to score examples, intervene, and select captions.

5 Related Work

  • Fairness and debiasing work addresses known biases through group-fairness optimization or training-time mitigation.The cited examples include [FFM+15] [HPP+16] [ZVR+17] and [SKH+20] [HZW19] [BPS+19] [CYZ19].Source: §5 · p. 10
  • Hard-subpopulation methods include manual dataset exploration, clustering, reweighting, and second-model approaches, but these generally target overall test accuracy rather than consistent failure modes.The paper identifies Eyuboglu et al. [EVS+22] as a close approach using a Gaussian mixture model in CLIP space; its SVM offers a stronger bias toward simple subclasses and useful geometry.Source: §5 · p. 11
  • Interpretability research includes attribution, activation analysis, and methods examining model vulnerabilities, while Abid et al. [AYZ22] use concept activation vectors to find latent perturbations that correct individual mistakes.Attribution methods can explain features relevant to predictions.Source: §5 · p. 11

6 Conclusion

The framework distills model failures into latent-space directions to identify, interpret, and address hard subpopulations. Its methods use linear SVMs to diagnose error patterns, surface examples and captions, and guide interventions.

6 Conclusion

  • The authors identify disentangling multiple failure modes within a class and improving caption candidate generation as directions for future work.The method currently identifies only the most prominent failure mode for a class.Source: §6 · p. 11

A.1 Formal Description of Method

  • Cross-validation balanced accuracy measures failure-mode strength, while interpretable examples and captions use the SVM’s simple patterns and accessible decision boundary.On CelebA, balanced accuracy rises as the planted spurious correlation strengthens.Source: §A.1 · p. 17
  • For each class, extreme SVM-scored test images summarize hard and easy examples; captions are selected by how strongly their CLIP-space directions align with the error direction.The SVM decision value provides a measure of input hardness through distance to its decision boundary.Source: §A.1 · pp. 17–18
  • Intervention retrains the model using flagged hard training examples, either by upweighting them or adding selected examples from an external pool.The flagged set contains training examples whose per-class SVM decision values are negative.Source: §A.1 · p. 18

A.2 Model Training Details

  • The experiments train ResNet18 models with dataset-specific settings, using SGD with momentum and weight decay and a triangular cyclic learning rate.ChestX-ray14 uses binary cross-entropy; other datasets use softmax classification with cross-entropy.Source: §A.2 · p. 18

A.3 SVM Training Details

  • SVMs are trained on whitened, normalized CLIP embeddings from held-out validation data to predict model correctness, with regularization selected by two-fold cross-validation.Balanced class weighting is used, and cross-validation balanced accuracy also quantifies the captured shift.Source: §A.3 · p. 18

A.4 Filtering Details

  • For external-pool filtering, the method adds images with the most negative SVM decision values and compares selection by confidence and at random.The described overall-accuracy comparison adds 100 images per superclass.Source: §A.4 · p. 19

A.5 Caption Generation

  • Caption candidates are generated from optional adjective and prepositional-phrase templates to keep them descriptive and relevant to the image class.Automatically curating candidates, for example with a language model, is left for future work.Source: §A.5 · p. 19

A.7 Tailored Data Augmentation using Text-To-Image generative Models

The appendix evaluates text-to-image augmentation and CelebA interventions and stress tests. It reports preserved overall accuracy after fine-tuning, minority-subpopulation gains from SVM-guided upweighting, and continued detection of gender correlation under a training-matched validation distribution.

A.7 Tailored Data Augmentation using Text-To-Image generative Models

  • Overall test accuracies were maintained up to 1.3% after fine-tuning on the original training data plus 100 synthetic images per class.The synthetic images were generated from easy, hard, and reference captions using Stable Diffusion.Source: §A.7 · p. 20

B.1.1 Experimental Details

  • The CelebA setup intensifies the age–gender correlation by including four times as many old men as old women and young women as young men.The held-out validation set is balanced across the four demographic categories in this setup.Source: §B.1.1 · p. 21

B.1.2 Upweighting Intervention

  • SVM-guided upweighting improves accuracy on minority CelebA subpopulations, with impact described as nearly matching oracle upweighting.The table reports mean and standard deviation over five runs; the base model’s near-zero training error makes its confidences unsuitable for an analogous intervention.Source: §B.1.2 · pp. 21–22, Table 1 · p. 22

B.1.3 Gradations of spuriousness

  • As the planted age–gender correlation strengthens, base-model test accuracy decreases, while the SVMs fit test data better.The experiment varies the correlation intensity from n = 1 to 8.Source: §B.1.3 · p. 22, Figure 14 · p. 22

B.1.4 CelebA experiments when the validation matches the training distribution.

  • With validation data matching the training distribution, the framework still captures gender correlation and isolates the hard subpopulation better than model confidences.This setting has fewer minority examples available to train the SVM, making error selection harder.Source: §B.1.4 · p. 23, Figure 15 · p. 23, Figure 16 · p. 23

B.1.5 Further experimental details on baselines

The appendix compares SVM-based failure directions with baselines, tests alternative embedding spaces, and extends experiments to unaltered CelebA and Colored MNIST. It also specifies the ImageNet-C corruption setting used in later results.

B.1.5 Further experimental details on baselines

  • The SVM more consistently selects CelebA’s hard minority-gender subpopulation than the compared baselines.Domino also captures gender, but its captions can be niche because it selects captions closest to each slice’s mean.Source: §B.1.5 · pp. 23–24, Figure 17 · p. 24

B.1.6 Using Other Latent Spaces

  • Inception embeddings perform only slightly worse than CLIP, while the model’s original latent space mirrors confidences and cannot support upweighting in this setting.CLIP also permits captioning because it embeds images and language in a shared space.Source: §B.1.6 · p. 25, Figure 18 · p. 25

B.1.7 Full CelebA Dataset

  • On unaltered CelebA, per-class SVMs capture the age–gender spurious correlation more strongly than model confidences.The experiment uses the standard CelebA validation set; Table 2 describes the correlation and Figures 19a–b show the comparison.Source: §B.1.7 · p. 26, Figure 19 · p. 26, Table 2 · p. 26

B.2 Colored MNIST

  • As the tint–label correlation strengthens in Colored MNIST, SVM fit improves, and doubling the weight of flagged hard examples increasingly improves test accuracy.Training tint correlates perfectly with the label for 90% of examples, versus 50% of validation and test examples.Source: §B.2 · p. 27, Figure 20 · p. 27, Figure 21 · p. 27

B.3.1 Experimental Details

  • The ImageNet-C experiment uses Gaussian pixel noise with variance 0.26, corrupting 10% of training data and 50% of validation and test data.A ResNet18 achieves 51.1% validation and 48.9% test accuracy, falling to 45.5% and 43.0% on corrupted images.Source: §B.3.1 · p. 28

B.3.2 Results

On ImageNet-C, per-class SVMs identify Gaussian-corrupted images as a shared hard pattern, although individual hard examples can be difficult for different reasons. The accompanying CIFAR-100 passage describes the dataset split and planted underrepresentation used for that experiment.

B.3.2 Results

  • Per-class SVMs capture Gaussian corruption even though the corruption is light and the hardest images are not necessarily corrupted.The figure compares the fraction of corrupted test images when ordered by SVM decision value versus model confidence.Source: §B.3.2 · p. 28, Figure 22 · p. 28, Figure 23 · p. 29
  • In ImageNet-C, the authors do not expect upweighting or subsetting to improve performance because corruption represents a hard class, not a spurious correlation or underrepresented subgroup.The experiment demonstrates capture of a shared pattern across errors despite there being multiple ways an image can be hard.Source: §B.3.2 · p. 29, Figure 23 · p. 29

B.4 CIFAR-100

  • The CIFAR-100 setup removes 75% of one worst-performing subclass per superclass and allocates the original training split as 20% validation, 40% training, and 40% extra data.The extra data is reserved for subset intervention.Source: §B.4 · p. 30

B.4.1 Results on 80% train, 20% validation

Across the described CIFAR-100 and CIFAR-10 experiments, SVM decision values identify hard subpopulations and support a filtering intervention that improves performance on minority subgroups. The appendix also details caption construction, examples, and a validation of caption-based subgroup selection.

B.4.1 Results on 80% train, 20% validation

  • With 80% of CIFAR-100 training data allocated to train and 20% to validation, SVM decision values more consistently identify top-K minority-subpopulation images than model confidences.As underrepresentation grows, mean CV score rises and is negatively correlated with minority-subpopulation accuracy; higher scores also align with a greater share of predicted errors belonging to the planted subgroup.Source: §B.4.1 · pp. 30–31, Figure 24 · p. 31, Figure 25 · p. 31

C.1 CIFAR-10

  • Most CIFAR-10 experiments use 20% of the original training split, 20% validation, and 60% extra data because the original model has 93% accuracy.The extra data is used for subset intervention.Source: §C.1.1 · p. 33

C.1.2 Caption Generation Details

  • CIFAR-10 captions combine a class reference with candidate adjectives, nouns, and prepositions; CIFAR-Simple keeps the class noun, while CIFAR-Extended expands nouns through WordNet.The main paper uses CIFAR-Simple because diffusion models struggle with the extended vocabulary.Source: §C.1.2 · pp. 33–34

C.1.3 Most Extreme Examples and Captions

  • The most extreme CIFAR-10 examples receive varied captions, including color, setting, and appearance descriptions.Examples include “a photo of a white cat on the grass” and “a photo of a blue automobile on a black background.”Source: §C.1.3 · p. 35

C.1.4 More examples of directly decoding SVM direction

C.1.5 Validating the Proxy of Evaluating Images Closest to the Captions

  • The caption-proximity proxy is evaluated by checking whether CIFAR-Extended images nearest a caption match its description.The method uses the K closest images to positive or negative SVM captions as a proxy for the corresponding subpopulation.Source: §C.1.5 · p. 37, Figure 30 · p. 37

C.1.6 Filtering Intervention for CIFAR-10

  • Selecting CIFAR-10 examples by SVM decision values improves performance on minority subgroups more than selection by confidence or at random.For each class, the intervention adds the top 100 examples selected by one of those methods; evaluation uses images closest to negative SVM captions as a proxy.Source: §C.1.6 · p. 38, Figure 31 · p. 38

C.1.7 Results for Full CIFAR-10 Dataset

On the full CIFAR-10 dataset, the framework finds similar failure modes and lower accuracy among identified negative than positive subpopulations. The appendix also gives ImageNet examples and normalizes CIFAR-10 and ImageNet plots by the fraction of each test set considered.

C.1.7 Results for Full CIFAR-10 Dataset

C.2 ImageNet

  • For ImageNet caption generation, classes are grouped by their closest selected WordNet ancestor to scale assignment of suitable prepositional phrases.The appendix shows additional surfaced examples for cauliflower and agama.Source: §C.2 · p. 40, §C.2.1 · p. 42, Figure 34 · p. 42

C.2.1 Further ImageNet Examples

C.2.2 ImageNet and CIFAR-10 normalized plots

  • The normalized CIFAR-10 and ImageNet plots use the fraction of the test set considered on the x-axis, reaching the entire test set at 100%.At that endpoint, the easy and hard subpopulations both have the overall test accuracy.Source: §C.2.2 · p. 42, Figure 35 · p. 42

C.2.3 Do failure directions generalize to other architectures?

ResNet18-derived failure directions transfer to other architectures, while an 80/20 ImageNet split without extra data yields behavior similar to Figure 10b.

C.2.3 Do failure directions generalize to other architectures?

C.2.4 Results on full Imagenet split

C.3.3 Results for Additional Conditions

Across Effusion, Mass, and Infiltration, the framework associates AP radiographs with different class-specific difficulty patterns; Mass also surfaces visual patterns not surfaced by confidence.

Effusion

  • For Effusion, the model associates AP images more strongly than confidence rankings do with hard healthy-class and easy non-healthy-class examples.Source: Figure 41 · p. 47, §C.3.3 · p. 47

Mass

  • For Mass, SVMs surface visual patterns that confidence misses and associate AP images with hard non-healthy examples, unlike Effusion’s healthy-class pattern.Source: §C.3.3 · pp. 47–48, Figure 42 · p. 48

Infiltration

  • For Infiltration, the framework associates AP images with easier non-healthy examples and, to a lesser extent, harder healthy examples.Source: Figure 43 · p. 49, §C.3.3 · p. 49
Loading 2206.14754v2…