PaperlayerView exact revision on arXiv

Source-linked AI summary

Discovering Bugs in Vision Models using Off-the-shelf Image Generation and Captioning

Olivia Wiles, Isabela Albuquerque, Sven Gowal

arXiv:2208.08831v2cs.CVcs.LGstat.ML

TL;DR

Finding vision-model failures on unseen data remains challenging, while existing tests can require new datasets or expert construction. This paper combines off-the-shelf image generation and captioning to search for interpretable failures, and reports that they generalize to real images and other architectures, with adversarial datasets generated at scale.

  • Problem

    Automatically finding vision-model failures on unseen data is challenging, while available approaches include collecting new datasets or creating expert-crafted synthetic tests.

    Source: §1 · p. 1

  • Method

    The approach generates class-conditioned images, clusters misclassified examples, captions failure clusters, and tests new images generated from those descriptions.

    Source: §1 · p. 1, §3 · p. 3

  • Results

    The discovered failures generalize across text-to-image models and occur in real Google Image Search images, while generated adversarial datasets generalize across classifier architectures.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The work is a proof of concept that large-scale generative models can support open-ended, automated discovery of interpretable bugs in machine-learning models.

    Source: §5 · pp. 7, 9

    The approach may miss failures in regions under-sampled by web-trained generators, and captioning constrains the search to failures describable in words.

    Source: §5 · p. 8

Abstract

from arXiv · show

Automatically discovering failures in vision models under real-world settings remains an open challenge. This work demonstrates how off-the-shelf, large-scale, image-to-text and text-to-image models, trained on vast amounts of data, can be leveraged to automatically find such failures. In essence, a conditional text-to-image generative model is used to generate large amounts of synthetic, yet realistic, inputs given a ground-truth label. Misclassified inputs are clustered and a captioning model is used to describe each cluster. Each cluster's description is used in turn to generate more inputs and assess whether specific clusters induce more failures than expected. We use this pipeline to demonstrate that we can effectively interrogate classifiers trained on ImageNet to find specific failure cases and discover spurious correlations. We also show that we can scale the approach to generate adversarial datasets targeting specific classifier architectures. This work serves as a proof-of-concept demonstrating the utility of large-scale generative models to automatically discover bugs in vision models in an open-ended manner. We also describe a number of limitations and pitfalls related to this approach.

1. Introduction

The paper addresses the limited availability of automatic tools for finding model failures on unseen data, and introduces an open-ended approach that uses off-the-shelf generative and captioning models. It reports interpretable failures that transfer across generators and to real images, and scales the approach to adversarial datasets.

1. Introduction

  • The paper motivates automated failure discovery by noting that existing tools are few, while dataset collection and expert-crafted synthetic tests can be demanding or limited in scope.It contrasts collected datasets with synthetic datasets designed to expose particular shortcomings.Source: §1 · p. 1
  • The pipeline uses text-to-image models to generate realistic images and captioning models to describe failure cases.The descriptions can be inspected and edited to investigate possible sources of failure.Source: §1 · pp. 1–2
  • The approach surfaces human-interpretable failures that generalize across text-to-image models and also occur in real images found through Google Image Search.Captions can be modified to test alternative hypotheses about why failures occur.Source: §1 · p. 2
  • The authors demonstrate scalability by generating adversarial datasets that align more closely with ImageNet’s training distribution than IMAGENET-A and generalize across classifier architectures.The work presents this as a contribution akin to IMAGENET-A [Hendrycks et al., 2019].Source: §1 · p. 2
  • The approach is presented as a proof of concept beyond ImageNet and vision, and treats the model under test as a black box.The authors describe combining off-the-shelf generative models to discover bugs and produce interpretable descriptions.Source: §1 · p. 2

2. Related Work

Prior approaches find and analyze model failures through collected test datasets, manually constructed tests, optimization, or interpretability techniques. The paper situates its approach among these methods and their costs or scope constraints.

2. Related Work

  • Prior work examines failures from shortcuts, data-collection changes, and contextual bias, including consistent mistakes across ImageNet models that may differ from human judgment.The cited studies cover spurious correlations, context, and model-to-model error patterns.Source: §2 · p. 2
  • Evaluation datasets can reveal performance under distribution shifts or targeted conditions, but constructing them is time-consuming and requires expert knowledge.Examples include natural adversarial, corruption, texture-shape conflict, and background-bias datasets.Source: §2 · p. 2
  • Some automated failure-discovery methods use constrained specifications, adversarial perturbations, or manually composed templates, while generative and perceptual approaches can be difficult to automate.Template-based methods rely on human creativity, and latent variables can be hard to relate to isolated input structures.Source: §2 · p. 2
  • A concurrent object-detection approach uses captioning and text-to-image models to create background images, but requires compositing and a background-image dataset rather than open-ended discovery.The comparison concerns Ge et al. (2022).Source: §2 · p. 2
  • Figure 2 depicts the paper’s method as comparing failures on generated images for a class with failures on images generated from descriptions of misclassified examples.The final description can be edited to investigate failure sources.Source: Figure 2 · p. 3
  • Interpretability methods may offer natural-language explanations using auxiliary attributes or embeddings, but often rely on additional datasets that limit their scope.The passage contrasts these with feature-level attacks and visual explanations such as LIME and Grad-CAM.Source: §2 · p. 3

3. Method

The method searches for descriptions whose generated images produce more classifier failures than images generated from a label alone. It estimates these rates with off-the-shelf text-to-image and captioning models, optionally clustering failures and refining captions.

3. Method

  • The search seeks a description z for images with label y whose misclassification rate exceeds the rate for images conditioned only on y.The paper also formulates a targeted-label variant that compares rates of prediction as a chosen wrong label.Source: §3 · p. 3

Generating failure cases

  • The pipeline first generates label-conditioned images to estimate a baseline failure rate, then describes misclassified images and generates more images from those descriptions to measure their failure rates.IMAGEN approximates image distributions and FLAMINGO approximates the caption distribution because the underlying distributions are unavailable.Source: §3 · pp. 3–4
  • The initial prompts ask for realistic photographs of the class, and the method counts only problematic errors whose top-three predictions are outside the true label’s WordNet parent category.The prompts are automatically constructed from the class and WordNet hierarchy.Source: §3 · p. 4

Optional clustering failure cases

  • Clustering is optional and can reduce the search’s computational burden by grouping failures by predicted label and similar feature representations.The stated goal is to avoid merging different causes of failure while keeping the number of clusters manageable.Source: §3 · p. 4

Failure case captioning

  • For each failure cluster, the method forms a caption from image-captioning outputs, then samples images from that caption and compares its failure rate with the baseline.Captions are assembled from sentences; the formulation assumes captions are equally likely under a label, an approximation the authors note may not generally hold.Source: §3 · p. 4

Caption refinement and counterfactual analysis

  • Caption refinement is optional: simple rewrite rules test shorter descriptions and users can probe alternative hypotheses; wrongful clustering may miss discoveries, although failures already found remain valid.The authors describe caption shortening as optional and intended to produce shorter failure descriptions.Source: §3 · pp. 4–5

4. Results

Open-ended search surfaced prompt-linked confusions for selected ImageNet classes, while scaled failure datasets transferred across models and were more ImageNet-like than ImageNet-A. The results also show failures on real Google Image Search photographs.

4.1. Open-ended failure search

  • Adding a green background made Persian-cat failures 5.72× more likely and snow-leopard predictions 14.3× more likely; flowers and nets raised fly→bee 497× and crayfish→chainlink-fence 3721×.The search used a TF-HUB RESNET-50 with three randomly selected labels and manually selected target labels.Source: §4.1 · pp. 5–6
  • With Google Image Search photographs, fly-to-bee failures rose from zero to two and crayfish-to-chainlink-fence failures from zero to four.The authors also report that the prompt-linked failures generalized across DALL·E 2 and Stable Diffusion images.Source: §4.1 · p. 6

4.2. Adversarial dataset generation at scale

  • IN-G-RN failures transferred to RN2 at 78%, and IN-G-VIT failures transferred to VIT2 at 77%; cross-architecture transfer was 39–53% and 50–61%, respectively.Figure 4 reports top-3 failure rates across models on the two generated datasets and ImageNet.Source: Figure 4 · p. 7, §4.2 · p. 7
  • Captioning 7,500 ImageNet-A images seeded datasets of 12,332 RESNET-50 failures and 9,536 VIT failures, with sampled images retained when they caused misclassification.The paper provides examples in which images visibly contain the correct class despite the model predicting another label.Source: §4.2 · pp. 6–7
  • Generated images were more similar to ImageNet than ImageNet-A under both FID and KID.Table 2 reports FID and KID scores in relation to the ImageNet test set.Source: Table 2 · p. 7, §4.2 · p. 7

5. Discussion

The discussion presents the pipeline as an automatic, open-ended way to find interpretable vision-model failures that can generalize to real images and other architectures. It also identifies limits in coverage, image-generation fidelity, captioning, and privacy.

5. Discussion

  • Because the generator approximates the image distribution and may lack coverage, the approach can demonstrate failures but cannot prove their absence; rare failures may go undiscovered.The authors note that meaningful probabilistic guarantees are possible if the generative model matches the true image distribution.Source: §5 · pp. 7–8
  • Generative-model bias can oversample high-density image regions, making failures there more likely to be found while failures in other regions may be missed.The authors suggest clever prompting or more active prompt discovery as possible ways to reduce this bias.Source: §5 · p. 8
  • Caption-based search is limited to failures describable in words, and captions can miss details or be ungrounded, so generated images may differ from those that caused the original failure.The crayfish-to-chainlink-fence example is obvious to a person but difficult to describe succinctly.Source: §5 · p. 8
  • Prompts beginning with “a realistic photograph” steer generation toward the intended distribution but do not always succeed; specialized domains may require fine-tuning.The authors give medical imagery as a case where finding a suitable prompt may not be obvious.Source: §5 · p. 8
  • Ambiguous prompts, including homonyms, and descriptions of multiple objects with complex relationships can lead to image-generation errors.The paper notes that subtle errors can occur even when prompts are unambiguous.Source: §5 · p. 8
  • Generating large amounts of data raises privacy risks; public, non-sensitive training data can mitigate them, but private generative modeling needs more research.Source: §5 · p. 8
  • The pipeline enables large-scale, open-ended investigations, with failures reported as human-interpretable and generalizing to other architectures and real images.The authors describe possible application to specialized tasks such as medical imaging as an encouraging prospect.Source: §5 · pp. 7, 9

A. Prompting the image-to-text model

  • Three publicly available images, manually captioned in four to seven short sentences each, serve as fixed examples for prompting the image-to-text model.The authors say the annotation cost is negligible compared with annotating a full test set.Source: A. Prompting the image-to-text model · p. 13

B.1. Open-ended failure search

The additional experiments report that failures found with generated data also occur on real images, across architectures, and in diverse forms. Tests of two open-ended cases find statistically significant differences between original and discovered captions.

B.1. Open-ended failure search

  • Illustrated RESNET-50 failures include misleading color patterns, spurious context, missing knowledge, and hallucinations.Figure 6 shows failure cases listed in Table 3, with correct labels on the left and incorrect predictions on the right.Source: Figure 6 · p. 14, §B.1 · p. 14
  • VIT-B/32 and VIT-B/8 show the same fly-on-a-flower-to-bee bias as RESNET-50, but the increase in failure rates is less pronounced.The reported increases are 114× and 13.5× for the VITs, compared with 497× for RESNET-50.Source: §B.1 · p. 16, Table 4 · p. 16
  • When the original failure rate is nonzero, discovered captions increase target failure rates on Google Image Search images across five randomly initialized RESNET-50s.The authors also report that some prompts had zero target failures for both original and modified captions, presumably because too few images were available.Source: Table 5 · p. 16, §B.1 · p. 16
  • Both tested open-ended cases show statistically significant higher target failure rates for images from discovered captions than from original captions.The reported p-values, 0.00015 and 6.34 · 10^-5, are below the 0.005 significance level.Source: §B.1 · pp. 16–17, Table 6 · p. 17

B.2. Additional comparisons on IN-G-RN, IN-G-VIT

Failures in the two generated datasets transfer across classifier architectures, but the tested ImageNet21K-pretrained VITs and BITs are fooled less often. Error patterns also tend to be similar within model classes.

B.2. Additional comparisons on IN-G-RN, IN-G-VIT

  • The error-consistency analysis combines IN-G-RN and IN-G-VIT and accounts for each model’s accuracy when comparing errors.The figure caption identifies this as error consistency across all models on the combined dataset.Source: §B.2 · p. 17, Figure 8 · p. 18
  • Failures from both IN-G-RN and IN-G-VIT transfer across model architectures, while ImageNet21K-pretrained VITs and BITs are fooled least often.Within model classes, larger models seem more robust; the authors conclude stronger pretraining and larger models seem to improve, but not complete, robustness.Source: Figure 7 · p. 17, §B.2 · p. 17
  • RESNETs trained similarly tend to make similar errors to one another, as do BITs, especially when their training methods match.Source: §B.2 · p. 18

B.3. Reliability of the text-to-image model

Across 2,000 generated images for 200 ImageNet-A labels, most images represented the requested class, with only one class systematically misrepresented.

B.3. Reliability of the text-to-image model

  • 3.95% of generated images did not represent the correct label, and 1.45% showed an item from the wrong WordNet family.The evaluation generated 10 images per class and counted only egregious classifier errors as wrong in the pipeline.Source: §B.3 · p. 19
  • Only porcupine was systematically misrepresented by the text-to-image model.The authors give an ocarina-to-maraca example of a wrong-WordNet-family image.Source: §B.3 · p. 19
  • The experiment indicates that the text-to-image model generally generates images of the requested class.Source: §B.3 · p. 19

C. Interpreting failure cases

The authors inspect nearest training-set neighbors of IN-G-VIT failures to identify visual cues associated with incorrect predictions, finding examples consistent with spurious correlations.

C. Interpreting failure cases

  • The analysis finds ten nearest ImageNet training-set neighbors for IN-G-VIT failures using cosine similarity in a ViT's second-to-last-layer embedding space.The examined ViT has 82.7% top-1 accuracy on ImageNet validation and a 100% failure rate on IN-G-VIT.Source: C. Interpreting failure cases · p. 19
  • Across the examined cases, neighbors from the predicted classes share background cues absent from same-label neighbors, consistent with the ViT relying on spurious features.The examples include human skin and grass, a net, and snow.Source: C. Interpreting failure cases · p. 19
  • The mushroom failure is classified as snail; its ten nearest neighbors are snail images with human-hand backgrounds, unlike the mushroom neighbors.Source: C. Interpreting failure cases · p. 19, Figure 9 · p. 20
  • The cabbage-butterfly and flagpole failures are classified as barn spider and ski, respectively, with neighbor sets sharing net and snow-and-blue-sky cues.Source: C. Interpreting failure cases · p. 19, Figure 10 · p. 21, Figure 11 · p. 22

D. Malicious usage and mitigation strategies

The authors discuss potential malicious use of generated failure cases and test training with those cases as a mitigation, while noting the discovery pipeline's computational cost.

D. Malicious usage and mitigation strategies

  • The authors note that malicious actors could use the approach to build adversarial images that bypass automated online filtering.Source: D. Malicious usage and mitigation… · p. 23
  • Finding a single failure case requires hundreds to thousands of calls to both the generative model and the vision classifier.The authors note that a rate-limited classifier API can act as a first line of defense.Source: D. Malicious usage and mitigation… · p. 23
  • Nearly 88% top-1 accuracy on the IN-G-VIT test set was achieved by training with IN-G-VIT data while minimally hurting ImageNet performance.Table 7 reports results for a ViT trained on ImageNet alone or on ImageNet plus IN-G-VIT; the experiment uses five random seeds for standard deviations.Source: Table 7 · p. 23, D. Malicious usage and mitigation… · p. 23

E. Additional visualizations

The additional visualizations show generated images and classifier failures, with labels indicating the true class and, for failure examples, incorrect top-3 predictions.

E. Additional visualizations

Loading 2208.08831v2…