PaperlayerView exact revision on arXiv

Source-linked AI summary

Identification of Systematic Errors of Image Classifiers on Rare Subgroups

Jan Hendrik Metzen, Robin Hutmacher, N. Grace Hua, Valentyn Boreiko, Dan Zhang

arXiv:2303.05072v2cs.CVcs.AIcs.LG

TL;DR

Rare, semantically coherent subgroups can expose classifier errors, yet identifying them is difficult when they lack annotations. PROMPTATTACK uses text-to-image synthesis and combinatorial testing to search prompt-defined subgroups; it demonstrates efficacy on controlled benchmarks and identifies rare-subgroup errors in ImageNet classifiers.

  • Problem

    Identifying semantically coherent subgroups with poor classifier performance is challenging when rare subgroups are unannotated and labeled examples are expensive to obtain.

    Source: §1 · p. 2

  • Method

    PROMPTATTACK synthesizes subgroup images from prompts encoding subgroup and class information, then uses combinatorial testing to search a prespecified operational design domain.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    PROMPTATTACK demonstrates efficacy on controlled benchmarks and identifies classifier-specific systematic misclassifications on rare ImageNet subgroups.

    Source: §1 · p. 2, §7 · p. 8

  • Takeaways & Limitations

    The identified errors include subgroups requiring combinations of semantic shifts, which can be missed by approaches considering single shifts.

    Source: §5 · p. 7

    Subsets difficult to describe in natural language, such as those sharing geometric layout, may not be directly identifiable by the procedure.

    Source: §6 · p. 8

Abstract

from arXiv · show

Despite excellent average-case performance of many image classifiers, their performance can substantially deteriorate on semantically coherent subgroups of the data that were under-represented in the training data. These systematic errors can impact both fairness for demographic minority groups as well as robustness and safety under domain shift. A major challenge is to identify such subgroups with subpar performance when the subgroups are not annotated and their occurrence is very rare. We leverage recent advances in text-to-image models and search in the space of textual descriptions of subgroups ("prompts") for subgroups where the target model has low performance on the prompt-conditioned synthesized data. To tackle the exponentially growing number of subgroups, we employ combinatorial testing. We denote this procedure as PromptAttack as it can be interpreted as an adversarial attack in a prompt space. We study subgroup coverage and identifiability with PromptAttack in a controlled setting and find that it identifies systematic errors with high accuracy. Thereupon, we apply PromptAttack to ImageNet classifiers and identify novel systematic errors on rare subgroups.

1. Introduction

Systematic errors are high-misclassification subgroups that remain semantically coherent, and finding them is difficult when rare groups lack annotations. PROMPTATTACK searches prompt-conditioned synthetic subgroups and identifies classifier-specific errors on rare ImageNet subgroups.

1. Introduction

  • Safety-critical deployment requires avoiding catastrophic, preventable failures across operational-design-domain subgroups, regardless of their rarity.Spurious training-data correlations can lead classifiers to use shortcut decision-making.Source: §1 · p. 1
  • A systematic error is a semantically coherent subgroup on which a pretrained classifier has high misclassification probability.The subgroup’s performance is considerably worse than on the domain’s data overall.Source: §1 · p. 2
  • Identifying errors in rare subgroups is difficult because labeled hold-out data can be expensive to acquire, while human-in-the-loop auditing raises costs.These data are important because systematic errors are more likely on subgroups rare in training.Source: §1 · p. 2
  • PROMPTATTACK encodes subgroup and class information in text-to-image prompts, using combinatorial testing to explore a relatively small set of subgroups across a large operational design domain.Unlike open-ended search, it targets a prespecified domain; it can also identify errors requiring several concurrent semantic shifts, unlike approaches focused on single shifts.Source: §1 · p. 2
  • PROMPTATTACK identifies classifier-specific, targeted systematic misclassifications on rare ImageNet subgroups.The paper also reports efficacy on carefully constructed benchmarks.Source: §1 · p. 2

2. Related Work

Prior subgroup-auditing approaches use annotated datasets, hold-out data, human review, or synthetic examples, each with limits in scalability, coverage, or the semantic shifts considered. PROMPTATTACK’s illustrated process combines domain-defined subgroups, combinatorial testing, prompt-conditioned image generation, and model-based ranking.

2. Related Work

  • The authors connect systematic-error identification to red teaming as a concern shared with large language models [38] [16].Source: §2 · p. 2
  • Annotated subgroup datasets require substantial collection and annotation effort, scale poorly, and can need exponentially more data to cover interacting semantic dimensions.Combinatorial t-wise testing is one partial remedy; synthetic corruptions do not simulate every semantic dimension.Source: §2 · p. 2
  • PROMPTATTACK defines semantic dimensions, uses combinatorial testing to generate subgroups, synthesizes prompt-conditioned samples, and ranks subgroups by source-class predictions.A low median source-class score signals a potential systematic error.Source: Figure 2 · p. 3
  • Hold-out-based error discovery can require rare labeled examples, while AdaVision uses human review; PROMPTATTACK does not require a human-in-the-loop.CLIP-based approaches can identify and annotate coherent subgroups but also require labeled hold-out data.Source: §2 · p. 3
  • Prior text-conditioned counterfactual methods considered single semantic shifts, whereas PROMPTATTACK searches combinations within a prespecified operational design domain.Wiles et al. [51] instead iteratively synthesize, cluster failures, and refine text descriptions; the approaches are complementary, and PROMPTATTACK does not require failure-case clustering and captioning.Source: §2 · p. 3

3. Method

PROMPTATTACK estimates classifier risk on text-to-image samples conditioned on a class and subgroup, then searches a compositional operational design domain using combinatorial testing.

3.1. Background

  • A subgroup z is modeled by conditioning the data distribution on a latent variable, without requiring an image captioning model.Source: §3.1 · p. 3

3.2. Systematic Errors: Definition

  • Subgroups are ranked by classifier risk minus baseline risk; a sufficiently high difference identifies a systematic error when baseline risk is low.The risk definition can also identify systematic misclassification into a specified target class.Source: §3.2 · pp. 3–4

3.3. Systematic Errors: Approximations

  • Because real subgroup samples may be unavailable, the method estimates risk from samples and generates images conditioned on both a subgroup and a source class.Explicit class conditioning avoids needing an estimate of the class probability given each image and subgroup.Source: §3.3 · p. 4
  • The method approximates excess risk with classifier risk when subgroups are designed to have negligible baseline risk and unambiguous class membership.Source: §3.3 · p. 4, §3.2 · p. 4
  • Generated samples may be out-of-subgroup, out-of-class, or out-of-distribution, and prompt engineering is needed to reduce such mismatches.Robust central-tendency estimators are used because a few mismatched samples may dominate the Monte Carlo estimate.Source: §3.3 · p. 4

3.4. Operational Design Domain Z

  • A compositional domain combines semantic dimensions in tuples; combinatorial testing covers every value combination across nC dimensions while testing fewer subgroups.Choosing nC below the total number of dimensions reduces the number tested but does not provide full coverage.Source: §3.4 · pp. 4–5

4. Quantitive Evaluation

The evaluation tests whether conditional synthesis covers a prespecified subgroup domain more evenly than unconditional synthesis and whether PromptAttack recovers injected errors across hyperparameter settings.

4.1. Coverage Analysis of Conditional versus Unconditional Synthesis

  • For every subgroup, conditional synthesis had a 95% confidence-interval lower bound above 0.005 versus unconditional synthesis’s minimum upper bound below 0.0001; recovery averaged about 89%, with none below 85%.The uniform subgroup probability is 0.01.Source: Figure 3 · p. 5, §4.1 · p. 5

4.2. Zero-Shot Systematic Error Benchmark

  • Stable Diffusion v1.5 greatly outperformed v2-base and v2-1-base, which the authors attribute to more prompt-faithful samples and better attribute binding.They use v1.5 in subsequent experiments.Source: Figure 4 · p. 6, §4.2 · p. 6
  • At least 16 samples were near-optimal, five inference steps worked despite impaired image quality, and class prompt weight 1.5 beat 1.0 while larger weights deteriorated performance.The subsequent setting was v1.5 with 16 samples, 20 inference steps, and weight 1.5.Source: Figure 4 · p. 6, §4.2 · p. 6

5. Qualitative Evaluation on ImageNet

PROMPTATTACK identifies ImageNet subgroups where classifier predictions shift toward specified target classes, including failures that require several semantic changes together. The experiments also find increased “ape” misclassification on an identified demographic subgroup.

Vehicle Experiment

  • For four selected target classes, PROMPTATTACK’s strongest prompts caused systematic misclassifications and considerably increased target-class confidence relative to a neutral baseline; Figure 5 reports median confidence over 16 samples.Source: §5 · p. 7, Figure 5 · p. 7
  • 25% of samples from the snowplow prompt were misclassified by VGG16, compared with 1% by ConvNeXt-B.The same prompt-matched samples retrieved from LAION-5B were also misclassified as snowplows by some models.Source: §5 · p. 7
  • For the minivan prompt template, ViT-B/16 consistently classified baseline and single-dimension-shifted samples as minivans, while PROMPTATTACK-selected shifts across all dimensions led to 300 of 1,000 samples being misclassified as snowplows.The cumulative fANOVA analysis reports that high fANOVA scores require at least 3 dimensions.Source: Figure 6 · p. 7, Figure 7 · p. 8

6. Limitations

The approach has limits tied to whether generated samples represent the intended source class, whether subgroups can be described in language, and whether the text-to-image model can generate marginalized subgroups.

False Positive Systematic Errors

  • When generated samples are not instances of the intended source class, the procedure can flag false-positive errors; without an oracle such as human review, these cannot be reliably identified.Moderate prompt engineering prevented such false positives in the operational design domains considered.Source: §6 · p. 8

Language Bottleneck

  • Subgroups that are difficult to describe in natural language, such as those defined by geometric layout, may not be identifiable through text prompts.The authors suggest scene-layout conditioning as a possible direction for addressing this language bottleneck.Source: §6 · p. 8

Bias Propagation

  • If the text-to-image model cannot generate samples for marginalized subgroups, the procedure cannot identify downstream classifier errors on those subgroups.The authors note that biases in the generator may propagate into systematic-error identification.Source: §6 · p. 8

7. Conclusion

PromptAttack uses text-to-image models to identify systematic errors on rare subgroups, and benchmark and classifier evaluations demonstrate its efficacy. Future work should address limitations through more controllable, versatile, and reliable image synthesis.

  • Quantitative results on carefully constructed benchmarks and qualitative results on multi-class image classifiers demonstrate PromptAttack's efficacy in identifying systematic errors on rare subgroups.Source: §7 · p. 8
  • PromptAttack leverages recent text-to-image progress to identify systematic errors on rare subgroups formed by combinations of semantic shifts.Source: §7 · p. 8
  • Future work should address the discussed limitations, for example by using more controllable, versatile, and reliable image-synthesis procedures.Source: §7 · p. 8

A.1. Experimental Setting: Vehicle Experiment

The vehicle experiment tests minivan misclassification across a five-dimension operational domain using combinatorial testing and synthesized samples.

  • The experiment evaluates ImageNet1k torchvision classifiers on minivan samples misclassified into vehicle classes at WordNet distance 2, excluding distance-1 classes as potentially fine-grained confusions.Source: §A.1 · p. 11
  • The operational domain varies viewpoint, object size, color, weather, and background, with neutral values leaving a dimension uncontrolled and simplifying prompts when appropriate.Source: §A.1 · p. 11
  • Combinatorial testing explores 1,230 of 18,720 subgroups and generates 16 Stable Diffusion v1.5 samples per subgroup.Source: §A.1 · p. 11

A.2. Experimental Setting: Person Experiment

The person experiment tests homo-to-ape misclassification with ImageNet21k MLP-Mixer models over a five-dimension prompt-defined domain.

  • The experiment evaluates MLP-Mixer models on homo-to-ape misclassification using a hypothetical binary classifier formed by retaining only the homo and ape logits.Source: §A.2 · p. 11
  • The operational domain varies age, gender, geographic region, hair type, and background, with neutral values allowing a dimension to remain uncontrolled.Source: §A.2 · p. 11
  • Combinatorial testing explores 1,371 of 12,150 subgroups and generates 16 Stable Diffusion v1.5 samples per subgroup.Source: §A.2 · p. 11

B. Samples Zero-Shot Benchmark

The section illustrates zero-shot benchmark samples across three image-synthesis settings: Stable Diffusion version, solver steps, and prompt class weight.

C. Samples ImageNet Experiments

ImageNet examples show PromptAttack identifying misclassifications of minivans and persons on synthesized subgroups. The reported patterns vary across models, and sample quality depends on the text-to-image model and generation settings.

C. Samples ImageNet Experiments

  • Thirty synthesized examples show minivans misclassified as snowplows, pickups, or police vans, and persons as apes; captions note imperfect viewpoints, missing background people, and narrower-than-prompt person coverage.The person samples are described as relatively similar and not fully representative of the prompt’s specified subgroup.Source: C. Samples ImageNet Experiments · p. 11, Figure 12 · p. 17, Figure 13 · p. 18
  • Vehicle-experiment strongest-error subgroups differ considerably across the five studied models.The table reports the strongest systematic error and misclassification subgroup for each model and across-model average.Source: Table 1 · p. 12
  • For ape misclassifications, the two MLP-Mixer models show distinct top-subgroup patterns: old male African persons for B/16 and young female Arab persons for L/16.The L/16 model generally has lower risk among its top-ranked subgroups.Source: Table 2 · p. 13
  • Stable Diffusion v1.5 samples show better attribute binding than v2-base and v2-1-base, which exhibit color leakage and partial object recoloring; the paper attributes v1.5’s better PromptAttack performance to this difference.The described color issues include background color leakage and, for vans, larger parts rendered white.Source: Figure 9 · p. 14
  • Even with 5 DPMSolver++ steps, PromptAttack reliably identifies systematic errors, although more steps produce more realistic samples.The figure compares samples generated with different numbers of steps.Source: Figure 10 · p. 15
  • At class weight 2.5, sample visual quality strongly deteriorates, explaining PromptAttack’s worse performance for that setting.The caption says the improved performance at weight 1.5 versus 1.0 is difficult to attribute to apparent sample properties.Source: Figure 11 · p. 16
Loading 2303.05072v2…