PaperlayerView exact revision on arXiv

Source-linked AI summary

Diagnosing and Rectifying Vision Models using Language

Yuhui Zhang, Jeff Z. HaoChen, Shih-Cheng Huang, Kuan-Chieh Wang, James Zou, Serena Yeung

arXiv:2302.04269v1cs.LGcs.AIcs.CLcs.CV

TL;DR

Vision-model diagnosis can require visual test data that is difficult to collect across distributions and failure modes. DrML uses language and cross-modal transferability to discover error slices, identify influential attributes, and rectify model behavior; the authors report effective diagnosis and rectification on image datasets with known errors.

  • Problem

    Many diagnosis methods require visual examples of failure modes, while collecting test data covering different distributions and potential failures is often impractical or infeasible.

    Source: §1 · pp. 1–2

  • Method

    DrML uses text embeddings as proxies for image inputs to discover error slices, identify influential attributes, and fine-tune vision classifiers using language-generated data.

    Source: §1 · p. 2, §2.3 · p. 5

  • Results

    DrML effectively identifies known error slices and influential attributes and rectifies model failure modes using language across image datasets.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The work indicates that vision classifiers built on contrastive image-text embedding spaces can be diagnosed and rectified using natural-language inputs.

    Source: §5 · p. 9

    The method assumes vision classifiers use image-text embeddings trained through multimodal contrastive learning and inherits limitations of those models and their pretraining data.

    Source: §4 · p. 9

Abstract

from arXiv · show

Recent multi-modal contrastive learning models have demonstrated the ability to learn an embedding space suitable for building strong vision classifiers, by leveraging the rich information in large-scale image-caption datasets. Our work highlights a distinct advantage of this multi-modal embedding space: the ability to diagnose vision classifiers through natural language. The traditional process of diagnosing model behaviors in deployment settings involves labor-intensive data acquisition and annotation. Our proposed method can discover high-error data slices, identify influential attributes and further rectify undesirable model behaviors, without requiring any visual data. Through a combination of theoretical explanation and empirical verification, we present conditions under which classifiers trained on embeddings from one modality can be equivalently applied to embeddings from another modality. On a range of image datasets with known error slices, we demonstrate that our method can effectively identify the error slices and influential attributes, and can further use language to rectify failure modes of the classifier.

1 INTRODUCTION

Existing diagnosis methods can depend on visual examples that are costly or impractical to collect across distributions and failure modes. DrML uses language to probe vision classifiers in shared image-text embedding spaces, identify errors and influential attributes, and rectify undesirable behavior.

1 INTRODUCTION

  • Multimodal contrastive learning produces shared image-text embedding spaces that capture visual and textual features and support strong single-modality classifiers.The paper presents language-based diagnosis as an additional advantage of these classifiers.Source: §1 · p. 1
  • Existing diagnosis methods often require visual examples of failure modes, making them reliant on datasets large enough to cover distributions and potential failures of interest.Collecting such datasets is often impractical or infeasible.Source: §1 · pp. 1–2
  • DrML uses language to discover error slices, including concepts without visual data, and to identify attributes with the greatest impact on model predictions.The approach is based on cross-modal transferability between image and text embeddings.Source: §1 · p. 2, Figure 1 · p. 2
  • The authors provide a theoretical explanation of when cross-modal transferability occurs and empirically verify its assumptions across multimodal contrastive models and datasets.Cross-modal transferability refers to using text inputs as proxies for image inputs in a shared embedding space.Source: §1 · p. 2
  • Across Waterbirds, FairFace, and dSpitesV, DrML identifies known error slices and influential attributes in simulated common failure modes.The paper describes these as three image datasets representing common types of model failure modes.Source: §1 · p. 2
  • DrML fine-tunes vision classifiers on text embeddings constructed during diagnosis to rectify undesirable behaviors and improve performance on identified error slices and influential attributes.The method aims to do this without collecting additional visual data.Source: §1 · p. 2

2 APPROACH

The approach treats text embeddings as proxies for image embeddings when modality-gap geometry supports cross-modal transfer. DrML uses generated language to find error slices, explain influential attributes, and fine-tune classifiers without collecting visual data.

2.1 TEXT EMBEDDINGS AS PROXIES FOR IMAGE EMBEDDINGS

  • A regularized linear classifier transfers across modalities when image–text gaps are constant and orthogonal to image embeddings, which have zero mean orthogonal to the gap.The theory also motivates closing the modality gap by centering image and text embeddings before training and evaluation.Source: §2.1 · pp. 3–4

2.2 DIAGNOSING VISION MODELS USING LANGUAGE

  • DrML generates diverse language inputs from attribute combinations and prompts, enabling discovery of more comprehensive, interpretable error slices, including unseen ones.Image test sets may cover only a small fraction of failure modes in the wild, while generated text inputs can be more diverse than available test images.Source: §2.2 · p. 4
  • DrML measures an attribute’s influence through the average prediction change associated with its presence versus absence, using language-based counterfactual inputs.The paper presents influential attributes and error slices as complementary: attributes explain why failures occur and help identify potential slices.Source: §2.2 · pp. 4–5

2.3 RECTIFYING VISION MODELS USING LANGUAGE

  • For each discovered slice, DrML composes attributes into text inputs and continues training the classifier, improving performance on that slice with minimal impact elsewhere and no visual data.The training uses the same hyperparameters as if the model were trained on corresponding images.Source: §2.3 · p. 5

3 EXPERIMENTS

The experiments test cross-modal transfer and language-based diagnosis and rectification, using CLIP and three datasets with simulated failure modes. DrML identifies known error slices and influential attributes, and language fine-tuning improves performance on error slices.

3.1 EXPERIMENTAL DETAILS

  • The experiments use CLIP embeddings and simulate spurious correlation on Waterbirds, underrepresentation on FairFaces, and unseen-data errors on dSpritesV.The experiments also evaluate cross-modal transfer on MS-COCO object classification.Source: §3.1 · p. 5

3.2 ARE TEXT EMBEDDINGS GOOD PROXIES FOR IMAGES EMBEDDINGS?

  • Table 1’s geometry measurements support a nearly constant modality gap, orthogonality to embedding spans, and zero embedding centers in the subspace orthogonal to the gap.The reported MS-COCO class-level gap for CLIP ViT-B/32 has magnitude 0.88±0.04 and direction cosine similarity 0.94 ± 0.04.Source: Table 1 · p. 6, §3.2 · p. 6
  • Text inputs support cross-modal classification: an image-trained linear classifier’s 67.90% macro F1 becomes 54.29% macro F1 on text inputs, with 96.37% prediction consistency.Closing the modality gap further improves transfer without harming in-modality evaluation.Source: Table 2 · p. 6, §3.2 · p. 6
  • Text/image slice-performance correlations are strong and improve with label probabilities, fluent prompt engineering, and ensembles; DrML outperforms text-to-image generation by a large margin.Source: Table 3 · p. 7, §3.2 · p. 7

3.4 IDENTIFIED INFLUENTIAL ATTRIBUTES

  • The influential-attribute analysis finds “ocean” for waterbird classification, “color” for dSpritesV, and no significantly influential attribute for FairFace.The unseen attribute “pink” raises the model’s predicted probability of “square” by 0.1 on dSpritesV.Source: §3.4 · pp. 7–8, Table 5 · p. 8

3.3 DISCOVERED ERROR SLICES

  • Language-based discovery identifies key error slices: waterbirds on land and landbirds in water, African American faces, and unseen green triangles, orange squares, and pink triangles.Image validation confirms the identified slices and the method’s predictions of their image-slice performance.Source: §3.3 · p. 7, Table 4 · p. 8

3.5 RECTIFIED MODEL MISBEHAVIORS

  • Continuing training on language inputs significantly improves error-slice performance on Waterbirds and FairFace, with minor effects on other slices.The reported comparison is between original and rectified models; GDRO is not directly comparable because it requires attribute annotations.Source: Table 6 · p. 8, §3.5 · p. 8
  • DrML outperforms JTT; GDRO performs similarly but requires costly image attribute annotations, while DrML can rectify errors on unseen data without visual inputs.Source: §3.5 · p. 8

4 RELATED WORK & DISCUSSION

The discussion positions language-based diagnosis and rectification as an additional use of multimodal contrastive embeddings. It connects text-space slice discovery and attribute-based interpretation to model fairness, while noting limits inherited from the encoders and their training data.

Multi-modal Contrastive Learning; Multi-modal Contrastive Representation Space Geometry

Slice Discovery

Interpretation

Algorithm Fairness

Limitations

  • The method assumes vision classifiers use multimodal contrastive embeddings, and low-level attributes such as texture or orientation can be difficult to diagnose in words.The discussion also notes that encoder quality may be affected by out-of-distribution examples or misaligned pretraining data.Source: §4 · p. 9

5 CONCLUSION

The conclusion argues that multimodal contrastive embedding spaces let vision classifiers be diagnosed and rectified using natural language. It reports theoretical and experimental support for cross-modal transferability and presents the framework as a direction for reliable computer vision and other domains.

CONCLUSION

  • The work combines theoretical analysis and experiments to verify cross-modal transferability, where text inputs can act as proxies for image inputs.This verification supports the proposed framework for diagnosing and rectifying vision classifiers with natural language.Source: §5 · p. 9
  • The paper proposes and validates a framework for diagnosing and rectifying vision classifiers using natural language inputs.Source: §5 · p. 9
  • The work suggests directions for reliable and trustworthy computer vision and for cross-modal transferability in other problem domains.Source: §5 · p. 9

ETHICS STATEMENT

The ethics statement reports that the approach can identify and rectify a FairFace behavior in which race affected gender predictions. It also warns that biased language prompts could be used to reinforce racial or gender biases.

ETHICS STATEMENT

  • In FairFace, race affected gender predictions, and the authors report rectifying this behavior with their approach.The example concerns predicting the label “female” for an image and the race attribute “black.”Source: ETHICS STATEMENT · p. 10
  • An adversary could use biased language prompts when fine-tuning a vision model to reinforce racial or gender biases.The statement also notes that the approach inherits limitations from the contrastive model and pretraining datasets.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

The appendix provides implementation and experimental resources, then describes modality-gap geometry and its relation to cross-modal transferability. It characterizes the gap as approximately constant and orthogonal to the embedding span, with embeddings centered in the orthogonal subspace.

REPRODUCIBILITY STATEMENT

  • The open-source implementation is intended to support reproducing the experiments and running analyses on additional multimodal models and datasets.Source: REPRODUCIBILITY STATEMENT · p. 10

OVERVIEW OF APPENDIX

  • The appendix covers modality-gap theory and results, dataset and experiment details, and two diagnosis baselines.The baselines include text-to-image generation, which sometimes fails to produce faithful images, and DOMINO slice discovery, which fails when error slices are absent from the dataset.Source: OVERVIEW OF APPENDIX · p. 14

A.1 MODALITY GAP GEOMETRY

  • The modality gap arises at initialization from the encoders’ cone effect and is preserved during optimization due to mismatched data.Figure 2 visualizes embeddings from the two modalities in the shared space.Source: §A.1 · p. 14, Figure 2 · p. 14
  • The gap is approximately a constant vector and orthogonal to the embedding span; embeddings are centered in the subspace orthogonal to the gap.The reported geometry findings suggest that cross-modal transferability is widespread in multimodal contrastive learning.Source: §A.1 · p. 14, Figure 3 · p. 15

A.2 THEORETICAL PROOF FOR CROSS-MODAL TRANSFERABILITY

The proof shows that a linear classifier trained with regularized quadratic loss transfers across modalities without accuracy loss under specific conditions on the image–text gap and image features.

A.2 THEORETICAL PROOF FOR CROSS-MODAL TRANSFERABILITY

  • A constant image–text gap orthogonal to the image-feature span, together with zero mean projected image features, guarantees cross-modal transfer without accuracy loss.The result applies to any linear classifier minimizing the regularized quadratic loss for λ > 0.Source: §A.2 · p. 16
  • The loss decomposition makes minimization independent along the gap direction and forces the classifier weights to satisfy Wg = 0.The proof uses the feature decomposition into a component orthogonal to g and a component along g.Source: §A.2 · p. 16
  • Because x − y = g and Wg = 0, the classifier produces identical outputs for paired image and text embeddings.This equality completes the proof of cross-modal transferability.Source: §A.2 · p. 16

A.3 ADDITIONAL CROSS-MODAL TRANSFERABILITY RESULTS

The additional experiments report cross-modal transfer on MS-COCO and ImageNet, including both transfer directions on MS-COCO and strong image-to-text results on ImageNet.

A.3 ADDITIONAL CROSS-MODAL TRANSFERABILITY RESULTS

  • The MS-COCO results include both image-to-text and text-to-image transfer, and the caption reports improvement from closing the modality gap without harm to in-modal evaluation.Table 7 covers multi-label classification and reports micro F1 and macro F1.Source: §A.3 · p. 16, Table 7 · p. 17
  • For ImageNet, the validation set was split into 40K training and 10K evaluation images, while prompt-generated texts were split into 64K and 16K.The texts were generated by applying CLIP’s 80 prompts to 1,000 class names.Source: §A.3 · pp. 16–17
  • 85.24% text-embedding accuracy followed 70.86% image classification accuracy for an image-trained classifier on ImageNet.Text-to-image transfer was much worse, which the authors attribute to prompt-generated texts lacking the diversity needed to train strong decision boundaries; closing the modality gap improves transferability in most cases.Source: §A.3 · p. 17, Table 8 · p. 17

A.4 THEORETICAL INTUITION FOR MODALITY GAP AND CROSS-MODAL TRANSFERABILITY

The theoretical intuition links contrastive learning to a modality gap while explaining how a linear image-trained classifier can still transfer to text representations.

A.4 THEORETICAL INTUITION FOR MODALITY GAP AND CROSS-MODAL TRANSFERABILITY

  • Under equal image and text presence probabilities, the contrastive objective is related to partitioning the image–text connection graph.The section describes this result as a generalization of HaoChen et al. (2021) to the multi-modal setting.Source: §A.4 · p. 18
  • Optimizing the multi-modal contrastive loss can separate image and text embeddings, and rescaling their encoders can preserve minimal loss while exhibiting a modality gap.Thus, minimizing the contrastive objective does not require the two modalities’ embeddings to be close.Source: §A.4 · p. 19
  • With equal-presence probabilities and label-consistent image–text pairings, infinite-dimensional contrastive representations admit a transferable image-trained linear classifier.The stated classifier has weight MF^T Yx and can be viewed as a class-mean classifier.Source: §A.4 · pp. 18–19
  • A class-mean linear head is similar to a trained linear head and is recovered by one gradient-descent step from zero initialization.The authors study this head because it is more amenable to theoretical analysis, helping explain transfer regardless of the modality gap.Source: §A.4 · p. 20

B DATASETS AND EXPERIMENTAL DETAILS

  • The experiments described in the datasets section cover MS-COCO, Waterbirds, FairFace, and dSpritesV.The passage identifies these as four datasets and mentions two major experiments.Source: B DATASETS AND EXPERIMENTAL DETAILS · p. 20

B.1 DATA PRE-PROCESSING

The experimental setup specifies dataset construction, known attributes, prompt templates, and use of a prompt ensemble.

B.1 DATA PRE-PROCESSING

  • The datasets use training/validation counts of 118K/5K MS-COCO image-caption pairs, 4.8K/1.2K Waterbirds images, 17K/11K FairFace images, and 1.3K/8.7K dSpritesV images.MS-COCO uses one randomly selected caption per image; dSpritesV is a synthesized variant with sampled visual attributes and restricted training combinations.Source: §B.1 · p. 20

B.2 ATTRIBUTES

  • The experiments use known attributes: Waterbirds species and place, FairFace race, age, and gender, and dSpritesV color, size, and shape.The authors state that attributes were not cherry-picked.Source: §B.2 · p. 20

B.3 PROMPT ENGINEERING

  • Dataset-specific raw and engineered prompt templates combine each dataset’s listed attributes into text inputs.These templates generate 800 Waterbirds prompts, 126 FairFace prompts, and 36 dSpritesV prompts.Source: §B.3 · pp. 20–21

B.4 PROMPT ENSEMBLE

  • The text inputs are augmented 80-fold using OpenAI CLIP’s prompts.The paper shows parts of the prompt set in Figure 4.Source: §B.4 · p. 21

B.5 EXPERIMENTAL DETAILS

The appendix reports experimental settings, discusses how attribute choice bounds diagnosis, and compares language-based diagnosis with text-to-image generation and DOMINO. It argues that language can reduce reliance on image collection, while attribute selection may still benefit from human iteration.

B.5 EXPERIMENTAL DETAILS

  • The appendix uses CLIP (ViT-B/32) embeddings in 512 dimensions and trains classifiers for 25 epochs, selecting checkpoints by validation loss on the training modality.Rectification continues training for 10 epochs and selects checkpoints using validation loss on text inputs.Source: §B.5 · p. 21

B.6 DISCUSSION: IS USING ATTRIBUTES AN APPROPRIATE CHOICE FOR MODEL DIAGNOSIS?

  • Initial attributes may miss essential errors, so human feedback can iteratively improve attribute design and reveal more vulnerabilities.The experiments used known attributes from dataset curation rather than cherry-picked attributes.Source: §B.6 · pp. 21–22
  • The authors argue that language makes it easier to generate diverse, controllable test inputs than to collect corresponding images, including for sensitive attributes without image data.They also state that performance correlation between image and text inputs is independent of attribute selection, although broader attribute coverage can reveal more errors.Source: §B.6 · p. 22

C.1 LANGUAGE-BASED VISION MODEL DIAGNOSIS BASELINE: TEXT-TO-IMAGE GENERATION

  • Text-to-image generation tests classifiers through generated images, but current models can produce low-fidelity images and require thousands more computations than DrML.The authors identify image fidelity as a reason this baseline is worse than their method.Source: §C.1 · p. 22

C.2 ERROR SLICE DISCOVERY BASELINE: DOMINO

  • The DOMINO baseline is evaluated as an image-based slice-discovery alternative, while the authors identify access to labeled images across distributions as the central practical challenge.Their discussion motivates language-synthesized test examples as a way to address that data constraint.Source: §C.2 · p. 22

C.3 MODEL RECTIFICATION BASELINES: GDRO AND JTT

The comparison emphasizes that DrML rectifies models using language without visual data, whereas GDRO and JTT depend on training images. The included baseline examples also show limitations in generated images and DOMINO’s slice discovery.

C.3 MODEL RECTIFICATION BASELINES: GDRO AND JTT

  • DrML rectifies model misbehavior without visual data, while GDRO and JTT require training images and therefore cannot fix errors on unseen data.GDRO additionally needs image attribute annotations; the authors leave combining their method with robust training for future work.Source: §C.3 · p. 23
  • The reported GDRO setup uses α = 0.2 and γ = 0.1, while JTT selects λup = 20 for Waterbirds and 5 for FairFace.Both train linear classifiers for 25 epochs with Adam at learning rate 0.001, keeping CLIP’s image encoder fixed.Source: §C.3 · p. 23
  • Text-to-image examples can misrepresent prompts through missing subjects, unnatural details, incorrect shapes, or polysemy.Examples include generated faces with multiple people or missing faces, birds with lobster claws, and “red square” interpreted as a Moscow landmark.Source: §C.3 · p. 25
  • For in-distribution datasets, DOMINO misses most top error slices, and its generated descriptions often fail to reflect slice images accurately.For out-of-distribution datasets, it captures some but not all slices; artificial OOD evaluation data remains challenging in real-world settings.Source: Figure 6 · p. 26, Figure 7 · p. 27
Loading 2302.04269v1…