PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Transferable Visual Models From Natural Language Supervision

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, Ilya Sutskever

arXiv:2103.00020v1cs.CVcs.LG

TL;DR

Natural-language image representation learning had shown weak benchmark performance, and its potential at web scale remained underexplored. CLIP learns image–text matching from 400 million internet pairs and uses class names for zero-shot classification. At sufficient scale, it can compete with task-specific supervised models across many datasets.

  • Problem

    Prior natural-language image representation learning showed weak benchmark performance, and its potential at web scale remained underexplored.

    Source: §1 · p. 2

  • Method

    CLIP learns image–text matching from 400 million internet pairs and uses class names as candidate text pairings for zero-shot classification.

    Source: §1 · p. 2, §3.1.2 · p. 6

  • Results

    Across many existing datasets, zero-shot CLIP can be competitive with task-specific supervised models at sufficient scale.

    Source: §9 · p. 27

  • Takeaways & Limitations

    The success of task-agnostic web-scale pre-training extends beyond NLP to computer vision.

    Source: §9 · p. 27

    Zero-shot CLIP remains weak on fine-grained and abstract tasks, with near-random performance possible on novel tasks.

    Source: §6 · p. 19

Abstract

from arXiv · show

State-of-the-art computer vision systems are trained to predict a fixed set of predetermined object categories. This restricted form of supervision limits their generality and usability since additional labeled data is needed to specify any other visual concept. Learning directly from raw text about images is a promising alternative which leverages a much broader source of supervision. We demonstrate that the simple pre-training task of predicting which caption goes with which image is an efficient and scalable way to learn SOTA image representations from scratch on a dataset of 400 million (image, text) pairs collected from the internet. After pre-training, natural language is used to reference learned visual concepts (or describe new ones) enabling zero-shot transfer of the model to downstream tasks. We study the performance of this approach by benchmarking on over 30 different existing computer vision datasets, spanning tasks such as OCR, action recognition in videos, geo-localization, and many types of fine-grained object classification. The model transfers non-trivially to most tasks and is often competitive with a fully supervised baseline without the need for any dataset specific training. For instance, we match the accuracy of the original ResNet-50 on ImageNet zero-shot without needing to use any of the 1.28 million training examples it was trained on. We release our code and pre-trained model weights at https://github.com/OpenAI/CLIP.

1. Introduction and Motivating Work

CLIP scales natural-language image supervision to 400 million image–text pairs and achieves zero-shot transfer competitive with task-specific supervised models. Its image and text encoders enable classifiers defined by class names or descriptions rather than fixed training labels.

1. Introduction and Motivating Work

  • NLP’s web-text pre-training and zero-shot transfer motivate investigating scalable natural-language supervision as an alternative to crowd-labeled computer-vision pre-training.Text-to-text interfaces enable downstream transfer without specialized output heads or dataset-specific customization (McCann et al., 2018; Raffel et al., 2019).Source: §1 · p. 1
  • Caption-word prediction produced representations comparable to ImageNet pre-training in Joulin et al. (2016), while Li et al. (2017) extended prediction to phrase n-grams.VirTex (Desai & Johnson, 2020), ICMLM (Bulent Sariyildiz et al., 2020), and ConVIRT (Zhang et al., 2020) subsequently explored transformer language modeling, masked language modeling, and contrastive objectives, respectively.Source: §1 · pp. 1–2
  • CLIP jointly trains image and text encoders to identify matching pairs, then embeds class names or descriptions to construct zero-shot linear classifiers.Unlike standard image models, it does not jointly train an image feature extractor with a fixed label classifier.Source: Figure 1 · p. 2
  • 11.5% zero-shot ImageNet accuracy for Li et al. (2017) illustrates the performance gap that discouraged natural-language supervision.This lagged 88.4% for Xie et al. (2020) and 50% for classic approaches. Targeted hashtag pre-training increased fine-tuned ImageNet accuracy by over 5% in Mahajan et al. (2018), while JFT-300M pre-training improved broader transfer benchmarks in Kolesnikov et al. (2019) and Dosovitskiy et al. (2020).Source: §1 · p. 2
  • 1000 and 18291 classes respectively constrain the weakly supervised approaches of Mahajan et al. (2018) and Kolesnikov et al. (2019).Their static softmax classifiers lack dynamic outputs, limiting flexibility and zero-shot capabilities; natural language can supervise a wider range of visual concepts.Source: §1 · p. 2
  • 400 million image–text pairs support training CLIP from scratch and studying natural-language supervision at large scale.Eight models span almost two orders of magnitude of compute, with transfer performance following a smoothly predictable function of compute.Source: §1 · p. 2
  • 4x greater learning efficiency comes from replacing bag-of-words prediction (Joulin et al., 2016) with CLIP’s contrastive objective.For zero-shot ImageNet classification, the transformer-based caption baseline learns 3x slower than the bag-of-words baseline.Source: Figure 2 · p. 3
  • Over 30 datasets reveal that zero-shot CLIP can compete with task-specific supervised models across diverse visual tasks.Tasks include OCR, geo-localization, and action recognition; linear-probe evaluation also shows CLIP outperforming the best publicly available ImageNet model with greater computational efficiency. Zero-shot CLIP is substantially more robust than supervised ImageNet models with equivalent accuracy.Source: §1 · pp. 2–3

2. Approach

CLIP jointly trains image and text encoders with a contrastive objective on 400 million internet image–text pairs. Natural-language supervision connects visual representations to language, enabling flexible zero-shot transfer.

2.1. Natural Language Supervision

  • Natural-language supervision can scale through existing internet text without standardized class labels while linking learned visual representations to language for flexible zero-shot transfer.Unlike conventional crowdsourced image classification, this approach does not require annotations formatted as a single categorical gold label.Source: §2.1 · p. 3

2.2. Creating a Sufficiently Large Dataset

  • 400 million internet image–text pairs form WebImageText (WIT), constructed to provide broader-scale natural-language supervision than existing datasets.Construction searches 500,000 queries and approximately balances coverage by including up to 20,000 pairs per query; WIT’s total word count resembles GPT-2’s WebText. Existing alternatives include roughly 100,000 training photos each in MS-COCO and Visual Genome, while filtering YFCC100M for English natural-language metadata leaves 15 million photos.Source: §2.2 · pp. 3–4

2.3. Selecting an Efficient Pre-Training Method

  • 4x greater efficiency in zero-shot transfer to ImageNet followed replacing the bag-of-words baseline’s predictive objective with a contrastive objective.The contrastive task predicts which complete text matches an image rather than its exact words. Earlier caption prediction with a 63-million-parameter transformer used twice its ResNet-50 encoder’s compute yet learned ImageNet recognition three times slower than bag-of-words prediction.Source: §2.3 · p. 4
  • CLIP jointly trains image and text encoders to distinguish matched pairs from mismatches within each batch using a symmetric contrastive objective.For N pairs, the encoders maximize cosine similarity for N correct pairings and minimize it for N^2 − N incorrect pairings, using symmetric cross-entropy loss.Source: §2.3 · p. 4
  • CLIP trains both encoders from scratch, using linear projections, random square image crops, and a learned temperature.Linear and nonlinear projections showed no training-efficiency difference, and sentence sampling is omitted because many paired texts contain only one sentence. Temperature starts at 0.07 and is clipped to prevent scaling logits by more than 100, avoiding training instability.Source: §2.3 · p. 4, §2.5 · p. 5

2.4. Choosing and Scaling a Model

  • Image encoders use modified ResNets with attention pooling or Vision Transformers (Dosovitskiy et al., 2020) with additional input layer normalization.ResNets incorporate ResNet-D improvements from He et al. (2019), antialiased rect-2 blur pooling from Zhang (2019), and single-layer multi-head attention pooling conditioned on global-average-pooled features. ResNet scaling allocates additional compute equally across width, depth, and resolution.Source: §2.4 · pp. 4–5
  • The masked-attention text Transformer represents lowercased BPE text through its final [EOS] activation, followed by normalization and linear projection.The base encoder has 63 million parameters, 12 layers, width 512, eight attention heads, a 49,152-token vocabulary, and maximum sequence length 76. Only text-encoder width scales with ResNet width; depth stays fixed because CLIP performance was less sensitive to text-encoder capacity.Source: §2.4 · p. 5

2.5. Training

  • ViT-L/14@336px is the best-performing model and default for reported CLIP results, adding one epoch at 336-pixel resolution after standard pre-training.Five ResNets and three Vision Transformers train for 32 epochs using Adam, decoupled weight decay, cosine learning-rate decay, and batches of 32,768. RN50x64 required 18 days on 592 V100 GPUs, while the largest Vision Transformer required 12 days on 256 V100 GPUs; training uses mixed precision and memory-saving techniques.Source: §2.5 · p. 5

3. Experiments

The experiments evaluate CLIP’s zero-shot transfer and linear-probe representation quality across datasets, comparing performance with supervised baselines and assessing robustness under natural distribution shifts. They also examine how prompt engineering, dataset-specific class names, and supervised ImageNet adaptation affect accuracy and robustness.

3.1.1. MOTIVATION

  • Zero-shot transfer evaluates generalization to unseen datasets as a proxy for task learning, rather than only recognition of unseen object categories.For benchmarks without a clearly defined real-world task, such as CIFAR-10, this primarily measures distribution-shift robustness and domain generalization rather than task generalization.Source: §3.1.1 · p. 6

3.1.2. USING CLIP FOR ZERO-SHOT TRANSFER

  • CLIP performs zero-shot classification by comparing image embeddings with text embeddings describing candidate classes, then applying temperature-scaled cosine similarities and softmax.This is a linear classifier with L2-normalized inputs and weights, no bias, and text-generated class weights; its text embeddings can be cached across predictions. Pre-training can consequently be interpreted as repeatedly optimizing proxy classification tasks containing 32,768 language-defined classes with one example per class.Source: §3.1.2 · p. 6

3.1.3. INITIAL COMPARISON TO VISUAL N-GRAMS

  • 76.2% ImageNet accuracy versus Visual N-Grams’ 11.5% (Li et al., 2017) lets zero-shot CLIP match the original supervised ResNet-50 without ImageNet training examples.CLIP also reaches 95% top-5 accuracy, matching Inception-V4 (Szegedy et al., 2016), reduces aYahoo errors by 95%, and more than doubles Visual N-Grams’ SUN accuracy. This is contextual rather than controlled: CLIP uses 10x more data, nearly 100x prediction compute, and likely over 1000x training compute, although a scratch-trained CLIP ResNet-50 matches Visual N-Grams’ ImageNet performance within one V100 GPU day on the same YFCC100M dataset.Source: §3.1.3 · pp. 6–7, Table 1 · p. 7

3.1.4. PROMPT ENGINEERING AND ENSEMBLING

  • Almost 5 points of average zero-shot improvement across 36 datasets come from prompt engineering and ensembling rather than contextless class names.Context resolves ambiguous class names and bridges the gap between isolated labels and pre-training sentences: “A photo of a {label}.” improves ImageNet accuracy by 1.3%, while task-specific descriptions further help. Averaging text embeddings permits caching without additional amortized prediction cost; on ImageNet, ensembling 80 prompts adds 3.5% beyond the default prompt.Source: Figure 4 · p. 7, §3.1.4 · pp. 7–8

3.1.5. ANALYSIS OF ZERO-SHOT CLIP PERFORMANCE

  • 16 of 27 datasets favor zero-shot CLIP over a fully supervised linear classifier trained on ResNet-50 features, including ImageNet.CLIP exceeds this baseline by over 20% on Stanford Cars and Food101, underperforms by over 10% on Flowers102 and FGVCAircraft, achieves 99.3% on STL10, and improves Kinetics700 and UCF101 by 14.5% and 7.7%, respectively. Weaknesses include satellite classification, tumor detection, counting, traffic signs, and driving-distance recognition; evaluating unfamiliar specialist tasks through zero-shot rather than few-shot transfer may itself be questionable.Source: Figure 5 · p. 8, §3.1.5 · pp. 8–9
  • 4-shot logistic regression on identical CLIP features matches zero-shot CLIP’s average performance, which also roughly matches the best evaluated 16-shot classifier.That 16-shot classifier uses BiT-M ResNet-152x2 features; natural language directly specifies concepts, but attempts to combine zero-shot weights with few-shot learning through L2 regularization often select essentially the zero-shot classifier. Estimated equivalent labels per class range from less than 1 to 184, with mean 20.8 and median 5.4; Flowers102 and EuroSAT underperform one-shot models, while ImageNet matches a 16-shot classifier on the same features.Source: Figure 6 · p. 9, §3.1.5 · pp. 9–10, Figure 7 · p. 10
  • 10 to 25 points typically separate zero-shot CLIP from fully supervised linear probes on its own features, despite strongly correlated performance.Assuming sufficiently large evaluation datasets, supervised linear probes approximately bound zero-shot performance; their correlation is 0.82 (p-value < 10^-6), and only five datasets approach parity within three points, all exceeding 90% accuracy under both evaluations. Across five CLIP ResNets, 39 evaluations on 36 datasets show a smooth log-log linear average-error trend over 44x model compute, although individual evaluations vary substantially.Source: Figure 8 · p. 10, §3.1.5 · p. 10, Figure 9 · p. 11

3.2. Representation Learning

  • 2.6% average improvement over Noisy Student EfficientNet-L2 on the 12-dataset linear-probe suite comes from CLIP ViT-L/14, trained one extra epoch at 336-pixel resolution.Linear probes expose representation failures that fine-tuning could mask and enable standardized comparisons of 66 models across 27 datasets, including OCR, geo-localization, emotion recognition, and action recognition. Small CLIP ResNets trail BiT-M and similarly costly EfficientNets, but ResNet-50x64 slightly surpasses Noisy Student EfficientNet-L2, while CLIP vision transformers are about 3x more compute-efficient than CLIP ResNets.Source: §3.2 · p. 11, Figure 10 · p. 12
  • 14.7% improvement on GTSRB exemplifies CLIP’s gains over EfficientNet, alongside advantages in OCR, geo-localization, scene recognition, video activity recognition, and fine-grained car recognition.EfficientNet nevertheless leads on ImageNet, CIFAR10, CIFAR100, PatchCamelyon, and CLEVRCounts, with both approaches performing poorly on the last two. The authors suggest narrow ImageNet supervision may obscure fine-grained distinctions, while CLIP’s lack of scale-based augmentation may partly explain its low-resolution weaknesses.Source: §3.2 · pp. 12–13
  • 21 of 27 datasets favor linear classifiers trained on CLIP features over Noisy Student EfficientNet-L2 features.On the broader evaluation suite, all CLIP scales outperform evaluated systems in compute efficiency, and the best model’s average-score advantage increases from 2.6% to 5%. SimCLRv2 also overtakes BiT-M on the 27-dataset suite despite trailing it on the 12 datasets of Kornblith et al. (2019).Source: §3.2 · p. 12, Figure 11 · p. 13

3.3. Robustness to Natural Distribution Shift

  • Seven natural-distribution-shift benchmarks from Taori et al. (2020) test robustness on newly collected images rather than synthetic perturbations of existing images.Reported averages cover the seven shifted datasets and corresponding ImageNet class subsets, averaging pm-0 and pm-10 accuracy for Youtube-BB and ImageNet-Vid; techniques improving synthetic-shift performance do not consistently improve natural-shift performance. Because distribution-specific correlations can fail elsewhere and most prior evaluations concern ImageNet-trained models, CLIP’s different data and natural-language supervision provide another way to investigate these failures.Source: §3.3 · p. 13
  • Up to 75% reduction in the gap between ImageNet accuracy and natural-distribution-shift accuracy accompanies substantially improved effective robustness across zero-shot CLIP models.Effective robustness measures shifted-distribution accuracy above that predicted from ImageNet accuracy, rather than any improvement in shifted-distribution accuracy. Zero-shot models can still exploit spurious correlations shared between pre-training and evaluation distributions.Source: §3.3 · pp. 13–14, Figure 13 · p. 15
  • Few-shot CLIP retains elevated effective robustness, but this advantage fades with additional supervision; zero-shot CLIP is more robust at equivalent ImageNet performance.Across experiments, limiting distribution-specific training data appears to improve effective robustness at the expense of dataset-specific performance, with most robustness benefits disappearing under full supervision. The authors favor task-agnostic pre-training and broad zero-shot/few-shot evaluation, as advocated by Yogatama et al. (2019) and Linzen (2020), to promote robustness and better assess performance.Source: §3.3 · pp. 15–16
  • 9.2% higher ImageNet accuracy, reaching 85.4% overall, follows supervised linear adaptation of CLIP, but average accuracy under natural distribution shift slightly decreases.ImageNetV2 improves significantly, while accuracy falls by 4.7% on ImageNet-R, 3.8% on ObjectNet, 2.8% on ImageNet Sketch, and 1.9% on ImageNet-A; changes on Youtube-BB and ImageNet Vid are insignificant. These results do not establish whether ImageNet supervision causes the robustness gap, whether other CLIP training choices explain it, or whether the pattern extends to end-to-end fine-tuning.Source: §3.3 · pp. 14–15, Figure 14 · p. 16
  • 5% improvement in average effective robustness results from constructing dataset-specific zero-shot CLIP classifiers directly from each dataset’s class names.This avoids imperfect ImageNet subclass pooling, used by Taori et al. (2020), when target categories differ from ImageNet categories. Improvements concentrate on a few datasets, although ObjectNet accuracy also increases by 2.3%.Source: §3.3 · p. 15, Figure 14 · p. 16

4. Comparison to Human Performance

Humans improved substantially with one example per breed, highlighting differences between human and CLIP few-shot learning. Images that were hardest for zero-shot CLIP were also difficult for humans.

4. Comparison to Human Performance

  • 3669 Oxford IIT Pets test images, spanning 37 breeds, were each labeled by five humans under zero-, one-, and two-example-per-breed conditions.Humans could abstain and could not search online in the zero-shot condition. Human and model few-shot conditions were imperfectly matched because models could not consult sample images as humans did.Source: §4 · p. 16
  • Integrating prior knowledge into CLIP’s few-shot learning could narrow the human–machine sample-efficiency gap noted by Lake et al. (2016).These CLIP evaluations did not effectively exploit prior knowledge, unlike humans. Linear classifiers on high-quality pretrained features were near state-of-the-art for few-shot learning (Tian et al., 2020), suggesting a broader gap.Source: §4 · p. 17
  • 76% average per-class accuracy with one example per breed, versus 54% zero-shot, demonstrated substantial human improvement, with minimal additional benefit from a second example.Almost all zero-to-one-shot gains occurred on images participants were uncertain about, suggesting humans could use a single example to update uncertain judgments.Source: Table 2 · p. 17, §4 · p. 17
  • The hardest images for zero-shot CLIP were also difficult for humans, with dataset noise and out-of-distribution images proposed as explanations for shared errors.Dataset noise includes mislabeled images; these explanations were hypotheses rather than established causes.Source: §4 · p. 17

5. Data Overlap Analysis

Detected training–evaluation overlap was generally limited and had little effect on CLIP’s overall zero-shot accuracy. Imperfect duplicate detection and distribution differences between overlapping and clean examples constrain this analysis.

5. Data Overlap Analysis

  • Pre-training overlap can invalidate generalization tests, but removing duplicates beforehand requires anticipating every evaluation dataset and can necessitate expensive retraining for new evaluations.This trade-off motivates measuring overlap and its performance effects rather than restricting evaluation to datasets known before training.Source: §5 · p. 17
  • CLIP RN50x64 contamination effects were estimated using All - Clean zero-shot accuracy, with duplicate detection and manually tuned thresholds defining Overlap and Clean subsets.Nearest-neighbor matches were manually inspected to prioritize precision while maximizing recall; contamination was measured as the size of Overlap divided by All. Statistical checks used one-tailed binomial tests against Clean accuracy and 99.5% Clopper-Pearson confidence intervals.Source: §5 · pp. 17–18
  • 0.1% was exceeded by overall accuracy shifts on only seven datasets, with only two statistically significant after Bonferroni correction.The maximum detected improvement was 0.6% on Birdsnap, which had 12.1% overlap.Source: §5 · p. 18
  • 2.2% median and 3.2% average overlap were detected across 35 datasets, with nine datasets showing no detected overlap.Datasets without detected overlap were generally synthetic, specialized, or collected after the pre-training dataset, supporting a low false-positive rate.Source: §5 · p. 18
  • 21.5% overlap on Country211, the highest detected rate, corresponded to only a 0.2% accuracy increase.Country211 was constructed from YFCC100M, a filtered subset of which appeared in pre-training. Its overlapping training captions often omitted location, potentially limiting their benefit for geo-localization.Source: §5 · p. 18
  • Imperfect duplicate recall and distribution shifts between Overlap and Clean limit attribution of accuracy changes to contamination and can mask overfitting.Detector recall could not be comprehensively checked across 400 million examples, despite high precision among inspected matches. Black transition frames explained Kinetics-700’s apparent 20% accuracy drop on Overlap, while low-resolution CIFAR-100 images produced false-positive duplicates.Source: §5 · p. 18

6. Limitations

CLIP’s zero-shot transfer remains below overall state-of-the-art performance, with weaknesses on specialized tasks and genuinely out-of-distribution data. Restricted outputs, data inefficiency, evaluation practices, and difficulty incorporating few-shot examples remain unresolved limitations.

6. Limitations

  • 0.6% on Birdsnap was the largest estimated overall test-accuracy increase attributable to detected data overlap.Detected overlap usually accounts for only a single-digit percentage of examples. Only six datasets showed statistically significant accuracy improvements under a one-sided binomial test.Source: Figure 17 · p. 19
  • Zero-shot CLIP averages competitive performance with a linear classifier on ResNet-50 features, a baseline below state of the art on most evaluated datasets.An estimated 1000x increase in compute would be needed to reach overall state-of-the-art performance, which is infeasible with current hardware; computational and data efficiency therefore require improvement.Source: §6 · pp. 18–19
  • CLIP struggles with fine-grained classification and counting, while novel tasks unlikely to appear in pre-training can yield near-random performance.Examples include distinguishing car models, flower species, and aircraft variants, and estimating the distance to the nearest car.Source: §6 · p. 19
  • 88% accuracy on MNIST leaves zero-shot CLIP below logistic regression on raw pixels, illustrating poor generalization to genuinely out-of-distribution data.CLIP performs well on digitally rendered text, which is common in pre-training, but retrieval finds almost no MNIST-like training images. Broad training coverage therefore does not resolve brittle generalization.Source: §6 · p. 19
  • CLIP can only select concepts supplied to its zero-shot classifier, unlike captioning models that can generate novel outputs.The tested captioning baseline was much less computationally efficient. Proposed alternatives include joint contrastive–generative training or inference-time search over natural-language explanations, similar to Andreas et al. (2017).Source: §6 · p. 19
  • 12.8 billion image presentations over 32 epochs illustrate CLIP’s poor data efficiency despite its scalable supervision source.Self-supervision (Henaff, 2020; Chen et al., 2020c) and self-training (Lee; Xie et al., 2020) are proposed ways to improve data efficiency. Unfiltered, uncurated internet image–text supervision also causes CLIP to learn social biases.Source: §6 · p. 20
  • Repeated full-validation-set feedback and a co-adapted evaluation collection limit how realistically CLIP’s methodology tests zero-shot transfer.The main evaluation uses a somewhat haphazard collection of 27 datasets, despite also reporting Kornblith et al. (2019)’s standardized 12-dataset suite. The authors propose a benchmark explicitly designed for broad zero-shot transfer.Source: §6 · p. 20
  • Text cannot easily specify every visual concept, and fitting linear classifiers on CLIP features can reduce performance when moving from zero-shot to few-shot learning.CLIP does not directly optimize few-shot performance; methods that combine its zero-shot capabilities with efficient learning from examples remain needed.Source: §6 · p. 20

7. Broader Impacts

This section considers useful applications of CLIP’s flexible zero-shot capabilities alongside bias and surveillance risks, emphasizing how class design, labeling thresholds, and deployment context shape potential harms. It also outlines future work to identify potential failure modes and areas for further research.

7. Broader Impacts

  • CLIP supports text–image retrieval and custom classification without retraining, expanding useful applications while requiring context-specific assessments of fitness, bias, and surveillance risks.Capabilities including OCR, action recognition, and geolocation can serve beneficial applications or surveillance. The initial bias probes are limited; deployment-specific analysis and broader contextual testing remain necessary.Source: §7 · p. 20

7.1. Bias

  • LR CLIP outperforms Linear Probe Instagram (Mahajan et al., 2018) and FairFace’s model on most FairFace tests, while zero-shot performance varies by category.Both variants exceed 95% gender-classification accuracy across all FairFace race categories. Benchmark accuracy does not establish real-world fairness, and smaller performance disparities need not mean smaller impact disparities.Source: Table 3 · p. 21, Table 4 · p. 21, §7.1 · pp. 21–22
  • 4.9% of FairFace images received non-human labels, with approximately 14% misclassification for Black images versus under 8% for every other race category.In the 10,000-image probe, crime-related misclassification affected 16.5% of male images versus 9.8% of female images. Ages 0–20 had 14% non-human and approximately 18% crime-related classification rates, versus approximately 12% crime-related for ages 20–60 and 0% for people over 70.Source: Table 6 · p. 22, §7.1 · p. 22
  • Adding a child class sharply reduced harmful classifications of under-20 faces, illustrating how class definitions, wording, and thresholds alter biases.FairFace’s two gender and seven race categories omit entire demographic groups and were used for comparison, not endorsement. Comparisons are also constrained by FairFace’s model using binary White/Non-White race classes.Source: §7.1 · pp. 21–23
  • 100% gender-classification accuracy on Members of Congress images coexisted with gendered occupation and appearance labels that changed with labeling thresholds.At a 0.5% occupation-label threshold, lower-quality outputs included nanny and housekeeper for women, versus prisoner and mobster for men. With commercial-system labels, women received more appearance labels and men more high-status occupations; lowering the threshold from 4% to 0.5% broadened occupational labels to women without similarly broadening female-associated descriptors to men.Source: §7.1 · p. 23, Figure 18 · p. 24

7.2. Surveillance

  • 91.8% top-1 CCTV classification accuracy dropped to 51.1% when similar competing captions were introduced, while fine-grained small-object presence detection performed near random.Testing covered 515 images from 12 video sequences, with at least six choices for initial coarse classification. Under the competing-caption evaluation, CLIP selected the similar incorrect answer 40.7% of the time.Source: §7.2 · p. 24
  • 59.2% zero-shot celebrity-identification top-1 accuracy with 100 candidate names fell to 43.3% with 1,000 names, below production-system performance.No task-specific dataset was used; CelebA overrepresents lighter skin tones, and the evaluation could not control demographic characteristics. CLIP lacks object-detection and semantic-segmentation support and has limited comparative surveillance appeal, but could lower data and skill barriers for niche applications.Source: §7.2 · pp. 24–25, Table 8 · p. 25

7.3. Future Work

  • Community exploration should map capabilities and limitations, identify beneficial and socially sensitive uses, characterize biases, and build early-development evaluation suites.The analysis is preliminary rather than comprehensive. Sensitive applications with many societal stakeholders may warrant policymaker intervention.Source: §7.3 · p. 25
  • The authors propose identifying potential failure modes and areas for further work.They plan to contribute to this work and hope their analysis provides motivating examples for subsequent research.Source: §7.3 · p. 25

8. Related Work

CLIP builds on natural-language supervision and text-image retrieval, but learns visual models from scratch with only a dot-product interaction between image and text representations. Its use of accompanying text also differs from query-labeled webly supervised learning.

8. Related Work

  • Natural-language supervision broadly encompasses distributional semantics and language modeling, while NLP also explicitly learns from explanations, instructions, advice, and interactive feedback.Semantic parsing converts explanations into features or additional labels (Hancock et al., 2018). ExpBERT (Murty et al., 2020) conditions contextual representations on explanations and relation descriptions to improve relation extraction.Source: §8 · pp. 25–26
  • Natural-language supervision predates CLIP in vision, spanning image retrieval, object classification, video understanding, grounded representations, and reinforcement-learning tasks.Ramanathan et al. (2013) used descriptions alongside other supervision for video event understanding, while He & Peng (2017) and Liang et al. (2020) improved fine-grained bird classification. Language-conditioned reinforcement-learning work demonstrated systematic zero-shot task accomplishment (Hill et al., 2019).Source: §8 · p. 26
  • CLIP’s text-image retrieval objective follows a research trajectory from predictive objectives toward joint multimodal embeddings, ranking objectives, and increasingly expressive models.Earlier embedding approaches used techniques such as kernel Canonical Correlation Analysis, while subsequent work explored combinations of objectives, transfer, and model expressiveness.Source: §8 · p. 26
  • Related work extends natural-language supervision beyond images to video and combines video, text, and raw audio for additional benefits.Stroud et al. (2020) pairs descriptive text with videos, while other work uses dense spoken supervision (Miech et al., 2019; 2020b). Alayrac et al. (2020) demonstrated benefits from combining all three modalities.Source: §8 · p. 26
  • 1–10 million training examples characterize earlier automatically constructed image-text datasets, which remain much smaller than WIT because of stronger filtering or task-specific design.Small crowd-sourced caption datasets limit achievable performance. Mithun et al. (2018) demonstrated that additional internet-collected image-text pairs could improve retrieval.Source: §8 · p. 26
  • Unlike query-labeled webly supervised learning, CLIP supervises training with full text accompanying images and restricts dataset queries to text-only substring matches.Webly supervised methods commonly obtain images through search engines with complex retrieval and filtering pipelines, potentially involving computer vision systems. Divvala et al. (2014) shares a notably similar ambition to CLIP.Source: §8 · p. 26
  • CLIP learns visual models from scratch, connecting image and text through a single dot product rather than densely connected joint-attention models.Related vision-language systems typically combine at least three pretrained subsystems and jointly fine-tune them for tasks such as visual question answering, commonsense reasoning, and multimodal entailment.Source: §8 · p. 27

9. Conclusion · A. Linear-probe evaluation

CLIP learns tasks during pretraining that support natural-language-prompted zero-shot transfer, with performance competitive with task-specific supervised models at sufficient scale. The appendix provides additional linear-probe evaluation details.

9. Conclusion

  • At sufficient scale, CLIP’s zero-shot performance can compete with task-specific supervised models, although considerable room for improvement remains.This transfer applies to many existing datasets.Source: §9 · p. 27
  • CLIP models learn to perform a wide variety of tasks during pretraining while optimizing their training objective.Source: §9 · p. 27
  • Natural-language prompting leverages tasks learned during pretraining to enable zero-shot transfer.Source: §9 · p. 27
  • Adopting task-agnostic, web-scale pretraining from NLP produces similar behaviors in computer vision.Source: §9 · p. 27
  • The paper also discusses the social implications of this line of research.Source: §9 · p. 27

A. Linear-probe evaluation

  • The linear-probe evaluation appendix provides additional experimental details, including the datasets and models used for evaluation.Source: A. Linear-probe evaluation · p. 37

A.1. Datasets · A.2. Models · A.3. Evaluation

The evaluation combines an expanded dataset suite and two newly constructed datasets with linear probes across CLIP variants and other pretrained models. A shared logistic-regression protocol tunes regularization on validation data before final evaluation.

A.1. Datasets

  • 15 additional datasets extend the 12-dataset suite introduced by, broadening evaluation across distributions and tasks.UCF101 and Kinetics700 use each video's middle frame as the input image. Results average all predefined splits for STL-10 and UCF101: 10 and 3, respectively.Source: §A.1 · p. 37
  • 211 country categories form Country211, a balanced geolocation dataset with 200 training and 100 test photos per country.Photos come from YFCC100m, restricted to ISO-3166 countries having at least 300 GPS-tagged photos.Source: §A.1 · p. 37
  • 448×448 images of Stanford Sentiment Treebank sentences, rendered as black text on white backgrounds, form Rendered SST2 for evaluating OCR.The dataset is designed to measure the optical character recognition capability of visual representations.Source: §A.1 · p. 37

A.2. Models

  • LM RN50 replaces contrastive learning with autoregressive text prediction, projecting ResNet-50 image features into four prefix tokens for a language model.It uses the same training dataset and number of epochs as other CLIP models.Source: §A.2 · p. 37
  • Five ResNet-based and four Vision Transformer CLIP variants span architectures, computational budgets, and input resolutions.ResNet variants include ResNet-50, ResNet-101, and three models using EfficientNet-style (Tan & Le, 2019) width, depth, and resolution scaling for roughly 4x, 16x, and 64x computation. Vision Transformer (Dosovitskiy et al., 2020) variants comprise ViT-B/32, ViT-B/16, and ViT-L/14 trained at 224-by-224 pixels, plus ViT-L/14 fine-tuned at 336-by-336 pixels.Source: §A.2 · p. 37
  • Supervised comparison models span convolutional and transformer architectures, with unavailable checkpoints limiting coverage of some model families.The comparison includes original and noisy-student EfficientNets (Tan & Le, 2019), four Instagram-pretrained ResNeXts (Mahajan et al., 2018) with two higher-resolution FixRes variants, and ResNet-50, ResNet-101, and ResNet152 (He et al., 2016b). It also includes BiT-S and BiT-M trained on ImageNet-1k and ImageNet-21k (Kolesnikov et al., 2019), plus ImageNet-21k-pretrained ViT-B/32, ViT-B/16, ViT-L/16, and ViT-H/14 (Dosovitskiy et al., 2020); BiT-L and the best-performing JFT-300M-trained ViTs were unavailable publicly.Source: §A.2 · pp. 37–38
  • Additional comparison models cover self-supervised learning and caption-based pretraining, using specified released checkpoints.These include seven pretrain-only SimCLRv2 checkpoints with selective kernels (Chen et al., 2020c), BYOL 50x1 and 200x2 (Grill et al., 2020), MoCo-v1 (He et al., 2020), and MoCo-v2 (Chen et al., 2020d). VirTex (Desai & Johnson, 2020) has a design similar to CLIP-AR but uses a 1000x smaller dataset of high-quality MSCOCO captions.Source: §A.2 · pp. 37–38

A.3. Evaluation

  • Linear probes train logistic regression on penultimate-layer features, using pre-projection features for CLIP-ViT and ignoring original classification heads.Scikit-learn’s L-BFGS runs for at most 1,000 iterations; L2 regularization spans 10^-6 to 10^6 over 96 logarithmically spaced steps, searched through binary refinement to eight steps per decade. Hyperparameters use provided validation data or a training-data split; validation examples then rejoin training for final evaluation on the unused split.Source: §A.3 · p. 38

A.4. Results · B. Zero-Shot Prediction

The best CLIP model achieved state-of-the-art linear-probe performance on 21 of 27 datasets under the reported confidence-interval criterion. Zero-shot results are presented through dataset-level scores and prediction visualizations for 36 classifiers.

A.4. Results

  • CLIP significantly outperforms other models on many datasets, demonstrating an advantage for natural-language supervision over traditional image-classification pre-training.Source: §A.4 · p. 38
  • The linear-probe evaluation uses Birdsnap and Kinetics700 resources available online at the time of writing.Source: Table 9 · p. 39
  • 21 of 27 datasets saw the best CLIP model achieve state-of-the-art linear-probe performance under the reported confidence-interval criterion.The model uses ViT-L/14 with 336-by-336-pixel images. Qualifying scores fall within the Clopper-Pearson 99.5% confidence interval around each dataset’s top score.Source: §A.4 · p. 38, Table 10 · p. 40
  • 27 datasets’ individual linear-probe scores are presented in Table 10 and plotted in Figure 20.Source: §A.4 · p. 38, Table 10 · p. 40, Figure 20 · p. 41

B. Zero-Shot Prediction

  • 36 zero-shot CLIP classifiers receive qualitative prediction visualizations, with randomly selected examples except Hateful Memes, which was reselected to avoid offensive content.Table 11 reports zero-shot performance over 27 datasets; Figure 22 compares CLIP’s zero-shot performance with linear-probe ResNet performance.Source: Figure 21 · p. 42, Table 11 · p. 43, Figure 22 · p. 43

A.4. Results

  • 5 highest-probability classes appear in each prediction visualization alongside their probabilities and the text representing each class.When multiple templates are used, only the first is shown. Ground-truth labels are green, while incorrect predictions are orange.Source: Figure 21 · p. 42

C. Duplicate Detector

The authors build a dedicated contrastive near-duplicate detector after finding semantic false positives and resizing-related false negatives in the model’s embedding space. The detector achieves nearly 100% accuracy on its proxy training task.

Duplicate detector

  • The model’s embedding space produces duplicate-detection false positives from semantic similarity and false negatives for textured near-duplicates resized using different interpolation algorithms.Distinct objects described similarly could have almost perfect similarity, while differently resized fur or stripe patterns could receive surprisingly low similarity.Source: C. Duplicate Detector · p. 44
  • Synthetic image transformations train the detector to match transformed versions while separating other batch images, using CLIP’s n-pair / InfoNCE loss.The loss uses a fixed temperature of 0.07. Augmentations combine cropping, zooming, aspect-ratio distortion, resolution changes, minor rotations, JPEG compression, HSV color jitter, and randomized interpolation algorithms.Source: C. Duplicate Detector · p. 44
  • Nearly 100% accuracy on the proxy training task was achieved by the ResNet-50 duplicate detector after training on approximately 30 million images sampled from the pretraining dataset.Training used a total batch size of 1,712; the reported accuracy concerns the proxy training task, not a separate duplicate-detection benchmark.Source: C. Duplicate Detector · p. 44
  • The detector uses an anti-aliased ResNet-50 (Zhang, 2019), weight normalization (Salimans & Kingma, 2016), and GELU activations (Hendrycks & Gimpel, 2016).Weight normalization replaces batch normalization (Ioffe & Szegedy, 2015) to avoid leaking duplicate information through batch statistics, a problem previously noted in (Henaff, 2020). GELU performed better for this task.Source: C. Duplicate Detector · p. 44

Training-data comparison

  • CLIP’s ResNet-50 shows similar average performance and win counts across zero-shot and linear-classifier evaluations when trained on YFCC100M or an equally sized WIT subset.Dataset-specific performance nevertheless differs substantially; Table 12 highlights the three strongest and weakest YFCC comparisons according to linear probes.Source: Table 12 · p. 44

D. Dataset Ablation on YFCC100M · E. Selected Task and Dataset Results · E.1. Image and Text Retrieval

Equally sized YFCC100M and WIT subsets yield similar average transfer performance, while zero-shot CLIP matches or outperforms prior zero-shot retrieval results on Flickr30k and MSCOCO.

D. Dataset Ablation on YFCC100M

  • Equally sized filtered YFCC100M and WIT subsets produce similar average performance across the evaluation suite in both zero-shot and linear-probe settings.Both models were trained for 32 epochs, when transfer performance began plateauing due to overfitting.Source: D. Dataset Ablation on YFCC100M · p. 44
  • Over 10% performance differences sometimes occur between YFCC100M- and WIT-pretrained models on specific fine-grained classification datasets.YFCC100M pretraining favors Birdsnap and Flowers102, whereas WIT favors car and pet classification. The authors speculate that these differences reflect the relative density of relevant pretraining data.Source: D. Dataset Ablation on YFCC100M · p. 44
  • The ablation suggests CLIP can use reasonably filtered image–text collections, with WIT’s much larger size suspected as its main advantage over YFCC100M.This is the authors’ interpretation rather than a demonstrated causal explanation.Source: D. Dataset Ablation on YFCC100M · p. 44
  • 3.7% of WIT comes from YFCC100M, an overlap that could underestimate performance differences between YFCC100M and the rest of WIT.The authors consider this unlikely because adding YFCC100M during WIT construction did not noticeably change model performance.Source: D. Dataset Ablation on YFCC100M · p. 45

E. Selected Task and Dataset Results

  • The selected-results sections supplement the main body’s aggregate analysis with performance details for specific task groups, datasets, and evaluation settings.The main body emphasizes overall results because of the large variety of datasets and experiments.Source: E. Selected Task and Dataset Results · p. 45

E.1. Image and Text Retrieval

  • Zero-shot CLIP matches or outperforms all prior zero-shot text and image retrieval results on Flickr30k and MSCOCO.Flickr30k text retrieval is competitive with overall state of the art; image retrieval trails overall state of the art but remains competitive with fine-tuned Unicoder-VL. On larger MS-COCO, fine-tuning significantly improves performance, and zero-shot CLIP is not competitive with the most recent work.Source: §E.1 · p. 45
  • Between 1 and 2 points of zero-shot R@1 improvement comes from prepending “a photo of” to image descriptions on Flickr30k and MSCOCO.The reported retrieval evaluations use this prompt.Source: §E.1 · p. 45

E.2. Optical Character Recognition

CLIP learns OCR capabilities that support sentence-level semantics and hateful-meme detection, but performance varies substantially across domains. It performs best on digitally rendered words and poorly on handwritten and street-view numbers.

E.2. Optical Character Recognition

  • 51% accuracy on full-number SVHN leaves CLIP below all published results, while zero-shot MNIST performance trails supervised logistic regression on raw pixels.Inspection suggests difficulties with repeated characters and SVHN’s low-resolution, blurry images.Source: §E.2 · p. 45, Table 14 · p. 46
  • Zero-shot CLIP on IIIT5K performs similarly to the early open-vocabulary OCR approach of Jaderberg et al. (2014).IIIT5K contains natural images of individually cropped words; the comparison approach combines deep learning with structured prediction.Source: §E.2 · p. 45
  • 80.5% accuracy on rendered SST-2 puts a linear classifier on CLIP representations on par with the 80% GloVe continuous-bag-of-words baseline.The baseline uses word vectors pre-trained on 840 billion tokens, while state-of-the-art accuracy is 97.5%. This demonstrates a non-trivial sentence-level representation derived from images of text.Source: §E.2 · p. 45, Table 14 · p. 46
  • CLIP’s OCR performance is strongest on digitally rendered words in Hateful Memes and SST-2, but noticeably weaker on handwritten and street-view numbers.Five datasets test character and word recognition (MNIST, SVHN, IIIT5K) and semantic tasks (Hateful Memes, SST-2). Primitive OCR capabilities emerged during CLIP development and appeared to improve steadily.Source: §E.2 · p. 45, Table 14 · p. 46
  • 0.7 points behind the single-model state of the art, fully supervised CLIP performs strongly on Hateful Memes without access to ground-truth text.CLIP exceeds the original paper’s best baseline by several points, while competing results use ground-truth text. Zero-shot CLIP also outperforms the best fully supervised linear probes among the 56 other evaluated models.Source: §E.2 · p. 45, Table 14 · p. 46

E.3. Action Recognition in Videos · E.4. Geolocalization

CLIP transfers strongly to action recognition in both linear-probe and zero-shot evaluations. Its geolocalization performance is similar to several task-specific models, but not competitive with the state of the art.

E.3. Action Recognition in Videos

  • CLIP’s linear probes match the best prior UCF101 result, outperform all other evaluated models, and exceed the original fine-tuned I3D baseline on Kinetics-700.CPU classifier costs force single-center-frame evaluation, likely moderately underestimating performance; these results are not directly comparable to prior video-based evaluations.Source: Table 15 · p. 46, §E.3 · pp. 46–47
  • Within 1% of fully supervised I3D on Kinetics-700, zero-shot CLIP averages predictions across all frames without a training stage.The I3D baseline was trained on 545000 labeled videos.Source: §E.3 · p. 47
  • 10 points above the prior state of the art, CLIP improves zero-shot recognition of unusual actions on RareAct (Miech et al., 2020a).The previous best model was S3D trained on automatically extracted captions from 100 million instructional videos.Source: §E.3 · p. 47
  • Architecture, training-data distribution, dataset size, and compute differences prevent isolating which design decisions produce CLIP’s strong action-recognition performance.These experiments test verb recognition to investigate whether broader language supervision transfers beyond ImageNet’s common-noun labels.Source: §E.3 · pp. 46–47

E.4. Geolocalization

  • 1 million reference images give CLIP IM2GPS performance similar to several task-specific models, despite using fewer reference images; it remains uncompetitive with the state of the art.Performance measures the percentage of images localized within a given radius.Source: Table 17 · p. 47, §E.4 · p. 47
  • CLIP geolocalizes IM2GPS images using the GPS coordinates of their nearest reference image in embedding space, making this nearest-neighbor regression rather than zero-shot evaluation.The authors also created Country211 to quantify location recognition, while IM2GPS enables comparison with prior work.Source: §E.4 · p. 47

E.5. Robustness to Distribution Shift · F. Model Hyperparameters

Zero-shot CLIP improves the state of the art on five of seven evaluated datasets, with its largest improvements on ImageNet-Vid, Youtube-BB, and ImageNet-R. Hyperparameter tables cover common CLIP settings, CLIP-ResNet, and CLIP-ViT.

E.5. Robustness to Distribution Shift

  • 5 of 7 datasets see state-of-the-art improvements from zero-shot CLIP: ImageNet-R, ObjectNet, ImageNet-Sketch, ImageNet-Vid, and Youtube-BB.The comparison uses state-of-the-art results reported in Taori et al. (2020)’s evaluation suite.Source: §E.5 · p. 47
  • CLIP’s improvements are among the largest on ImageNet-Vid and Youtube-BB because of its flexible zero-shot capability.Source: §E.5 · p. 47
  • The appendix supplements Section 3.3’s high-level ImageNet robustness analysis with per-dataset numerical results in Table 16.Source: §E.5 · p. 47
  • CLIP’s large improvement on ImageNet-R likely reflects significant creative content in its pre-training distribution.Source: §E.5 · p. 47
  • Similar behavior has been documented for Instagram-pre-trained ResNeXt models, as discussed in Taori et al. (2020).Source: §E.5 · p. 47

F. Model Hyperparameters

Related papers

Loading 2103.00020v1…