PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Transferable Visual Models From Natural Language Supervision

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, Ilya Sutskever

arXiv:2103.00020v1cs.CVcs.LG

TL;DR

Conventional vision models rely on fixed labeled categories, limiting their ability to recognize broader concepts. CLIP learns from 400 million image-text pairs and transfers via language, achieving competitive performance across many tasks while remaining below overall state of the art.

  • Problem

    Natural-language supervision had shown limited computer-vision benchmark performance, leaving unclear whether web-scale text could support competitive, general-purpose visual representation learning.

  • Method

    CLIP pre-trains on 400 million image-text pairs to predict matching pairs, then performs zero-shot classification by comparing image embeddings with language descriptions of candidate classes.

  • Results

    Across over 30 vision datasets, CLIP transfers non-trivially to most tasks and can be competitive with task-specific supervised models.

  • Takeaways & Limitations

    Web-scale natural-language pre-training can produce visual models whose learned task knowledge supports zero-shot transfer across many existing computer-vision datasets.

  • Takeaways & Limitations

    CLIP remains below overall state-of-the-art performance, and reaching it may require an infeasible approximately 1000-fold increase in compute with current hardware.

Abstract

from arXiv · show

State-of-the-art computer vision systems are trained to predict a fixed set of predetermined object categories. This restricted form of supervision limits their generality and usability since additional labeled data is needed to specify any other visual concept. Learning directly from raw text about images is a promising alternative which leverages a much broader source of supervision. We demonstrate that the simple pre-training task of predicting which caption goes with which image is an efficient and scalable way to learn SOTA image representations from scratch on a dataset of 400 million (image, text) pairs collected from the internet. After pre-training, natural language is used to reference learned visual concepts (or describe new ones) enabling zero-shot transfer of the model to downstream tasks. We study the performance of this approach by benchmarking on over 30 different existing computer vision datasets, spanning tasks such as OCR, action recognition in videos, geo-localization, and many types of fine-grained object classification. The model transfers non-trivially to most tasks and is often competitive with a fully supervised baseline without the need for any dataset specific training. For instance, we match the accuracy of the original ResNet-50 on ImageNet zero-shot without needing to use any of the 1.28 million training examples it was trained on. We release our code and pre-trained model weights at https://github.com/OpenAI/CLIP.

1. Introduction and Motivating Work

CLIP extends web-scale, task-agnostic pre-training to vision by learning image–text pairings and using language to construct zero-shot classifiers. It addresses the limited scale, vocabulary, flexibility, and benchmark performance of earlier natural-language supervision approaches, achieving competitive transfer and improved efficiency and robustness.

  • CLIP approach: CLIP jointly trains image and text encoders to predict matching image–text pairs, then uses class names or descriptions to synthesize a zero-shot linear classifier.This replaces a fixed classifier with language-based outputs for target datasets.
  • Limitations of prior work: Prior weakly supervised vision models restricted supervision to 1,000 or 18,291 classes and used static softmax classifiers, limiting their visual vocabulary, dynamic outputs, and zero-shot flexibility.Natural language can express a substantially wider set of visual concepts than these fixed class inventories.
  • Contribution: CLIP closes the scale gap between earlier language-supervised vision studies trained on one to two hundred thousand images and weakly supervised models trained on millions to billions, enabling competitive task-specific transfer.The cited comparison contrasts accelerator-days training for VirTex, ICMLM, and ConVIRT with accelerator-years training for Mahajan et al. (2018) and Kolesnikov et al. (2019).
  • Efficiency: Contrastive training improves zero-shot ImageNet transfer efficiency by 4x beyond a bag-of-words baseline, while transformer language modeling learns 3x slower than that baseline.These efficiency comparisons are reported for CLIP relative to the image-caption baseline.
  • Results: CLIP outperforms the best publicly available ImageNet model in linear-probe representation learning and produces zero-shot models more robust than equally accurate supervised ImageNet models.The paper also reports competitiveness with prior task-specific supervised models.

2. Approach

CLIP learns visual representations from natural-language supervision by training on 400 million publicly collected image–text pairs. Its contrastive objective matches images with their paired text while separating mismatched pairs, using scalable image and text encoders.

  • Data: CLIP trains on WIT, a dataset of 400 million image–text pairs collected from publicly available internet sources to broaden visual concept coverage.The dataset was constructed because existing caption datasets were small and would underestimate the potential of internet-scale natural-language supervision.
  • Contrastive objective: The contrastive approach replaces exact-text prediction, which is difficult because images co-occur with varied descriptions, comments, and related text.An initial transformer language-model approach recognized ImageNet classes three times slower than a simpler bag-of-words baseline while using twice the compute of its ResNet-50 encoder.
  • Contrastive objective: Given N image–text pairs, CLIP identifies the N observed pairings among N × N candidates by maximizing matched embedding similarity and minimizing mismatched similarity.It jointly trains image and text encoders in a multimodal embedding space with a symmetric cross-entropy loss over similarity scores.
  • Training: CLIP is trained from scratch without ImageNet initialization or pretrained text weights, taking advantage of the large pre-training dataset to simplify training details.The approach does not use the nonlinear projection between encoder representations and the contrastive embedding space.
  • Architectures: The image encoder uses either a modified ResNet-50 or Vision Transformer, while the text encoder is a Transformer operating on lower-cased byte-pair-encoded text.The ResNet incorporates ResNet-D improvements, antialiased blur pooling, and attention pooling; the base text model has 63M parameters, 12 layers, 512-wide representations, and 8 attention heads.

3. Experiments

Experiments evaluate CLIP as a zero-shot classifier across unseen datasets, finding performance that often matches or exceeds supervised and few-shot baselines while revealing weaknesses on complex tasks. Natural-language prompts and ensembling improve transfer, but dataset naming and word-sense ambiguities constrain performance.

  • Zero-shot transfer: CLIP performs zero-shot classification by embedding each image and candidate class text, then selecting the most probable image–text pairing.The candidate texts are the names of all classes in each dataset, reusing the image–text pairing capability learned during pre-training.
  • Zero-shot transfer: On ImageNet, CLIP reaches 76.2% accuracy, matching the original ResNet-50 without using its 1.28 million labeled training examples, and achieves 95% top-5 accuracy, matching Inception-V4.The best CLIP model improves ImageNet accuracy from the 11.5% proof of concept to 76.2%.
  • Prompting and ensembling: The prompt “A photo of a {label}.” improves ImageNet accuracy by 1.3%, while ensembling 80 context prompts adds another 3.5%.Prompting bridges the gap between single-word class labels and the full-sentence descriptions common in CLIP’s pre-training data; embedding-space ensembling can be cached with the compute cost of one classifier when amortized over many predictions.
  • Zero-shot transfer: Zero-shot CLIP wins on 16 of 27 datasets and slightly outperforms the supervised baseline more often than not.Performance varies widely: it exceeds ResNet-50-feature logistic regression by over 20% on Stanford Cars and Food101 but trails by over 10% on Flowers102 and FGVCAircraft.
  • Limitations: Zero-shot CLIP is weak on specialized, complex, or abstract tasks, including satellite imagery, tumor detection, synthetic counting, traffic signs, and car-distance recognition.Transfer is also hindered when datasets lack class-name mappings or when labels are polysemous, because names alone may not specify the intended visual concept.
  • Few-shot comparison: Zero-shot CLIP matches four-shot logistic regression on the same feature space and roughly matches the best 16-shot classifier evaluated, using BiT-M ResNet-152x2 features.The comparison is motivated by zero-shot being the limit of few-shot learning; natural language directly communicates visual concepts rather than requiring them to be inferred from labeled examples.

4. Comparison to Human Performance

On Oxford IIT Pets, humans improved from 54% zero-shot accuracy to 76% with one example per class, while additional examples yielded minimal gains. Human and CLIP difficulty patterns were correlated, plausibly because dataset noise and out-of-distribution images challenge both.

  • Evaluation setup: Humans evaluated 3,669 Oxford IIT Pets test images across 37 cat and dog breeds in zero-shot, one-shot, and two-shot settings.Five participants selected the best-matching breed or “I don’t know”; zero-shot participants received no breed examples or internet search, while one-shot participants saw one sample per breed.
  • Human performance: 54% human accuracy rose to 76% with one training example per class, while a second example produced only a minimal marginal gain.Nearly all zero-to-one-shot improvement occurred on images humans initially found uncertain, suggesting they update priors selectively after seeing one example.
  • Implications: The human few-shot advantage reveals a gap in sample efficiency because humans use prior knowledge and can incorporate example images more effectively than CLIP’s few-shot evaluations.The authors suggest integrating prior knowledge into few-shot learning as an important direction for reducing this gap.
  • Human–model comparison: CLIP’s hardest problems were also hard for humans, consistent with shared effects from dataset noise, mislabeled images, and out-of-distribution examples.The authors present these factors as hypotheses explaining the observed alignment between human accuracy and CLIP’s zero-shot accuracy.

5. Data Overlap Analysis

CLIP’s pre-training data shows limited overlap with downstream evaluation datasets, and this contamination rarely changes measured zero-shot accuracy substantially. The analysis estimates contamination effects while acknowledging detector and distribution-shift limitations.

  • Method: For each evaluation dataset, nearest-neighbor duplicates were manually inspected to set a high-precision threshold, producing Overlap, Clean, and unaltered All subsets.Contamination was measured as Overlap divided by All, and the main effect metric was the difference between All and Clean zero-shot accuracy for CLIP RN50x64.
  • Results: The analysis found no detected overlap in 9 of 35 datasets, supporting a low false-positive rate for the duplicate detector.These datasets were typically synthetic, specialized, or otherwise unlikely to appear as ordinary internet images.
  • Results: Median overlap was 2.2% and average overlap 3.2%; accuracy shifted by more than 0.1% on only 7 of 35 datasets, with just 2 significant after Bonferroni correction.The maximum detected improvement was 0.6% on Birdsnap, while Country211 had the largest overlap at 21.5%.
  • Limitations: The contamination estimates are imperfect because detector recall cannot be checked across 400 million examples and Overlap may differ from Clean in distribution or difficulty.These shifts can produce apparent accuracy changes, such as Kinetics-700’s 20% drop, or mask contamination effects.
  • Related work: The findings agree with Mahajan et al. (2018) and Kolesnikov et al. (2019), who reported similar overlap rates and minimal overall-performance changes.Kolesnikov et al. (2019) also found little difference between full deduplication and the adopted overlap-analysis approach.

6. Limitations

CLIP remains competitive with a simple supervised baseline on many datasets but has substantial limitations in task coverage, out-of-distribution generalization, data efficiency, evaluation methodology, and bias. Its zero-shot interface is flexible, yet it cannot generate concepts outside a specified classifier and does not directly optimize few-shot performance.

  • Performance limitations: Zero-shot CLIP is competitive with a linear classifier on ResNet-50 features but remains below overall state-of-the-art performance on most datasets, with roughly 1000x more compute estimated for state-of-the-art zero-shot accuracy.Scaling has steadily improved performance, but substantial additional computation may be required.
  • Task limitations: CLIP performs poorly on fine-grained classification, counting, and novel tasks unlikely to appear in its pre-training data.Weaknesses include distinguishing car models, flower species, and aircraft variants, as well as counting objects.
  • Out-of-distribution generalization: CLIP generalizes poorly to truly out-of-distribution data: it reaches 88% accuracy on handwritten MNIST digits, while logistic regression on raw pixels performs better.Its semantic OCR representation works well on digitally rendered text but not on handwritten digits, which have almost no resembling images in its training data.
  • Interface limitations: CLIP requires a predefined set of candidate concepts, limiting its ability to produce novel outputs compared with image captioning, although the tested captioning baseline was much less computationally efficient.Natural-language classifiers are flexible across tasks but remain restricted to the concepts specified in each classifier.
  • Data efficiency: CLIP does not solve deep learning’s poor data efficiency: training on 12.8 billion images for 32 epochs would take 405 years at one image per second.The approach compensates for data inefficiency by scaling internet supervision to hundreds of millions of training examples.
  • Methodological limitations: The evaluation methodology is not fully representative of true zero-shot use because development repeatedly queried full validation sets, while text-only specification can be difficult and the model does not directly optimize few-shot performance.The paper also acknowledges concerns about evaluation-dataset selection and falls back to linear classifiers on CLIP features for few-shot settings.
  • Bias: Because CLIP learns from unfiltered internet image-text pairs, it can acquire social biases.The paper refers to Section 7 for detailed analysis, quantification, and mitigation strategies.

7. Broader Impacts

CLIP’s flexible zero-shot classification enables promising retrieval and search applications but also creates substantial risks when deployed for surveillance or sensitive human classification. Biases vary across demographics, labels, thresholds, and design choices, so benchmark accuracy alone cannot establish real-world fairness or safety.

  • Capabilities and risks: CLIP supports broadly applicable zero-shot image retrieval and search, while its ease of steering toward bespoke applications also enables uses with significant social implications.The paper emphasizes that applications such as classifying shoplifters or deploying surveillance systems require context-specific evaluation and broader-impact analysis.
  • FairFace evaluation: LR CLIP outperformed Linear Probe Instagram and FairFace’s model on most FairFace classification tests, but benchmark accuracy remains only an approximation of algorithmic fairness.Gender-classification performance exceeded 95% across all race categories, yet higher accuracy or lower subgroup disparities do not guarantee fairness in real-world contexts.
  • Demographic bias: 4.9% of images were misclassified into non-human classes, rising to approximately 14% for Black images and people aged 0–20 years.All other races had misclassification rates under 8%, while the reported confidence interval for the overall rate was 4.6%–5.4%.
  • Demographic bias: 16.5% of male images versus 9.8% of female images received crime-related labels, with approximately 18% of people aged 0–20 receiving them versus 0% of people over 70.The authors also found significant disparities across races for crime-related classifications.
  • Bias mechanisms and limitations: Thresholds and design choices changed biased labels: a 0.5% threshold reduced label quality and produced gendered appearance or occupation terms absent or uncommon at 4%.The authors argue that training data, architecture, class design, and thresholding can heighten or reduce harms, requiring broader and more contextual bias testing.
  • Surveillance evaluation: 91.8% top-1 accuracy on initial CCTV evaluation fell to 51.1% when captions differed only in fine-grained details, with 40.7% incorrect “close” choices.The fine-grained detection experiment performed near randomly and targeted only the presence or absence of small objects in image sequences.

8. Related Work

CLIP builds on broad efforts to use natural language as supervision, including prior work in multimodal retrieval, webly supervised learning, and joint vision-language modeling. Its closest precedents span natural-language supervision beyond images, large-scale image-text datasets, and text-image retrieval objectives.

  • Natural Language Supervision: Natural-language supervision encompasses distributional semantics, dialog feedback, semantic explanations, and language-conditioned feature learning across a broad range of methods.Examples include topic models, word and sentence vectors, language models, dialog-based learning, semantic parsing, and ExpBERT.
  • Natural Language Supervision: CLIP extends earlier natural-language supervision for non-language domains, including video event understanding using descriptions alongside other supervision [Ramanathan et al. (2013)].Related work also applies natural-language supervision to videos and other modalities, suggesting its broader potential for perceptual systems.
  • Text-Image Retrieval: CLIP’s text-image retrieval objective follows research from early image retrieval through joint multimodal embeddings trained with correlation and ranking objectives.Representative prior work includes Mori et al. (1999), Weston et al. (2010), Socher & Fei-Fei (2010), and Hodosh et al. (2013).
  • Image-Text Datasets: CLIP’s image-text dataset addresses the small scale of crowd-sourced caption benchmarks such as Pascal1K, Flickr8K, and Flickr30K, alongside automatically constructed larger datasets.The related work frames dataset scale as a limitation on achievable image-text retrieval performance.
  • Webly Supervised Learning: CLIP is related to webly supervised learning, which uses image-search queries as noisy labels and can make large automatically collected datasets competitive with smaller carefully labeled ones.Image-query pairs are also used as additional training data for standard datasets.
  • Vision-Language Models: CLIP also relates to richly integrated vision-language models for visual question answering, visual commonsense reasoning, and multimodal entailment, typically combining multiple pretrained subsystems.These systems commonly combine image features, region proposals or object detection, and masked language modeling components.

9. Conclusion · A. Linear-probe evaluation

The paper investigates transferring task-agnostic, web-scale NLP pre-training to computer vision, finding similar behaviors and enabling diverse task learning through natural-language prompting. The appendix provides dataset and model details for the paper’s linear-probe experiments.

  • 9. Conclusion: CLIP investigates whether task-agnostic, web-scale pre-training can transfer from NLP to computer vision.
  • 9. Conclusion: The study finds that adopting this pre-training formula produces similar behaviors in computer vision.
  • 9. Conclusion: The conclusion discusses the social implications of this research direction.
  • 9. Conclusion: During pretraining, CLIP models learn to perform a wide variety of tasks to optimize their training objective.
  • 9. Conclusion: Natural-language prompting can leverage the task learning acquired during CLIP pretraining.
  • A. Linear-probe evaluation: The linear-probe appendix provides additional details about the experiments presented in the paper.
  • A. Linear-probe evaluation: These details include the datasets and models used for linear-probe evaluation.

A.1. Datasets · A.2. Models · A.3. Evaluation

The evaluation spans 27 datasets, including purpose-built benchmarks for geolocation and OCR, and compares CLIP against diverse vision and multimodal model families using standardized linear-probe evaluation. Models are evaluated from penultimate-layer features with validation-based hyperparameter selection and final testing on held-out data.

  • A.1. Datasets: The benchmark combines 12 datasets from Kornblith et al. (2019) with 15 additional datasets covering broader distributions and tasks.The suite includes MNIST, facial expression recognition, STL-10, EuroSAT, NWPU-RESISC45, and GTSRB.
  • A.1. Datasets: Country211 measures geolocation with 211 balanced country categories, each containing 200 training and 100 testing photos filtered from YFCC100m.Countries require an ISO-3166 code and at least 300 GPS-tagged photos.
  • A.1. Datasets: Rendered SST2 measures optical character recognition by rendering Stanford Sentiment Treebank sentences as 448×448 black-text images on white backgrounds.Two example images are shown in Figure 19.
  • A.2. Models: The comparison includes five ResNet-based contrastive CLIP models, four CLIP-ViT models, and an autoregressive LM RN50 trained on the same data and epochs as CLIP.CLIP-RN scales ResNet models up to roughly 64× computation, while CLIP-ViT includes ViT-B/32, ViT-B/16, ViT-L/14, and a 336×336 fine-tuned ViT-L/14.
  • A.2. Models: Baselines span EfficientNet, Instagram-pretrained ResNeXt, BiT, ImageNet-21k ViT, SimCLRv2, BYOL, MoCo, VirTex, and original ResNet checkpoints.The collection covers supervised, self-supervised, multimodal, and standard convolutional or transformer architectures, with publicly unavailable variants excluded.
  • A.2. Models: VirTex provides a related caption-trained comparison, using a design similar to CLIP-AR but training on a 1000× smaller MSCOCO caption dataset.The comparison also includes MoCo-v1, MoCo-v2, and ResNet-50, ResNet-101, and ResNet152 checkpoints.
  • A.3. Evaluation: Evaluation uses penultimate-layer image features and scikit-learn L-BFGS logistic regression, selecting hyperparameters on validation data or a training split when validation labels are unavailable.For final results, available validation data is recombined with training data before measuring performance on the unused split.

A.4. Results · B. Zero-Shot Prediction

CLIP’s best linear-probe model achieved state-of-the-art performance on 21 of 27 datasets, while the paper also reports dataset-level zero-shot scores and qualitative predictions across 36 classifiers.

  • A.4. Results: 21 of 27 datasets fell within the 99.5% Clopper-Pearson confidence interval of the top score using CLIP ViT-L/14 at 336-by-336 resolution.For many datasets, CLIP significantly outperformed other models, supporting the advantage of natural-language supervision over traditional pre-training.
  • A.4. Results: The linear-probe evaluation examined datasets listed in Table 9, using online resources for Birdsnap and Kinetics700 available when the paper was written.The dataset resources therefore reflect their online availability at the time of the study.
  • A.4. Results: Table 10 reports linear-probe performance for various pre-trained models across the 27 evaluated datasets.Scores within each dataset’s 99.5% Clopper-Pearson confidence interval of the top score are shown in bold.
  • A.4. Results: The STL10 scores were updated after fixing a CUDA-related bug in the previous version of the paper.This note qualifies the reported STL10 comparison as revised from an earlier version.
  • A.4. Results: Figure 20 plots the individual linear-probe results for all 27 datasets using the scores from Table 10.The plotted comparison provides a dataset-by-dataset view of the linear-probe evaluation.
  • B. Zero-Shot Prediction: Zero-shot performance is summarized through random predictions from 36 classifiers in Figure 21 and dataset-level scores in Table 11 and Figure 22.Figure 21 displays the top five class probabilities, class-representation text, and whether each prediction matches the ground-truth label; Figure 22 compares zero-shot CLIP with linear-probe ResNet performance.

C. Duplicate Detector

The authors abandoned embedding-space nearest neighbors because semantic similarity produced many false duplicate matches. They instead built a synthetic-augmentation-based detector using a modified ResNet-50.

  • Limitations: Embedding-space nearest neighbors produced false positives because semantically similar but distinct objects, such as soccer balls and flowers, could have nearly perfect similarity.The model’s feature space was weighted heavily toward semantic similarity, limiting its usefulness for duplicate detection.
  • Method: The custom detector trained on images transformed by random crops, zooms, aspect-ratio distortion, rescaling, rotations, JPEG compression, HSV jitter, and varied interpolation.These manipulations formed a synthetic data augmentation pipeline designed to address the embedding-space detector’s failures.
  • Implementation: The detector used a ResNet-50 modified with anti-aliasing, weight normalization, and GELU activations.Weight normalization replaced batch normalization to avoid duplicate-related information leakage through batch statistics.
  • Dataset comparison: CLIP trained on YFCC100M had similar average performance and numbers of wins to an equally sized WIT subset, despite dataset-specific differences.The comparison covered zero-shot and linear-classifier evaluations; YFCC100M performed best and worst relative to WIT on different datasets.

D. Dataset Ablation on YFCC100M · E. Selected Task and Dataset Results · E.1. Image and Text Retrieval

CLIP performs similarly when trained on equally sized filtered subsets of YFCC100M and WIT, suggesting robustness to the choice of reasonably filtered image-text data while warranting a caveat about dataset overlap. The paper then examines selected task results, including zero-shot image and text retrieval as a sanity check for the pre-training objective.

  • D. Dataset Ablation on YFCC100M: YFCC100M and WIT perform similarly on average across the evaluation suite when used to train equally sized filtered-subset models for 32 epochs.The comparison is made after transfer performance begins to plateau because of overfitting.
  • D. Dataset Ablation on YFCC100M: The ablation suggests that CLIP can use any reasonably filtered collection of paired text and image data.This finding is consistent with reported positive results for contrastive pre-training on medical imaging and only slight gains from JFT300M over YFCC100M in noisy-student self-training.
  • D. Dataset Ablation on YFCC100M: The YFCC100M-versus-WIT comparison may underestimate performance differences because the filtered WIT subset includes YFCC100M data.The authors do not expect this overlap to matter substantially because YFCC100M comprises only 3.7% of WIT and adding it to WIT’s existing blend did not noticeably change model performance.
  • E. Selected Task and Dataset Results: The selected-task section summarizes detailed performance for specific groups of tasks, datasets, and evaluation settings after the main body’s focus on overall results.This organization reflects the large variety of datasets and experiments considered in the work.
  • E.1. Image and Text Retrieval: Zero-shot image and text retrieval tests whether CLIP transfers successfully to the image-text retrieval task used during pre-training.The authors describe this evaluation as an important sanity check and proof of concept for representation and task learning.
  • E.1. Image and Text Retrieval: The retrieval evaluation covers both text retrieval and image retrieval using zero-shot transfer performance.The results are presented in Table 13, but the supplied passage does not provide the table’s numerical outcomes.

E.2. Optical Character Recognition

CLIP’s OCR performance is variable across five direct and indirect OCR datasets, depending on image domain and whether it recognizes numbers or words. It is strongest on digitally rendered, word-heavy tasks and can support higher-level semantic representations of text.

  • Motivation: ImageNet representations are not sufficiently fine-grained for OCR, motivating the use of custom OCR-engine outputs and features in systems requiring this capability.Prior visualizations showed ImageNet features responding to text presence, but not with enough detail for direct optical character recognition (Zeiler & Fergus, 2014).
  • Evaluation: The evaluation covers five OCR-related datasets: MNIST, SVHN, and IIIT5K test character or word recognition, while Hateful Memes and SST-2 test OCR use in semantic tasks.Table 14 reports accuracy on the test set for all datasets except Hateful Memes, which reports ROC AUC on the dev set.
  • Results: CLIP’s OCR performance varies with image domain and text type, performing strongest on Hateful Memes and SST-2, where text is digitally rendered and mostly words.On IIIT5K’s natural images of individually cropped words, zero-shot CLIP performs less strongly and is similar to Jaderberg et al. (2015).
  • Results: 80.5% accuracy on rendered SST-2 sentences matches the 80% continuous-bag-of-words baseline using GloVe vectors pretrained on 840 billion tokens.A linear classifier is fitted on CLIP’s representations, testing whether low-level OCR can be converted into a higher-level representation.

E.3. Action Recognition in Videos · E.4. Geolocalization

CLIP transfers strongly to video action recognition despite single-frame evaluation constraints, while its location-recognition ability motivates evaluation on Country211 and IM2GPS. The comparisons also highlight broader language supervision as a potential advantage over ImageNet’s noun-only labels, while leaving architectural and data differences unresolved.

  • E.3. Action Recognition in Videos: CLIP may transfer better because image-text pretraining supervises common nouns, proper nouns, verbs, and adjectives, whereas ImageNet-1K labels only common nouns.The passage presents this broader conceptual supervision as the motivation for comparing CLIP and ImageNet models on video action recognition.
  • E.3. Action Recognition in Videos: The action-recognition experiment evaluates verb-recognition datasets including UCF-101 and Kinetics-700, but CPU-based linear evaluation requires reducing each video to one center frame.This single-frame setup makes linear CLIP and linear NS ENet-L2 comparisons unlike prior video methods.
  • E.3. Action Recognition in Videos: CLIP matches the best prior UCF-101 result under linear probing, outperforms the evaluated models, and surpasses the fine-tuned I3D baseline on Kinetics-700.The supplied passages report these comparisons but provide no numerical scores.
  • E.3. Action Recognition in Videos: CLIP’s zero-shot video recognition averages predictions across all frames and improves on the prior S3D state of the art by 10 points.The zero-shot evaluation avoids a separate training stage; the prior S3D model used automatically extracted captions from 100 million instructional videos.
  • E.3. Action Recognition in Videos: The action-recognition comparison is not causally isolated because architecture, training-data distribution, dataset size, and compute differ across models.The authors state that further work is needed to identify which design choices drive high performance.
  • E.4. Geolocalization: CLIP’s observed ability to recognize places motivates the new Country211 benchmark and an additional comparison on the established IM2GPS test set.IM2GPS is used to compare with prior geolocalization work, while Country211 is evaluated throughout the paper as a new benchmark.
  • E.4. Geolocalization: For IM2GPS, CLIP estimates GPS coordinates by selecting the nearest image in a search procedure because the benchmark is framed as regression.The supplied passage truncates before specifying the complete procedure or results.

E.5. Robustness to Distribution Shift · F. Model Hyperparameters

Zero-shot CLIP improves the state of the art on 5 of 7 distribution-shift datasets, with ImageNet-R’s gains potentially reflecting CLIP’s pre-training distribution. The appendix also tabulates common, CLIP-ResNet, and CLIP-ViT hyperparameters.

  • E.5. Robustness to Distribution Shift: Zero-shot CLIP improves the state of the art on 5 of 7 datasets: ImageNet-R, ObjectNet, ImageNet-Sketch, ImageNet-Vid, and Youtube-BB.The results are reported per dataset in Table 16 and compared with Taori et al. (2020)’s evaluation suite.
  • E.5. Robustness to Distribution Shift: Table 16 provides per-dataset robustness results against current state-of-the-art results from Taori et al. (2020)’s evaluation suite.
  • E.5. Robustness to Distribution Shift: ImageNet-R shows particularly large gains, likely because CLIP’s pre-training distribution included substantial creative content.A similar pattern was documented for Instagram-pretrained ResNeXt models by Taori et al. (2020).
  • F. Model Hyperparameters: Table 18 lists common CLIP hyperparameters.
  • F. Model Hyperparameters: Table 19 lists CLIP-ResNet hyperparameters.
  • F. Model Hyperparameters: Table 20 lists CLIP-ViT hyperparameters.
Loading 2103.00020v1…