PaperlayerView exact revision on arXiv

Source-linked AI summary

Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision

Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V. Le, Yunhsuan Sung, Zhen Li, Tom Duerig

arXiv:2102.05918v2cs.CVcs.CLcs.LG

TL;DR

Visual and vision-language representation learning depends on costly curation, limiting dataset scale. ALIGN trains a dual encoder on over one billion noisy image-text pairs and achieves state-of-the-art results across retrieval and visual tasks, including 76.4% zero-shot ImageNet top-1 accuracy.

  • Problem

    Visual and vision-language pre-training datasets require costly annotation, cleaning, and balancing, keeping them far smaller than large-scale vision and NLP corpora.

  • Method

    ALIGN uses minimally filtered noisy image-text pairs to train dual image and text encoders with contrastive learning.

  • Results

    76.4% top-1 accuracy in ImageNet zero-shot classification, alongside state-of-the-art Flickr30K and MSCOCO retrieval and strong visual-task performance.

  • Takeaways & Limitations

    Large-scale noisy image-text data can support visual and vision-language representations that match or outperform systems trained with curated or labeled data.

  • Takeaways & Limitations

    Further analysis is needed to assess harmful text, stereotype reinforcement, and mislabeled sensitive content before practical use.

Abstract

from arXiv · show

Pre-trained representations are becoming crucial for many NLP and perception tasks. While representation learning in NLP has transitioned to training on raw text without human annotations, visual and vision-language representations still rely heavily on curated training datasets that are expensive or require expert knowledge. For vision applications, representations are mostly learned using datasets with explicit class labels such as ImageNet or OpenImages. For vision-language, popular datasets like Conceptual Captions, MSCOCO, or CLIP all involve a non-trivial data collection (and cleaning) process. This costly curation process limits the size of datasets and hence hinders the scaling of trained models. In this paper, we leverage a noisy dataset of over one billion image alt-text pairs, obtained without expensive filtering or post-processing steps in the Conceptual Captions dataset. A simple dual-encoder architecture learns to align visual and language representations of the image and text pairs using a contrastive loss. We show that the scale of our corpus can make up for its noise and leads to state-of-the-art representations even with such a simple learning scheme. Our visual representation achieves strong performance when transferred to classification tasks such as ImageNet and VTAB. The aligned visual and language representations enables zero-shot image classification and also set new state-of-the-art results on Flickr30K and MSCOCO image-text retrieval benchmarks, even when compared with more sophisticated cross-attention models. The representations also enable cross-modality search with complex text and text + image queries.

1. Introduction

ALIGN scales visual and vision-language representation learning with over one billion noisy image alt-text pairs, avoiding the costly curation required by existing datasets. A simple dual-encoder contrastive objective aligns image and text representations, enabling strong retrieval and zero-shot classification results.

  • Motivation: Existing visual and vision-language pre-training datasets depend on costly curation, annotation, semantic parsing, cleaning, or balancing, limiting vision-language datasets to roughly 10M examples.Vision datasets include ImageNet, OpenImages, and JFT-300M, while vision-language datasets include Conceptual Captions, Visual Genome Dense Captions, and ImageBERT.
  • Dataset: ALIGN uses over one billion noisy image alt-text pairs with only simple frequency-based filtering instead of complex cleaning and post-processing.The dataset follows the Conceptual Captions collection procedure but omits its complex filtering and post-processing steps.
  • Method: A simple dual-encoder architecture learns shared visual-language embeddings with a contrastive normalized-softmax loss that brings matched pairs together and separates non-matched pairs.The model is named ALIGN, short for A Large-scale ImaGe and Noisy-text embedding.
  • Results: Over 7%: ALIGN outperforms the previous state-of-the-art method in most zero-shot and fine-tuned R@1 metrics on Flickr30K and MSCOCO.The aligned representations support cross-modality matching and retrieval tasks.
  • Results: 76.4% top-1 accuracy: ALIGN achieves this result on ImageNet zero-shot image classification by feeding class names into the text encoder.The aligned image and text representations also support cross-modal search, including image-to-text, text-to-image, and joint image-plus-text queries.

2. Related Work

Related work spans labeled, self-supervised, semi-supervised, and image-caption approaches to visual representation learning. Vision-language models increasingly use cross-modal attention, but ALIGN instead adopts a simple VSE-style architecture and raw alt-text supervision, differing from CLIP’s curated concept-based data collection.

  • Visual representation learning: Visual representations are commonly pre-trained on large-scale labeled datasets, with self-supervised and semi-supervised learning studied as alternatives.These approaches target classification or retrieval representations without relying exclusively on labeled pre-training data.
  • Image-caption representation learning: Image-caption methods learn visual representations by predicting captions from images, but prior work is limited to small datasets such as Flickr and COCO Captions.This caption-supervision direction directly inspires ALIGN’s approach.
  • Vision-language representation learning: Vision-semantic embeddings evolved from basic VSE models toward systems using object detectors, dense feature maps, multi-attention layers, and cross-modal attention.Cross-modal attention models achieve superior image-text matching performance but are orders of magnitude slower for real-world retrieval systems.
  • Vision-language representation learning: ALIGN retains the simplest VSE form while outperforming previous cross-attention models on image-text matching benchmarks, whereas CLIP uses different encoders and curated high-frequency visual concepts from English Wikipedia.ALIGN follows the natural distribution of raw image-text alt-text pairs, contrasting with CLIP’s allowlist-based dataset construction.

3. A Large-Scale Noisy Image-Text Dataset

The section introduces a large-scale dataset of raw English image–alt-text pairs, trading curation quality for scale. It also describes image- and text-based filters used to remove unsuitable images, duplicates, and noisy captions.

  • Dataset construction: The dataset scales beyond existing resources by using raw English alt-text image pairs and relaxing Conceptual Captions’ heavy filtering and post-processing.The authors explicitly trade quality for scale to support larger-scale visual and vision-language representation learning.
  • Image-based filtering: The dataset removes duplicates and near-duplicates of downstream test images, including those from ILSVRC-2012, Flickr30K, and MSCOCO.This prevents training on evaluation images.
  • Text-based filtering: Text filtering excludes alt-texts shared by more than 10 images, containing rare tokens, or shorter than 3 or longer than 20 unigrams.These rules target captions that are often irrelevant to image content or contain noisy metadata-like strings.

4. Pre-training and Task Transfer

ALIGN pre-trains dual image-text encoders with a cosine-similarity objective using matched pairs as positives and in-batch pairs as negatives. The learned representations are transferred to retrieval and visual classification tasks, including zero-shot and fine-tuned settings.

  • Pre-training: ALIGN pairs an EfficientNet image encoder with a BERT text encoder and combines their representations using cosine similarity.The text encoder uses a 100k-wordpiece vocabulary and [CLS] token embeddings.
  • Pre-training: Matched image-text pairs serve as positives, while other image-text combinations within each training batch serve as negatives under normalized softmax loss.The objective sums image-to-text and text-to-image classification losses.
  • Pre-training: ALIGN increases in-batch negative effectiveness by concatenating embeddings across computing cores, while learning the temperature jointly with model parameters.Both image and text embeddings are L2-normalized before similarity-based optimization.
  • Task Transfer: ALIGN is evaluated on image-to-text and text-to-image retrieval using Flickr30K, MSCOCO, and the semantically annotated CxC extension, with and without fine-tuning.CxC adds human similarity judgments for caption-caption, image-image, and image-caption pairs.
  • Task Transfer: ALIGN is also tested in zero-shot visual classification on ImageNet ILSVRC-2012 and related ImageNet-R, ImageNet-A, and ImageNet-V2 variants.These variants share the same or a subset of ImageNet classes while differing in image characteristics or difficulty.
  • Task Transfer: The image encoder is transferred to ImageNet and smaller fine-grained datasets, reporting both frozen-encoder top-layer training and full fine-tuning settings.The fine-grained datasets include Oxford Flowers-102, Oxford-IIIT Pets, Stanford Cars, and Food101.

5. Experiments and Results

ALIGN is evaluated across cross-modal retrieval, zero-shot image classification, and ImageNet transfer, achieving state-of-the-art results across retrieval benchmarks and strong classification performance. Its gains are largest for inter-modal retrieval, while intra-modal improvements are less pronounced.

  • Experimental Setup: ALIGN uses EfficientNet-L2 and BERT-Large as its primary image and text encoders, with images trained at 289 × 289 resolution.Models are trained from scratch using an opensourced EfficientNet implementation and BERT.
  • Flickr30K and MSCOCO Retrieval: ALIGN achieves state-of-the-art results on all Flickr30K and MSCOCO retrieval metrics, improving zero-shot image retrieval by more than 7% over CLIP.Fine-tuned ALIGN also outperforms existing methods, including models using more complex cross-modal attention layers.
  • Crisscrossed Captions Retrieval: ALIGN achieves state-of-the-art results on all CxC retrieval metrics, including +22.2% R@1 for image-to-text and 20.1% R@1 for text-to-image.It also improves over the previous state of the art on the SITS task by 5.7%.
  • Limitations: ALIGN is less impressive on intra-modal CxC tasks despite substantially stronger inter-modal performance, likely because its objective emphasizes cross-modal matching.The paper suggests multitask learning as a possible way to produce more balanced representations.
  • Zero-Shot Classification: ALIGN supports zero-shot image classification by feeding classnames into the text encoder and shows robustness across ImageNet and its variants.The comparison uses the same prompt ensembling method as CLIP.
  • ImageNet Transfer: 85.5% top-1 accuracy is achieved by ALIGN with frozen visual features on ImageNet, slightly outperforming CLIP and establishing a state-of-the-art result.After fine-tuning, ALIGN surpasses BiT and ViT but remains below Meta Pseudo Labels; it also saves 44% FLOPS versus comparable EfficientNet-L2 methods.

6. Ablation Study

The ablations show that ALIGN benefits from scaling image and text backbones, larger embedding dimensions, and abundant in-batch negatives. Data-size scaling can overcome increased noise, with larger noisy ALIGN subsets catching up to and surpassing cleaned Conceptual Captions data.

  • Backbone scaling: Model quality improves with larger image and text backbones, while ImageNet KNN saturates from BERT-Base to BERT-Large with EfficientNet-B7 and EfficientNet-L2.Scaling image capacity matters more for vision tasks: EfficientNet-L2 with BERT-Mini outperforms EfficientNet-B7 with BERT-Large.
  • Architecture parameters: Higher embedding dimensions improve performance, motivating dimensions that scale with the EfficientNet backbone, including 1376 for L2.The baseline uses embedding dimension 640.
  • Architecture parameters: Using fewer in-batch negatives degrades performance, while the baseline uses all negatives and a learnable softmax temperature converged to about 1/64.The tested reductions retain 50% or 25% of in-batch negatives.
  • Training data scaling: The dataset ablation trains models from scratch on full ALIGN data, 10% sampled ALIGN data, and CC-3M, with CC-3M receiving one-tenth the default training steps.The evaluated configurations are EfficientNet-B7 + BERT-base and EfficientNet-B3 + BERT-mini.
  • Training data scaling: ALIGN data performs worse than cleaned CC-3M at equal 3M size, but quality rapidly catches up at 6M and 12M and surpasses Conceptual Captions at only 4x size.The comparison uses randomly sampled ALIGN subsets of 3M, 6M, and 12M images against cleaned CC-3M on the B7+BERT-base model.

7. Analysis of Learned Embeddings

ALIGN embeddings support precise image retrieval from detailed, fine-grained, and instance-level text queries, while combined image-and-text queries demonstrate compositionality across vision and language. These capabilities enable multimodal search paradigms beyond text-only or image-only queries.

  • Text-to-image retrieval: ALIGN retrieves precise images for detailed scene descriptions and fine-grained or instance-level concepts, including landmarks and artworks.The demonstration uses a 160M-image index of CC-BY licensed images separate from training.
  • Multimodal retrieval: Combined image-and-text queries add the normalized image and text embeddings before retrieving relevant images by similarity.A text-to-image scale of 2 and image scale of 1 performed best, although equal weighting also worked well.
  • Compositional search: Image-plus-text retrieval demonstrates compositionality across vision and language and enables search with multimodal queries.The approach supports queries that would be difficult to express using only text or only an image.
  • Compositional search: Examples include retrieving the “Australia” or “Madagascar” equivalent of pandas and changing black shoes to identically-looking beige shoes.These examples use image-plus-text queries to modify or specify visual concepts.

8. Multilingual ALIGN Model

ALIGN extends its language-agnostic noisy-data pipeline to over 100 languages, producing a multilingual model trained on 1.8B image-text pairs. ALIGNmling achieves strong zero-shot multilingual retrieval, outperforming M3P across languages and approaching fine-tuned baselines except on Czech.

  • Multilingual training: ALIGN extends its language-agnostic data-processing pipeline to over 100 languages and matches the English dataset size with 1.8B image-text pairs.The resulting multilingual model is called ALIGNmling.
  • Evaluation: Multi30k evaluates zero-shot multilingual image-text retrieval in English, German, French, and Czech using 31,783 images with language-specific caption counts.The dataset has five captions per image in English and German, and one caption per image in French and Czech.
  • Results: +57.8 absolute mR improvement on fr is ALIGNmling’s largest gain over zero-shot M3P, while ALIGNmling outperforms M3P on all languages.ALIGNmling is comparable to fine-tuned M3P and UC2 except on cs, and performs slightly worse than English-only ALIGNEN on en.

9. Conclusion

The paper presents ALIGN, a simple dual-encoder trained with contrastive loss on large-scale noisy image-text data with minimal cleaning. ALIGN supports cross-modal retrieval and performs competitively with or better than state-of-the-art models on visual-only downstream tasks.

  • Conclusion: ALIGN scales visual and vision-language representation learning using large-scale noisy image-text data, avoiding heavy curation and annotation with minimal frequency-based cleaning.The method is designed to leverage noisy data without expensive filtering or post-processing.
  • Conclusion: ALIGN uses a simple dual-encoder model trained with a contrastive loss.The architecture learns from paired images and text through contrastive training.
  • Conclusion: ALIGN supports cross-modal retrieval and significantly outperforms SOTA VSE and cross-attention vision-language models.Its retrieval capability is reported across multimodal retrieval settings.
  • Conclusion: On visual-only downstream tasks, ALIGN is comparable to or outperforms SOTA models trained with large-scale labeled data.The comparison concerns downstream visual tasks rather than multimodal retrieval.

10. Social Impacts and Future Work

Before practical deployment, the model requires additional analysis of harmful alt-text content, fairness risks, and demographic or cultural skew. Its unintended use for surveillance or other nefarious purposes should also be prohibited.

  • Future analysis: Additional analysis is needed before practical use because harmful text in alt-texts could reinforce existing harms.The paper identifies harmful alt-text content as a potential risk requiring further consideration.
  • Fairness: Data balancing may be required to prevent web data from reinforcing stereotypes and to address skew from demographic and cultural distributions.The analysis should cover humans and cultural items such as clothing, food, and art, particularly for production use.
  • Misuse: Unintended misuse of the models for surveillance or other nefarious purposes should be prohibited.

A. Remove Near-Duplicate Test Images from Training Data · B. Evaluation on SimLex-999

The paper removes near-duplicate downstream test images from ALIGN’s training data using embedding-based clustering and a cosine-similarity threshold. It then evaluates ALIGN word representations on SimLex-999, finding performance between Picturebook and GloVe while showing stronger results for nouns and concrete categories than for adjectives and less concrete categories.

  • B. Evaluation on SimLex-999: ALIGN achieves the highest performance in the hard category, where similarity is difficult to distinguish from relatedness.This supports the observation that image-based word embeddings are less likely than text-based distributional methods to confuse similarity with relatedness.
  • A. Remove Near-Duplicate Test Images from Training Data: A separate high-quality image embedding model and 4K k-means clusters identify candidate near-duplicate pairs across ALIGN training images and downstream test images.For each query and index image, the method retrieves their top-10 nearest clusters by embedding distance.
  • A. Remove Near-Duplicate Test Images from Training Data: Images sharing a bucket are marked near-duplicated when their embedding cosine similarity exceeds 0.975.Buckets represent all combinations of 3 clusters out of the 10 nearest clusters, and the threshold is trained using human-rated and augmented synthesized data.
  • B. Evaluation on SimLex-999: ALIGN word representations are evaluated on SimLex-999, which compares word similarity across 999 word pairs and reports nine subtasks.The evaluation follows Kiros et al. (2018) and includes subsets such as all and adjectives.
  • B. Evaluation on SimLex-999: ALIGN performs better overall than Picturebook but slightly worse than GloVe on SimLex-999.The comparison uses Spearman’s ρ across the reported subtasks.
  • B. Evaluation on SimLex-999: ALIGN’s word embeddings outperform on nouns and most concrete categories but underperform on adjectives and less concrete categories relative to Picturebook.This category pattern is similar to Picturebook’s, despite ALIGN’s overall position between Picturebook and GloVe.
  • B. Evaluation on SimLex-999: ALIGN forms word embeddings by feeding a word’s wordpieces into the model and using the [CLS] representation before the projection layers.This implementation accommodates words split into multiple wordpiece tokens.
Loading 2102.05918v2…