PaperlayerView exact revision on arXiv

Source-linked AI summary

Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts

Soravit Changpinyo, Piyush Sharma, Nan Ding, Radu Soricut

arXiv:2102.08981v2cs.CVcs.CL

TL;DR

V+L datasets are often constrained by filters inherited from narrower downstream tasks, limiting their scale and diversity. The paper introduces CC12M by relaxing CC3M’s pipeline and evaluates it against CC3M across downstream tasks, finding state-of-the-art results on nocaps and Conceptual Captions.

  • Problem

    V+L pre-training datasets are limited in scale and visual diversity because collection requirements inherited from downstream tasks discard potentially useful image-text pairs.

  • Method

    The paper constructs CC12M by relaxing multiple CC3M filters and retaining rawer Web alt-texts for V+L pre-training.

  • Results

    CC12M consistently outperforms CC3M across generation and matching evaluations, achieving state-of-the-art results on nocaps and Conceptual Captions.

  • Takeaways & Limitations

    Scaling and diversifying V+L pre-training data, including noisy Web-scale image-text pairs, benefits long-tail recognition on generation and matching tasks.

  • Takeaways & Limitations

    The analysis does not use mouse traces synchronized with the text, although the dataset contains them.

Abstract

from arXiv · show

The availability of large-scale image captioning and visual question answering datasets has contributed significantly to recent successes in vision-and-language pre-training. However, these datasets are often collected with overrestrictive requirements inherited from their original target tasks (e.g., image caption generation), which limit the resulting dataset scale and diversity. We take a step further in pushing the limits of vision-and-language pre-training data by relaxing the data collection pipeline used in Conceptual Captions 3M (CC3M) [Sharma et al. 2018] and introduce the Conceptual 12M (CC12M), a dataset with 12 million image-text pairs specifically meant to be used for vision-and-language pre-training. We perform an analysis of this dataset and benchmark its effectiveness against CC3M on multiple downstream tasks with an emphasis on long-tail visual recognition. Our results clearly illustrate the benefit of scaling up pre-training data for vision-and-language tasks, as indicated by the new state-of-the-art results on both the nocaps and Conceptual Captions benchmarks.

1. Introduction

Vision-and-language pre-training datasets remain limited in scale and visual diversity, motivating CC12M, a higher-recall dataset created by relaxing CC3M’s filters. Benchmarking shows that scaling pre-training benefits generation, novel-object captioning, and zero-shot image retrieval, with state-of-the-art results on nocaps and Conceptual Captions.

  • Motivation: V+L datasets remain limited in scale and visual diversity, with many popular resources built from a narrow set of COCO images.These limitations make it difficult for V+L models to perform adequately in the wild.
  • Existing data: CC3M provides a promising automatically collected source of 3.3M image-caption pairs derived from Web alt-texts without additional human intervention.Its filtering and postprocessing pipeline produces data effective for image captioning and V+L pre-training.
  • CC12M: CC12M relaxes multiple CC3M image and text filters to produce a less precise but 4x larger image-text dataset with wider visual-concept coverage.The dataset is designed for V+L pre-training rather than exclusively for high-precision image captioning.
  • Evaluation: The experiments compare CC12M and CC3M across vision-to-language generation and vision-and-language matching, emphasizing long-tail recognition.The reported gains include image captioning, novel object captioning, and zero-shot image retrieval.
  • Results: CC12M delivers state-of-the-art results on the nocaps and Conceptual Captions benchmarks.The contribution emphasizes consistent superiority over CC3M on downstream generation and matching evaluations.

2. Vision-and-Language Pre-Training Data

CC12M extends the CC3M Web-alt-text pipeline by relaxing unimodal filters and retaining rawer text, prioritizing recall for V+L pre-training. Compared with CC3M, it is larger, more diverse, longer-captioned, and lower-precision, exposing a precision–recall tradeoff while adding long-tail concepts.

  • Data pipeline: CC3M applies image, text, and image-text filtering to create clean, high-precision captions, but this reduces recall by discarding useful pairs.CC12M retains image-text filtering while relaxing unimodal filters.
  • Text transformation: CC12M keeps alt-texts relatively raw, omitting hypernymization and digit substitution while retaining person-name substitution for privacy.The design is motivated by the possibility that more difficult pre-training data can improve transferability.
  • Basic statistics: CC12M contains 12.4M image-text pairs, about 4x the size of CC3M, with a lower token-to-type ratio and longer average captions.The lower ratio indicates a longer-tail distribution and greater diversity of captured concepts.
  • Quality: CC12M’s estimated caption precision is 76.6%, compared with 90.3% for CC3M, illustrating the precision–recall tradeoff from relaxing filters.The jellyfish example is included in CC12M despite being filtered out by CC3M’s stricter criteria.
  • Visual concept distribution: CC12M introduces long-tail concepts absent or rare in CC3M, including mangosteen, sumo, pokemon, chevrolet, and mehndi.The distribution is attributed to increased scale and the absence of fine-grained entity hypernymization.
  • Pre-training tasks: The pre-training setup covers image captioning for vision-to-language generation and visual-linguistic matching for vision-and-language understanding.These are the two principal task families used to study CC12M’s downstream utility.

3. Evaluating Vision-and-Language PreTraining Data

The evaluation tests CC12M and CC3M as pre-training sources across vision-to-language generation and vision-and-language matching, emphasizing long-tail recognition and out-of-distribution generalization.

  • Evaluation focus: The study compares CC12M and CC3M on image captioning, novel object captioning, and image retrieval tasks.The experiments target downstream settings where pre-training and downstream objectives match, including zero-shot retrieval.
  • Vision-to-language generation: Image captioning pre-training predicts a target caption from image features using standard cross-entropy loss.
  • Vision-to-language generation: Nocaps evaluates novel object captioning when COCO training images and Open Images evaluation images cover substantially different object distributions.Open Images covers 600 object classes, compared with 80 in COCO.
  • Vision-and-language matching: Vision-and-language matching predicts whether an image and text are paired, using original pairs as positives and other minibatch pairs as negatives.Training uses a contrastive softmax loss over image-text pairs.
  • Vision-and-language matching: The retrieval evaluation includes Flickr30K, zero-shot Flickr30K, and Localized Narratives Flickr30K, with Recall@1, Recall@5, and Recall@10 as metrics.Zero-shot retrieval discards target-task training data and applies the pre-trained model without fine-tuning.
  • Modeling: The model represents images with global, regional, and tag features and uses Transformer-based encoders, with a decoder retained for image captioning but discarded for matching.For matching, mean-pooled image and text representations are combined before softmax.

4. Experimental Results

Across generation and retrieval experiments, CC12M generally outperforms CC3M, with the strongest gains appearing after fine-tuning and on long-tail or zero-shot settings.

  • Vision-to-language generation: CC12M outperforms CC3M by +14.2 CIDEr points after fine-tuning on nocaps, while combining CC3M and CC12M adds another +2.8 points.The comparison is between fine-tuned models using the respective pre-training datasets.
  • Vision-to-language generation: CC12M succeeds on fine-grained nocaps concepts despite their textual out-of-domain status, with gains largely attributed to out-of-domain performance.The result holds despite nocaps annotator priming for displayed fine-grained object classes.
  • Vision-to-language generation: CC12M achieves higher CIDEr than CC3M on Localized Narratives, but its gain is smaller than on nocaps because longer texts make novel-concept insertion harder.Localized Narratives annotations also use more generic terms because they do not use priming.
  • Conceptual Captions benchmark: Fine-tuning a CC12M-pre-trained model on CC3M raises dev CIDEr from 100.9 to 105.4, a gain of +4.5 CIDEr points.
  • Vision-and-language matching: On image retrieval, CC12M significantly outperforms CC3M across Flickr30K and Localized Narratives, while combining both datasets performs best.Both datasets improve over from-scratch training in R1.
  • Vision-and-language matching: Zero-shot Flickr30K retrieval reaches 42.5 R1 with CC12M and 47.1 R1 with CC3M+CC12M, despite the model being smaller than the compared ViLBERT system.CC3M alone scores 35.4 R1 versus ViLBERT’s 31.9 R1.

5. Related Work

Prior vision-and-language pre-training work primarily uses existing image-text datasets and BERT-influenced objectives. This paper studies simpler generation and matching objectives while emphasizing long-tail visual recognition.

  • Most vision-and-language pre-training uses image-text resources, especially captioning datasets such as CC3M.
  • BERT-inspired objectives extend masked language modeling to visual inputs and use next sentence prediction analogues for multimodal learning.
  • The paper studies generation and matching separately, including vision-to-language generation with an encoder-decoder rather than only encoder-only pre-training.
  • Long-tail recognition requires adaptation across visual and textual distribution shifts, including nocaps and zero-shot image retrieval.

6. Conclusion

The paper introduces CC12M by extending the CC3M pipeline and argues that larger, more diverse pre-training data benefits vision-and-language tasks. It particularly emphasizes long-tail recognition and noisy Web-scale image-text pairs.

  • CC12M extends the CC3M data-collection pipeline as a new vision-and-language pre-training resource.
  • Scale and diversity of pre-training data matter for both generation and matching, especially for long-tail recognition such as nocaps.
  • Noisy Web-scale image-text pairs are presented as a promising direction for vision-and-language research.

A. Broader Impact

CC12M is intended to support multiple vision-and-language tasks through broader long-tail visual-concept coverage. The paper also highlights risks from Web data, dataset bias, and in-domain evaluation setups.

  • CC12M provides higher long-tail visual-concept coverage than previous resources, including CC3M.
  • The authors expect models trained on CC12M to be more robust in the wild.
  • The paper encourages downstream setups that shift from in-domain data toward out-of-domain or in-the-wild evaluation.
  • Web-curated datasets can contain unsuitable content and unintended privacy leakage, which the paper mitigates through image and text filtering and person-name replacement.
  • Dataset biases may be amplified by machine learning models, so users and trained systems should remain aware of these risks.

B.1. Out-of-domain (OOD) visual concepts on an expanded list of datasets

The analysis measures coverage of nocaps out-of-domain visual concepts across diverse vision-and-language datasets. CC12M has substantially greater OOD coverage and declines less sharply as the required concept frequency increases.

  • B.1. Out-of-domain (OOD) visual concepts on an expanded list of datasets: The analysis uses 394 nocaps out-of-domain classes as a proxy for OOD visual concepts across datasets, tasks, and usage stages.
  • B.1. Out-of-domain (OOD) visual concepts on an expanded list of datasets: Table 9 counts OOD-containing texts using both absolute frequency and normalized frequency per 1M text instances.
  • B.1. Out-of-domain (OOD) visual concepts on an expanded list of datasets: 6.7x median and 5.8x mean are CC12M’s OOD-instance advantages over the second-best CC3M, with CC12M remaining highest after normalization.
  • B.1. Out-of-domain (OOD) visual concepts on an expanded list of datasets: Figure 6 evaluates 25% and 50% CC12M subsets on nocaps novel object captioning and Flickr30K zero-shot retrieval.
  • B.1. Out-of-domain (OOD) visual concepts on an expanded list of datasets: Figure 5 plots OOD classes against minimum frequency K per 1M captions, showing sharper declines for other datasets than CC12M.

B.2. The impact of the dataset size

Subsampling CC12M shows that larger pre-training datasets improve downstream performance, with gains increasing from 25% to 50% and the full dataset.

  • 25% of CC12M (3.1M images) achieves performance similar to CC3M on nocaps and zero-shot Flickr30K image retrieval.The experiment compares randomly subsampled CC12M against CC3M on novel object captioning and zero-shot image retrieval.
  • Larger CC12M subsets produce a consistent larger-is-better trend across novel object captioning and zero-shot image retrieval.

C. Qualitive Results for Image Retrieval

CC12M pre-training improves qualitative Flickr30K image retrieval rankings, apparently by helping models recognize rare words; pre-training data style also affects transfer performance.

  • CC12M pre-training corrects image-retrieval rankings that are wrong for both the from-scratch model and the CC3M-pre-trained model.Figure 7 compares top-3 retrievals from models trained from scratch or pre-trained on CC3M or CC12M, then fine-tuned on Flickr30K.
  • The authors suspect that improved familiarity with rare words explains CC12M’s qualitative retrieval corrections.
  • LocNar Open Images underperforms CC3M and CC12M on nocaps, which the authors attribute to different narrative styles and less specific object naming.LocNar narratives differ from COCO Captions and nocaps, while nocaps annotation does not prime annotators to mention object names.

D.2.1 Masked Vision-to-Language Generation

The study compares masked vision-to-language objectives with image captioning and finds that masking design and rate strongly affect generation performance, while image captioning remains a strong baseline.

  • Objectives: The image-captioning objective predicts text from image features, while the visual-linguistic matching setup encodes concatenated image and text feature sequences with a Transformer encoder.The study uses simple objectives to keep the effect of pre-training data sources clean.
  • Masked objectives: Masked Language Modeling replaces selected text tokens with [MASK], random tokens, or unchanged tokens, then predicts the target sequence with a decoder.The mixed masking strategy uses proportions of 80%, 10%, and 10%, respectively.
  • Masked objectives: Masked Sequence to Sequence Modeling masks a contiguous text segment and sequentially predicts that segment with the decoder.
  • Results: Image captioning clearly outperforms masked vision-to-language pre-training at low masking rates, while remaining competitive overall.Image captioning is slightly below mlm[.8] in overall CIDEr but higher on out-of-domain CIDEr.
  • Results: High text masking rates make masked objectives more similar to image captioning, and low-rate configurations consistently underperform in the generation setup.The authors relate this trend to reducing models’ reliance on text during pre-training.
  • Auxiliary objectives: Adding masked object classification slightly improves nocaps performance, whereas adding visual-linguistic matching slightly reduces it.Both auxiliary losses are multiplied by 0.1 before being added to the main image-captioning loss.

E.1. Data Preprocessing and Feature Embedding

The implementation uses Transformer-based image, text, and decoder components with 512-dimensional embeddings, large-batch TPU training, and limited hyperparameter tuning under matched comparison resources.

  • Feature embedding: Image features include geometric information and up to 16 tag vectors, while visual-linguistic matching additionally encodes text subtoken vectors of size 512.
  • Model architecture: The image-captioning model uses a Transformer encoder-decoder, while the visual-linguistic matching model uses separate image and text Transformer encoders.
  • Model architecture: Transformer components use 6 layers, 512-dimensional embeddings, hidden size 1024, feedforward size 4096, and 8 attention heads.
  • Optimization: Training uses 32-core TPUs with total batch size 4096, Adam optimization, 20 warm-up epochs, and 0.95 learning-rate decay.
  • Hyperparameters: Pre-training uses minimal hyperparameter tuning, while fine-tuning tunes the initial learning rate and sometimes training steps for nocaps.
  • Experimental controls: Comparisons between pre-training settings receive similar computational resources, including comparisons between CC3M and CC12M.
Loading 2102.08981v2…