Source-linked AI summary
CoCa: Contrastive Captioners are Image-Text Foundation Models
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, Yonghui Wu
TL;DR
Vision foundation models need transferable capabilities across visual and vision-language tasks while existing pretraining paradigms specialize in different representations. CoCa combines contrastive and captioning objectives in one encoder-decoder model, achieving state-of-the-art results across a broad task spectrum with a single checkpoint. The model reaches 86.3% zero-shot top-1 accuracy on ImageNet and 91.0% with a finetuned encoder.
Problem
Existing vision and image-text pretraining paradigms provide different visual, textual, and multimodal capabilities rather than one unified image-text foundation model.
Method
CoCa uses a modified encoder-decoder with unimodal early decoder layers, multimodal cross-attending later layers, and jointly trained contrastive and captioning objectives.
Results
CoCa achieves state-of-the-art performance across visual recognition, crossmodal alignment, image captioning, and multimodal understanding tasks with a single pretrained checkpoint.
Takeaways & Limitations
CoCa bridges existing vision pretraining approaches by combining aligned unimodal representations and fused multimodal embeddings for broad downstream transfer.
Takeaways & Limitations
Further analysis is needed because CoCa uses the same pretraining data as previous methods, may remain vulnerable to untested image corruptions, and requires broader-impact exploration.
Abstract
from arXiv · showhide
Exploring large-scale pretrained foundation models is of significant interest in computer vision because these models can be quickly transferred to many downstream tasks. This paper presents Contrastive Captioner (CoCa), a minimalist design to pretrain an image-text encoder-decoder foundation model jointly with contrastive loss and captioning loss, thereby subsuming model capabilities from contrastive approaches like CLIP and generative methods like SimVLM. In contrast to standard encoder-decoder transformers where all decoder layers attend to encoder outputs, CoCa omits cross-attention in the first half of decoder layers to encode unimodal text representations, and cascades the remaining decoder layers which cross-attend to the image encoder for multimodal image-text representations. We apply a contrastive loss between unimodal image and text embeddings, in addition to a captioning loss on the multimodal decoder outputs which predicts text tokens autoregressively. By sharing the same computational graph, the two training objectives are computed efficiently with minimal overhead. CoCa is pretrained end-to-end and from scratch on both web-scale alt-text data and annotated images by treating all labels simply as text, seamlessly unifying natural language supervision for representation learning. Empirically, CoCa achieves state-of-the-art performance with zero-shot transfer or minimal task-specific adaptation on a broad range of downstream tasks, spanning visual recognition (ImageNet, Kinetics-400/600/700, Moments-in-Time), crossmodal retrieval (MSCOCO, Flickr30K, MSR-VTT), multimodal understanding (VQA, SNLI-VE, NLVR2), and image captioning (MSCOCO, NoCaps). Notably on ImageNet classification, CoCa obtains 86.3% zero-shot top-1 accuracy, 90.6% with a frozen encoder and learned classification head, and new state-of-the-art 91.0% top-1 accuracy on ImageNet with a finetuned encoder.
1 Introduction
Existing vision foundation models provide transferable visual representations, while image-text models add crossmodal alignment but do not unify these capabilities. CoCa addresses this by combining contrastive and captioning objectives in one image-text foundation model.
- Vision foundation models transfer across visual tasks but rely heavily on image annotations and lack free-form natural-language knowledge.
- Dual-encoder image-text models learn aligned visual and textual embeddings for zero-shot classification and image-text retrieval.
- CoCa unifies single-encoder, dual-encoder, and encoder-decoder paradigms through a modified encoder-decoder trained with contrastive and captioning losses.
- CoCa omits early decoder cross-attention for unimodal text representations and uses later cross-attending layers for multimodal image-text representations.
- 86.3% zero-shot ImageNet accuracy, 90.6% with a frozen encoder, and 91.0% after finetuning demonstrate strong transfer across recognition, retrieval, captioning, and multimodal tasks.The reported results also include 82.3% on VQA and 120.6 CIDEr on NoCaps after lightweight finetuning.
2 Related Work
Vision pretraining evolved from annotated visual encoders toward multimodal transformers and image-text foundation models. These approaches progressively broadened transferable visual and crossmodal capabilities.
- Large-scale annotated ConvNet and Transformer pretraining supports visual recognition tasks including classification, localization, segmentation, video recognition, and tracking.
- Self-supervised vision methods such as BEiT, MAE, and SimMIM explore image representation learning without relying solely on standard supervised targets.
- Early vision-language pretraining used object-detection modules, while later methods trained unified multimodal transformers from scratch.
- CLIP and ALIGN showed that contrastively pretrained dual encoders learn strong image and text representations for crossmodal alignment and zero-shot image classification.
3 Approach
CoCa unifies classification, contrastive, and captioning paradigms in one image-text encoder-decoder model. Its decoupled decoder and task-specific pooling support efficient pretraining and transfer across vision and vision-language tasks.
- Natural Language Supervision: CoCa combines single-encoder classification, dual-encoder contrastive learning, and encoder-decoder captioning under one image-text foundation model.The model uses natural-language supervision across these paradigms rather than treating them as separate systems.
- CoCa Architecture: The decoder’s first half omits cross-attention for unimodal text representations, while later layers cross-attend to image features for multimodal representations.This decoupled design allows the same decoder to support contrastive and generative objectives.
- Training Objectives: CoCa jointly optimizes contrastive and captioning losses, with weighting hyper-parameters λCon and λCap combining the objectives.The contrastive objective uses global image-text representations, whereas captioning supplies autoregressive generation and finer-grained features.
- Attentional Poolers: Task-specific attentional poolers produce one image embedding for contrastive learning and 256 queries for generative learning.The paper describes pooled global features as useful for recognition and more visual tokens as beneficial for region-level multimodal understanding.
- Pretraining Efficiency: A single forward propagation computes both training losses by sharing most decoder computation between the contrastive and generative objectives.The efficiency comes from causal masking over complete sentences in the decoupled autoregressive decoder.
- Downstream Transfer: CoCa transfers through zero-shot evaluation, frozen-feature evaluation, or finetuning across image, video, retrieval, and multimodal tasks.Frozen-feature evaluation learns task-specific pooling while sharing the backbone image encoder; video recognition applies an additional pooler over spatial and temporal features.
4 Experiments
CoCa is evaluated as a single pretrained foundation model across visual recognition, crossmodal alignment, image captioning, and multimodal understanding. Across these task groups, it achieves strong or state-of-the-art results with zero-shot transfer, frozen features, or lightweight adaptation.
- Training setup: CoCa is pretrained in a single stage on JFT-3B annotated images and ALIGN noisy alt-text, treating labels as paired text.The setup combines web-scale alt-text data with annotated images and trains with combined contrastive and captioning objectives.
- Experimental scope: CoCa is evaluated across visual recognition, crossmodal alignment, image captioning, and multimodal understanding using one pretrained checkpoint.The evaluation includes zero-shot transfer, frozen-feature evaluation, finetuning, and ablations of objectives and architecture.
- Main results: CoCa sets state-of-the-art results across visual recognition, crossmodal alignment, image captioning, and multimodal understanding with minimal adaptation.The model produces aligned unimodal representations and fused multimodal embeddings, supporting all three broad task groups from one checkpoint.
- Visual recognition tasks: 91.0% Top-1 accuracy on ImageNet is achieved after finetuning, while frozen-encoder evaluation reaches 90.6% on ImageNet and strong video recognition results.CoCa is also reported to outperform recent video approaches after finetuning and to use fewer visual-encoder parameters than other methods.
- Crossmodal alignment tasks: CoCa improves zero-shot image-text retrieval on MSCOCO and Flickr30K and sets new state-of-the-art zero-shot classification results on ImageNet.The zero-shot classification gains extend across all six considered datasets, and the unified single-stage framework avoids sequential pretraining.
- Image captioning and multimodal understanding tasks: CoCa obtains state-of-the-art performance on VQA, SNLI-VE, NLVR2, and NoCaps while outperforming strong MSCOCO captioning baselines.Its encoder-decoder design supports multimodal classification and caption generation without further fusion adaptation.
- Ablation analysis: Ablations show that captioning pretraining matches single-encoder classification pretraining and that combining contrastive and captioning losses adds capabilities with minimal extra cost.The combined objectives improve zero-shot alignment and VQA relative to contrastive-only training while retaining captioning-only efficiency.
5 Broader Impacts
CoCa transfers across many vision and vision-language tasks, but its training data and broader impacts require further analysis before practical use.
- CoCa uses the same pretraining data as previous methods, motivating additional analysis of the data and resulting model before practical deployment.
- The models remain potentially vulnerable to image corruptions not captured by current evaluation sets or real-world scenarios.
- Further community exploration is needed to understand broader impacts, including fairness, social bias, and potential misuse.
6 Conclusion
CoCa is an image-text foundation model family that combines contrastive and captioning objectives in one encoder-decoder model. A single checkpoint achieves state-of-the-art performance across diverse vision and vision-language tasks.
- CoCa combines contrastive and captioning objectives in an encoder-decoder model pretrained on image-text pairs from varied data sources in a single stage.
- CoCa subsumes existing vision pretraining paradigms through natural language supervision.
- A single CoCa checkpoint achieves state-of-the-art performance across a wide spectrum of vision and vision-language problems.
A Visual Recognition Finetuning Details
Visual recognition finetuning adds a pooler and classification loss, then adapts both the encoder and pooler when finetuning.
- Table 9 lists the hyper-parameters used in the visual recognition experiments.
- Frozen-feature evaluation adds an attentional pooling layer and a softmax cross-entropy classification loss to visual feature sequences.
- Finetuning uses the frozen-feature architecture with poolers while updating both the encoder and pooler.
B Multimodal Understanding Finetuning Details
Multimodal understanding uses CoCa’s decoder outputs with task-specific pooling or input construction, while captioning is finetuned with the pretraining cross-entropy objective.
- Table 10 lists the hyper-parameters used in the multimodal experiments.
- CoCa’s final decoder outputs support multimodal understanding and generation on vision-language benchmarks.
- Multimodal classification pools decoder outputs with a single-query attentional pooler before training a linear classifier.
- VQA v2 is formulated as classification over the 3,129 most frequent training-set answers.
- NLVR2 constructs two image-text input pairs, concatenates them for classification, and uses no image augmentation.
- Image captioning uses the pretraining cross-entropy loss and finetunes on MSCOCO to predict MSCOCO test and NoCaps evaluations.