PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Visual Representations with Caption Annotations

Mert Bulent Sariyildiz, Julien Perez, Diane Larlus

arXiv:2008.01392v1cs.CV

TL;DR

Caption annotations are rarely used to train visual representations from scratch, despite their potential as a less costly alternative to fine-grained labels. The paper introduces ICMLM, which predicts masked caption words using visual cues. Using 100,000 captioned images, its representations are competitive with recent self-supervised approaches using 100 million images and, in some cases, fully supervised approaches.

  • Problem

    Caption annotations have rarely been used to train visual representations from scratch, and prior caption-based methods treated words individually rather than using whole-sentence structure.

    Source: §1 · p. 2

  • Method

    ICMLM trains visual representations by predicting masked caption words from visual cues, using multimodal architectures that relate image features to individual caption concepts.

    Source: §1 · p. 3

  • Results

    Using 100,000 captioned images, ICMLM representations are competitive with recent self-supervised approaches using 100 million images and, in some cases, fully supervised approaches.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Caption cues can provide image-level and localized semantic information for visual representations.

    Source: §1 · pp. 2–3

    The cluster-prediction protocol may largely ignore token-level details, especially in long captions.

    Source: §3.2 · p. 7

Abstract

from arXiv · show

Pretraining general-purpose visual features has become a crucial part of tackling many computer vision tasks. While one can learn such features on the extensively-annotated ImageNet dataset, recent approaches have looked at ways to allow for noisy, fewer, or even no annotations to perform such pretraining. Starting from the observation that captioned images are easily crawlable, we argue that this overlooked source of information can be exploited to supervise the training of visual representations. To do so, motivated by the recent progresses in language models, we introduce {\em image-conditioned masked language modeling} (ICMLM) -- a proxy task to learn visual representations over image-caption pairs. ICMLM consists in predicting masked words in captions by relying on visual cues. To tackle this task, we propose hybrid models, with dedicated visual and textual encoders, and we show that the visual representations learned as a by-product of solving this task transfer well to a variety of target tasks. Our experiments confirm that image captions can be leveraged to inject global and localized semantic information into visual representations. Project website: https://europe.naverlabs.com/icmlm.

1 Introduction

The paper proposes learning transferable visual representations from image-caption pairs, using caption-derived tags and image-conditioned masked language modeling (ICMLM). It argues that captions can provide useful supervision at a smaller data scale while enabling visual representations to capture localized semantics.

Motivation

  • Manually annotated datasets are costly to create, especially for fine-grained categories, and training a separate model from scratch for every task is inefficient.Self-supervised methods avoid annotations but may require extremely large datasets or longer training with larger networks, demanding substantial compute.Source: §1 · p. 2
  • Image captions and other metadata can be available without additional annotation, and sanitation procedures can produce high-quality large-scale captioned datasets.The introduction identifies companion text as a potentially richer source of information than image tags.Source: §1 · p. 2
  • Caption-based pretraining may reduce the data scale needed for visual representations, and short captions can be easier to obtain than expert fine-grained labels.Prior caption-based approaches predicted individual words or topic probabilities rather than using sentence structure.Source: §1 · p. 2

Contributions

  • Predicting image tags extracted from captions is a simple proxy task that the authors report learns transferable visual representations and is competitive with other pretraining strategies.The paper also introduces ICMLM as a second proxy task to use captions more effectively.Source: §1 · pp. 2–3
  • ICMLM trains visual representations by predicting masked caption words from images, and the proposed multimodal architectures can attend to relevant image regions.The task is intended to help representations localize semantic concepts in captions.Source: §1 · p. 3
  • With 100,000 captioned images, ICMLM representations are competitive with self-supervised approaches using 100 million images and, in some cases, fully supervised approaches.The authors present these results as evidence that text is a powerful cue for visual representation learning.Source: §1 · p. 3

2 Related Work

Prior work learns visual features from labeled, webly supervised, or self-supervised data, while some vision-language systems rely on detector features. This work instead uses captions and a pretrained language model to learn visual representations from scratch for visual tasks.

Weakly/Webly-supervised learning

  • Webly supervised methods use web metadata such as captions and hashtags, while ImageNet upscaling generates pseudo-labels but requires initial annotations and extremely large image sets.The paper contrasts these approaches with its use of companion captions and at most 118 thousand images.Source: §2 · p. 3
  • ImageNet-1K pretraining is common for visual features, but creating fully annotated datasets requires effort, prior knowledge, and domain expertise.CNN pretraining on external datasets is standard, particularly when target-domain data is scarce.Source: §2 · p. 3

Unsupervised representation learning

  • Unsupervised methods use proxy tasks or contrastive learning [43], but their computational and data efficiency remain inferior to supervised models.They often benefit from curated datasets such as ImageNet; work on uncurated data found that 96 million images may be needed, whereas this work leverages text with a much smaller image set.Source: §2 · p. 4

Vision and language

  • Existing vision-language systems fine-tune BERT to fuse modalities but rely on detector-produced visual features and expensive bounding-box annotations.This work instead learns visual representations from scratch for purely visual tasks by leveraging captions.Source: §2 · p. 4

Learning visual features using text

  • Prior approaches use captions for auxiliary prediction tasks, n-gram labels, topic-model embeddings, or task-specific semantic retrieval.The paper distinguishes its approach by using a pretrained language model to represent captions and individual words.Source: §2 · p. 4
  • A pretrained language model represents captions and their words, rather than reducing language to n-gram statistics, to learn visual representations useful across tasks.The paper argues that language has complex structure that n-gram statistics do not capture.Source: §2 · p. 4

3 Method

The method trains visual representations from image-caption pairs using complementary image-level tag prediction and image-conditioned masked language modeling to capture global and localized semantics.

3 Method

  • The framework uses caption concepts for global tags and localized masked-word prediction, training the CNN visual encoder while keeping the pretrained language encoder frozen.The two objectives are combined because tag prediction and masked language modeling encourage attention to global and local image semantics, respectively.Source: §3 · p. 5, §3.2 · p. 9

3.1 Capturing image-level semantics

  • Tag prediction assigns images multi-label concept tags from captions, using either POS-selected nouns, adjectives, and verbs or clusters of BERT [CLS] caption representations.The POS variants use three different combinations of those word classes; the clustering approach predicts hard k-means cluster assignments.Source: §3.1 · pp. 5–6

3.2 Capturing localized semantics

  • ICMLM masks a caption concept and predicts it from image and text features, assuming caption concepts describe observable content in the associated image.It addresses the risk that sentence-level cluster prediction can miss token details, particularly in long captions.Source: §3.2 · p. 7
  • ICMLMtfm fuses spatial visual features with caption-token features in a transformer, while ICMLMatt-fc predicts the masked concept from localized visual features alone.The visual-only design separates concept localization from label prediction to reduce reliance on the remaining caption as a prediction cue.Source: §3.2 · p. 8
  • ICMLMatt-fc scores visual locations against caption tokens, suppresses vague-token attention, pools the visual grid, and predicts using pretrained BERT vocabulary representations.Candidate-token probabilities use dot products with pretrained token embeddings rather than a fully connected projection onto the vocabulary.Source: §3.2 · pp. 8–9

4 Experiments

Experiments evaluate how ICMLM proxy-task performance transfers to target tasks, compare full, weak, and self-supervision, test backbone consistency, and inspect attention to image regions.

4 Experiments

  • The experiments train on COCO or Visual Genome captions and evaluate frozen visual backbones on VOC, IN-1K, and Places-205 classification tasks.VGG16 is the main backbone, with ResNet50 also evaluated; training-only modules are discarded before target-task evaluation.Source: §4 · pp. 10–11

4.1 Ablative study on the proxy task

  • ICMLM models improve masked-token prediction over BERTbase, but ICMLMatt-fc transfers better to VOC than ICMLMtfm despite ICMLMtfm's stronger proxy-task performance.The authors attribute ICMLMtfm's proxy-task advantage to blending visual and textual cues, and suggest that ICMLMatt-fc's visual-only predictions better learn training-set semantic concepts.Source: §4.1 · p. 11

4.2 Comparison of fully-, weakly- and self-supervised methods

  • On VOC and Places-205, at least one ICMLM variant surpasses full ImageNet pretraining, while both beat all TP⋆ baselines; an IN-1K comparison is unfair.TPCluster outperforms TPLDA across the three target datasets but is on par with or worse than TPPostag. ICMLMatt-fc leads ICMLMtfm on VOC, while on IN-1K and Places-205 it is on par with or worse than ICMLMtfm; reduced ImageNet subsets and TPLabel perform far worse than full ImageNet.Source: Table 2 · p. 12, §4.2 · pp. 12–13

4.3 Additional results with ResNet50

  • ICMLM⋆ exceeds TPPostag by at least 4.7% on VOC and 4.0% on IN-1K, and TPLabel by at least 7.1% and 13.9%, respectively.With ResNet50, ICMLMatt-fc does slightly better on VOC, while ICMLMtfm performs better on IN-1K.Source: Table 3 · p. 13, §4.3 · p. 14
  • ICMLMtfm attention maps show the model detecting possible concepts of interest and identifying which concept a caption asks about.The visualizations use a ResNet50 ICMLMtfm model trained on COCO.Source: Figure 3 · p. 13, §4.3 · p. 14

5 Conclusion

The paper introduces ICMLM, a proxy task that uses image captions to supervise visual representations. Its approach performs on par with state-of-the-art self-supervised methods across varied tasks using a fraction of the data, and rivals supervised ImageNet pretraining in some settings.

5 Conclusion

  • The approach performs on par with state-of-the-art self-supervised methods across varied tasks using a fraction of the data, and rivals fully supervised ImageNet pretraining in some settings.The authors highlight its relevance for domains where images are scarce but companion text is abundant.Source: §5 · p. 14
  • ICMLM predicts masked caption words using visual cues, providing an image-caption proxy task that automatically derives image labels from captions.The authors propose it as a cheaper alternative to ground-truth labels for training visual representations.Source: §5 · p. 14
  • The authors propose a novel approach to ICMLM that produces general-purpose visual representations.Captions are described as easier to collect than fine-grained category annotations, while processing very large unlabeled datasets can be difficult.Source: §5 · p. 14

A Label sets vs. target task performances

The section compares visual representations trained with image labels against caption-derived tags and finds that tag-based supervision improves performance across the evaluated target tasks.

A Label sets vs. target task performances

  • Caption-derived label sets use frequent nouns, adjectives, or verbs as TPPostag tags, while ICMLM⋆ uses them as maskable tokens.The analysis compares TPLabel using annotated image labels with TPPostag using caption-derived tags.Source: A Label sets vs. target task… · p. 19
  • 2.4%, 9.9%, and 2.0% gains over TPLabel on VOC, IN-1K, and COCO, respectively, came from TPPostag with 5K nouns, adjectives, and verbs.Representations were extracted from frozen ResNet50 backbones; evaluation used linear SVMs on VOC and COCO and linear logistic regression on IN-1K.Source: A Label sets vs. target task… · p. 19
  • All TPPostag label-set variants outperform TPLabel, suggesting caption descriptions provide more comprehensive supervision than a small set of image classes.Source: A Label sets vs. target task… · p. 19
  • The performance gaps are more significant on IN-1K, which the authors attribute to larger tag vocabularies encoding more discriminative patterns.Source: A Label sets vs. target task… · p. 19

B ICMLM vs. target task performances

The section tests how ICMLM proxy-task performance transfers to target tasks, finding that stronger masked-language modeling can come at the expense of visual transfer and that the two training losses are complementary.

B ICMLM vs. target task performances

  • With #L and #H set to 1, ICMLM⋆ improves TPPostag by at least 3.1% on VOC, 3.3% on IN-1K, and 2.1% on COCO, and TPLabel by at least 5.5%, 13.2%, and 4.1%, respectively.The comparison is between Tab. 4 and Tab. 5.Source: Table 5 · p. 20, B ICMLM vs. target task performances · p. 20
  • More hidden layers or attention heads improve MLM performance but reduce target-task results; ICMLMtfm exceeds ICMLMatt-fc on MLM and IN-1K, while ICMLMatt-fc is slightly better on VOC and COCO.The ResNet-50 ablation sets λ = 0 and evaluates pre-extracted features on VOC, IN-1K, and COCO, with top-1 MTP scores on COCO.Source: B ICMLM vs. target task performances · p. 20
  • With VGG16, one attention head and λ = 0 yielded inferior models, which the authors attribute to absent residual connections and consequent MLM overfitting, consistent with.Source: B ICMLM vs. target task performances · p. 20
  • With ResNet-50, λ = 0.1 minimizes validation ℓtp loss and improves target-task results, supporting complementarity between ℓtp and ℓmlm.The tested λ values were 0.0, 0.1, and 1.0; λ = 1.0 minimized validation ℓtp loss with VGG16.Source: B ICMLM vs. target task performances · p. 21

C Zero-shot Object Classification

This section evaluates zero-shot classification on CUB and AWA2 using attribute-based bilinear classifiers trained on visual features from TP⋆ and ICMLM⋆ backbones. ICMLM results vary by dataset, pretraining corpus, and feature layer, with some features also showing evidence of overfitting to the MLM task.

Zero-shot Object Classification

  • The benchmarks use seen classes for training and testing and reserve unseen-class images for testing; CUB has 150 seen and 50 unseen classes, while AWA2 has 40 seen and 10 unseen classes.CUB comprises about 12K images of 200 fine-grained bird species with 312 attributes; AWA2 has about 38K images of 50 coarse-grained animals with 85 attributes.Source: C Zero-shot Object Classification · p. 21
  • The method trains bilinear score functions between pooled, flattened activations from the last three convolutional layers and class-level attribute vectors.The VGG16 backbones come from TP⋆ or ICMLM⋆ models pretrained on COCO or Visual Genome; attribute vectors have 312 dimensions for CUB and 85 for AWA2.Source: C Zero-shot Object Classification · p. 21

Results

  • On CUB, ICMLMtfm pretrained on VG improves over TP⋆ by up to 1.4% with C-11, 1.3% with C-12, and 2.2% with C-13 features.The comparison reports top-1 accuracy over seen and unseen classes.Source: Table 6 · p. 22, C Zero-shot Object Classification · p. 22
  • On AWA2, ICMLMtfm pretrained on COCO improves over TP⋆ by up to 1.1% with C-11, 0.9% with C-12, and 1.0% with C-13 features.ICMLM⋆ models tend to perform slightly better on AWA2, particularly with C-13 features, when pretrained on COCO, whose concepts are described as more semantically similar to AWA2.Source: Table 6 · p. 22, C Zero-shot Object Classification · p. 22
  • With VG pretraining, ICMLMatt-fc C-13 features score up to 0.9% below TPPostag, while its C-11 and C-12 features perform better, suggesting richer semantics in earlier layers.The authors interpret the C-13 drop as slight overfitting of the VGG16 backbone to the MLM task.Source: C Zero-shot Object Classification · p. 22

D Additional qualitative results · E Transformer network in ICMLM

Appendix D shows additional attention maps and their successes and limitations; Appendix E details the transformer encoder used in ICMLMtfm.

D Additional qualitative results

E Transformer network in ICMLM

  • ICMLMtfm contextualizes BERTbase token embeddings with mapped visual features concatenated into one sequence, using a one-layer, eight-head transformer.The concatenated sequence contains visual and textual tokens, and the eight attention heads are computed in parallel.Source: E Transformer network in ICMLM · p. 23
  • Each head applies scaled dot-product attention; head outputs are concatenated, then followed by residual connections, dropout, LayerNorm [2], ReLU, and linear projections.Source: E Transformer network in ICMLM · pp. 23, 26

F Implementation details

This section outlines technical details for training the proxy-task model and evaluating it on target tasks.

F Implementation details

F.1 Training for proxy tasks

This section specifies the training procedures for VGG16- and ResNet50-based proxy-task models, including dataset limits, optimization schedules, and implementation choices.

F.1 Training for proxy tasks

  • VGG16 backbones were first trained on VG or COCO for rotation prediction using only 103K or 118K images, then fine-tuned on proxy tasks.The rotation-prediction stage ran for 120K iterations; the fully connected layers were removed before proxy-task fine-tuning.Source: §F.1 · p. 26
  • TP⋆ models trained for 100K iterations, while ICMLM models trained for 200K iterations with batches of 896; ICMLMatt-fc attention heads received a 50K-iteration warm-up.ICMLM training used 2.5M–13M unique (image, caption, masked token) triplets, depending on dataset and label set. During warm-up, VGG16 backbones were frozen to prevent noisy gradients.Source: §F.1 · p. 26
  • ResNet50 TPLabel and TPPostag models were trained from scratch for 100K iterations, and ICMLM⋆ models were trained for 500K iterations from pretrained TPPostag checkpoints.Both stages used SGD with momentum 0.9 and a cosine-based learning-rate schedule; the ICMLM⋆ batch size was 512 rather than 128.Source: §F.1 · p. 27
  • Hyperparameters and design choices were validated on VG and COCO validation sets, and the pretrained HuggingFace BERTbase model was frozen during ICMLM⋆ training.Experiments used PyTorch and NVIDIA Apex mixed precision.Source: §F.1 · p. 27

F.2 Evaluation on target tasks

This section describes two frozen-backbone linear evaluation protocols and the feature-extraction and classifier-training choices used to assess visual representations.

F.2 Evaluation on target tasks

  • Evaluation uses layer-wise logistic-regression probes on VGG16 backbones or linear classifiers on last-convolutional ResNet50 features, with backbones frozen.The ResNet50 classifiers are linear SVMs and logistic regression.Source: §F.2 · p. 27
  • For VGG16 probes, publicly available code was modified to use heavier data augmentation and more classifier-training iterations.Source: §F.2 · p. 27
  • ResNet50 crops yield 7 × 7 × 2048 tensors; VOC/COCO SVMs use pooled, flattened, L2-normalized 8192-dimensional features, while IN-1K logistic regression uses 2048-dimensional global-pooled features for computational tractability.Images are resized so the smallest dimension is 224 pixels, then center-cropped to 224 × 224.Source: §F.2 · p. 27
  • Linear SVMs evaluate Pascal-VOC 2007 and MS-COCO 2017 in a one-vs.-all manner, with the cost parameter selected by Optuna from 40 log-uniform samples between 10^-5 and 10^5.Source: §F.2 · pp. 27–28
  • Linear logistic regression uses SGD with momentum 0.9 and batch size 1024, while Optuna tunes learning rate and weight decay over 25 trials with cosine-based learning-rate annealing.Learning rates are sampled log-uniformly from 10^-1 to 10^2 and weight decays uniformly from 0 to 10^-5.Source: §F.2 · p. 28

Related papers

Loading 2008.01392v1…