PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Visual Representations with Caption Annotations

Mert Bulent Sariyildiz, Julien Perez, Diane Larlus

arXiv:2008.01392v1cs.CV

TL;DR

Visual pretraining often depends on costly annotations or very large unlabeled datasets. This paper introduces ICMLM, which predicts masked caption words from visual cues, and reports transferable representations from relatively few captioned images, while noting a limitation of global caption pooling.

  • Problem

    The paper addresses the need for visual representations without relying on costly fine-grained labels or extremely large unlabeled datasets.

  • Method

    ICMLM learns visual representations from image-caption pairs by predicting masked caption words using visual cues, with hybrid visual-textual architectures.

  • Results

    ICMLM produces useful transferable visual representations, using only a hundred thousand captioned images and competing with methods trained on a hundred million images.

  • Takeaways & Limitations

    Caption structure provides global and localized semantic information for learning visual representations from image-caption pairs.

  • Takeaways & Limitations

    Global caption representations can suppress token-level details, motivating explicit relations between individual image concepts and caption tokens.

Abstract

from arXiv · show

Pretraining general-purpose visual features has become a crucial part of tackling many computer vision tasks. While one can learn such features on the extensively-annotated ImageNet dataset, recent approaches have looked at ways to allow for noisy, fewer, or even no annotations to perform such pretraining. Starting from the observation that captioned images are easily crawlable, we argue that this overlooked source of information can be exploited to supervise the training of visual representations. To do so, motivated by the recent progresses in language models, we introduce {\em image-conditioned masked language modeling} (ICMLM) -- a proxy task to learn visual representations over image-caption pairs. ICMLM consists in predicting masked words in captions by relying on visual cues. To tackle this task, we propose hybrid models, with dedicated visual and textual encoders, and we show that the visual representations learned as a by-product of solving this task transfer well to a variety of target tasks. Our experiments confirm that image captions can be leveraged to inject global and localized semantic information into visual representations. Project website: https://europe.naverlabs.com/icmlm.

1 Introduction

The paper targets the cost and data-efficiency limits of supervised and self-supervised visual pretraining by exploiting image captions as supervision. It proposes image-caption proxy tasks, including ICMLM, and reports transferable visual representations from captioned images.

  • Manual annotation is costly for fine-grained taxonomies, while retraining dedicated models for each downstream task is inefficient.
  • Self-supervised methods avoid annotations but may require up to a hundred million images, motivating more data-efficient supervision.
  • Captioned images provide freely available textual metadata from which high-quality large-scale datasets can be automatically constructed.
  • The paper asks whether limited image-caption pairs can train transferable visual representations and how images and captions should interact.
  • Image-conditioned masked language modeling predicts masked caption words using visual cues and helps representations localize semantic concepts in captions.
  • Using only a hundred thousand captioned images, ICMLM representations are competitive with self-supervised approaches using a hundred million images and sometimes with fully supervised approaches.

2 Related Work

Related work contrasts annotation-heavy, weakly supervised, and self-supervised visual representation learning with multimodal approaches. The paper emphasizes learning from smaller image collections by using the structure of language and image-caption interactions.

  • Standard visual pretraining commonly uses ImageNet-trained CNNs as feature extractors or starting points for target tasks, but fully annotated datasets require substantial effort and expertise.
  • Weakly supervised and self-supervised methods reduce annotation requirements but can require up to hundreds of millions of images and substantial processing.
  • Unsupervised methods often benefit from curated dataset structure, while raw-data training may require 96 millions images to express their full potential.
  • The paper uses pretrained language models because language structure cannot be reduced to n-gram statistics, and reports useful visual representations across a broad range of tasks.

3 Method

The method learns visual representations from image-caption pairs through complementary proxy tasks: global tag prediction and image-conditioned masked language modeling. ICMLM connects masked caption concepts to image regions using multimodal architectures that preserve spatial and token-level information.

  • Proxy tasks: The method uses image-caption pairs to train CNN visual representations through global concept prediction and localized masked-token prediction.The two proxy tasks target complementary global and local semantics.
  • Tag prediction: Tag prediction formulates captions as multi-label image classification, assigning each image binary labels for concepts appearing in its caption.Concept sets can be derived from caption n-grams, POS-selected tokens, or BERT-based cluster assignments.
  • Model composition: The models combine CNN, language-model, transformer, attention-plus-fully-connected, and tag-prediction modules, while only the CNN is used during target-task evaluation.TP⋆ uses CNN plus tag prediction; ICMLMtfm and ICMLMatt-fc additionally use language and multimodal modules.
  • Image-conditioned masked language modeling: ICMLM masks a caption concept and predicts it from visual information, training representations to reconstruct missing caption content from the corresponding image.The task assumes caption concepts are observable in the described image and masks one concept token at a time.
  • ICMLM architectures: ICMLMtfm fuses flattened visual features with token features using a transformer, then predicts the masked token from its transformed representation.The model combines CNN features, language-model token embeddings, and a transformer encoder.
  • ICMLM architectures: ICMLMatt-fc computes a spatial attention map conditioned on the masked token, pools visual features around the concept, and maps the pooled feature through fully connected layers.The attention module suppresses vague textual tokens and emphasizes image locations associated with the masked concept.

4 Experiments

Experiments compare caption-supervised proxy tasks with fully, weakly, and self-supervised baselines across transfer tasks. ICMLM models consistently produce strong transferable representations, with architecture-dependent differences across datasets.

  • Experimental setup: The evaluations use frozen-backbone features with linear classifiers across VOC, IN-1K, and Places, while Table 2 reports VGG16 results and Table 3 reports ResNet50 results.Table 2 reports VOC mAP and IN-1K and Places top-1 accuracy; Table 3 reports VOC mAP and IN-1K top-1 accuracy.
  • Ablative study on the proxy task: ICMLMtfm achieves higher masked-token prediction than ICMLMatt-fc, while ICMLMatt-fc generalizes better to VOC.The transformer architecture benefits from blending visual and textual cues, whereas the attention-only model predicts using visual cues alone.
  • Comparison of supervision methods: Reducing ImageNet training data to 100K images by classes or images per class significantly hurts performance.The comparison highlights the importance of scale for fully supervised representations.
  • Comparison of supervision methods: ICMLM models significantly improve over TP⋆ baselines across VOC, IN-1K, and Places, while their relative ranking depends on the target dataset.ICMLMatt-fc performs better on VOC, whereas ICMLMtfm performs on par with or better than it on IN-1K and Places.
  • Comparison of supervision methods: TPCluster outperforms TPLDA for caption-based training, but performs on par with or worse than TPPostag.This suggests that global caption representations can suppress the importance of individual tokens.
  • Additional results with ResNet50: With ResNet50, ICMLM⋆ outperforms TPPostag by at least 4.7% and 4.0% on VOC and IN-1K, respectively.ICMLM⋆ also exceeds TPLabel by at least 7.1% and 13.9% on those tasks.

5 Conclusion

The paper presents captions as a cheaper alternative to expert labels for learning general-purpose visual representations. ICMLM uses image-caption pairs to produce representations that perform on par with state-of-the-art self-supervised approaches across multiple tasks.

  • Conclusion: ICMLM uses captions to train visual representations from image-caption pairs instead of relying on expensive fine-grained ground-truth labels.The proxy task automatically produces image labels by leveraging caption information.
  • Conclusion: ICMLM representations perform on par with state-of-the-art self-supervised approaches across a variety of tasks.The paper reports this result using at most 118 thousand images together with companion captions.

A Label sets vs. target task performances

Caption-derived label sets provide broader supervision than small manually annotated class sets. Increasing the caption vocabulary improves transfer performance, especially on IN-1K.

  • Caption-derived versus annotated label sets: All TPPostag models improve over TPLabel, indicating that captions provide more comprehensive supervision than a small set of image classes.The gaps are more significant on IN-1K, where a larger tag vocabulary can encode more discriminative patterns.
  • Caption-derived versus annotated label sets: 2.4%, 9.9%, and 2.0% are the TPPostag improvements over TPLabel on VOC, IN-1K, and COCO using the 5K most frequent nouns, adjectives, and verbs.The same caption-derived label set is used for the reported comparison across all three target datasets.
  • Evaluation protocol: The comparison extracts representations from frozen ResNet50 backbones and evaluates them with linear classifiers on VOC, IN-1K, and COCO.This setup tests transfer and training-set generalization after pretraining the backbone.

B ICMLM vs. target task performances

The analysis compares masked-language-modeling performance with transfer performance across target tasks and model configurations. More complex multimodal modules improve the proxy task but can weaken visual representations, while ICMLM variants remain competitive across VOC, IN-1K, and COCO.

  • Results: More hidden layers or attention heads improve MLM performance but reduce target-task results.The authors attribute this trade-off to complex modules capturing more visual-textual interconnections and leaving the visual model with weaker semantic features.
  • Results: ICMLMtfm significantly outperforms ICMLMatt-fc on MLM and IN-1K, while ICMLMatt-fc is slightly better on VOC and COCO.The authors associate the latter pattern with ICMLMatt-fc overfitting to concepts present in the training set.
  • Results: ICMLM⋆ improves TPPostag by at least 3.1%, 3.3%, and 2.1% and TPLabel by at least 5.5%, 13.2%, and 4.1% on VOC, IN-1K, and COCO, respectively.These comparisons use configurations with one hidden layer and one attention head.
  • Limitations: With VGG16 backbones, one attention head and λ = 0 produced inferior ICMLM⋆ models.The authors believe the absence of residual connections causes overfitting to the MLM task.
  • Method: The ICMLM⋆ objective combines ℓtp and ℓmlm losses to supervise both global and localized image semantics.The coefficient λ controls the combination of these loss terms.

C Zero-shot Object Classification

The zero-shot evaluation tests visual features from caption-based pretraining on fine-grained birds and coarse-grained animals using class-level attributes. ICMLM variants improve transfer over tag-prediction models in several dataset and feature-layer settings, while some later-layer features show overfitting.

  • Evaluation: CUB contains roughly 12K images of 200 bird species defined by 312 attributes, while AWA2 contains roughly 38K images of 50 animals defined by 85 attributes.The evaluation reports top-1 accuracy over all seen and unseen classes.
  • Method: The evaluation trains bilinear score functions between pooled visual features and class-level attribute vectors.The visual features use activations from the last three convolutional layers, with approximately 9K dimensions.
  • Results: ICMLMtfm pretrained on Visual Genome improves TP⋆ on CUB by up to 1.4%, 1.3%, and 2.2% with C-11, C-12, and C-13 features.These are top-1 prediction accuracy improvements among all classes.
  • Results: ICMLMtfm pretrained on COCO improves TP⋆ on AWA2 by up to 1.1%, 0.9%, and 1.0% with C-11, C-12, and C-13 features.The authors note that COCO-pretrained ICMLM⋆ models tend to perform slightly better on AWA2, particularly with C-13 evaluations.
  • Results: For Visual Genome pretraining, ICMLMatt-fc C-13 features score up to 0.9% below TP⋆, while its C-11 and C-12 features perform better.The authors interpret this as slight overfitting to the MLM task in the VGG16 backbone, with richer semantics retained in earlier layers.

D Additional qualitative results

Additional attention maps show that ICMLMatt-fc can localize concrete objects and some abstract, color, and texture concepts from masked captions. The model still fails on ambiguous concepts and on grouping multiple related concepts.

  • Qualitative results: The att module localizes visually clear object categories, including bananas, babies, cats, sheep, and beds in cluttered scenes.These examples are shown in qualitative attention maps for COCO image-caption pairs.
  • Qualitative results: The model also localizes abstract visual concepts such as mirrors and glass, including glass as both a drinking object and a material.The glass examples span the drinking glass and the material of a table and vase.
  • Qualitative results: ICMLMatt-fc learns visual appearance associated with colors and textures, including blue, striped, and colorful.These concepts are illustrated in the additional qualitative results.
  • Failure cases: Ambiguous concepts such as middle and open can produce meaningless attention maps, causing masked-word prediction to rely on the rest of the caption.Grouping several concepts, such as the different colors of three shirts, remains beyond the model’s capacity.
  • Implementation: The transformer variant contextualizes BERTbase token embeddings with image features mapped into the token-embedding space using multi-headed attention.Its transformer encoder concatenates visual and textual representations, then applies residual connections, dropout, LayerNorm, ReLU, and linear projections.

F Implementation details

The implementation-details section introduces the technical description of training the proxy-task models and evaluating them on target tasks.

  • Implementation details: The section provides technical details for training models on proxy tasks and evaluating them on target tasks.

F.1 Training for proxy tasks

The proxy-task models are trained from scratch under dataset-limited settings, with separate optimization configurations for VGG16 and ResNet50 backbones. ICMLM training uses image-caption-mask-token triplets, while pretrained BERT is frozen during ICMLM⋆ training.

  • VGG16 backbones: VGG16 models are trained on Visual Genome or MS-COCO using rotation prediction, restricting all pipeline stages to 103K or 118K training images.The setup avoids existing RotNet checkpoints because they had processed millions of images.
  • ICMLM training: ICMLM models use 100K RAdam iterations with batch size 128, and training exposes 2.5M–13M unique image-caption-masked-token triplets depending on dataset and label set.The learning rate starts at 1e-4 and is decayed by 0.1 after 80K and 90K iterations.
  • ResNet50 backbones: ResNet50 TPLabel and TPPostag models are trained from scratch for 100K SGD iterations, while ICMLM⋆ models use initialized TPPostag backbones and train for 500K iterations.ICMLM⋆ uses batch size 512; the other ResNet50 proxy models use batch size 128.
  • Validation and implementation: Hyperparameters are validated on Visual Genome and COCO, and ICMLM⋆ freezes the pretrained HuggingFace BERTbase model during training.Experiments use PyTorch and NVIDIA Apex mixed precision.

F.2 Evaluation on target tasks

Target-task evaluation freezes the visual backbones and trains lightweight linear classifiers on extracted representations or intermediate VGG16 features. The protocols cover linear probing for VGG16 and SVM or logistic-regression evaluation for ResNet50 features.

  • VGG16 evaluation: VGG16 representations are evaluated by probing linear logistic-regression classifiers after various backbone layers.The classifiers are trained with SGD updates and data augmentation using a modified publicly available evaluation implementation.
  • ResNet50 evaluation: ResNet50 representations are evaluated by extracting last-convolutional-layer features and training linear SVMs or logistic-regression classifiers while keeping backbones frozen.The extracted tensors are 7 × 7 × 2048, with pooling choices depending on the target dataset.
  • Feature extraction: For VOC and COCO, 7 × 7 × 2048 ResNet50 tensors are pooled to 8192-dimensional features and ℓ2-normalized before linear SVM training.For IN-1K, global average pooling produces 2048-dimensional features to reduce computational demands.
  • Classifier training: SVM cost is selected from 40 log-uniform values between 10^-5 and 10^5, while logistic-regression learning rate and weight decay are tuned with Optuna.Logistic-regression learning rates are sampled between 10^-1 and 10^2 with cosine-based annealing.
Loading 2008.01392v1…