PaperlayerView exact revision on arXiv

Source-linked AI summary

VirTex: Learning Visual Representations from Textual Annotations

Karan Desai, Justin Johnson

arXiv:2006.06666v3cs.CVcs.CL

TL;DR

VirTex addresses whether high-quality visual representations can be learned from fewer images than traditional classification pretraining. It trains visual models using semantically dense captions and finds competitive downstream performance with up to 10× fewer images than ImageNet-based methods.

  • Problem

    Pretraining seeks data-efficient alternatives to ImageNet classification that learn high-quality visual representations from fewer images.

  • Method

    VirTex jointly trains a visual backbone and bidirectional captioning head from scratch on COCO Captions to learn transferable visual features.

  • Results

    VirTex matches or exceeds supervised and unsupervised ImageNet pretraining across downstream vision tasks despite using up to 10× fewer images.

  • Takeaways & Limitations

    Semantically dense natural-language supervision can provide transferable visual representations with better data-efficiency than other pretraining approaches.

  • Takeaways & Limitations

    The study focuses on directional captioning models and leaves masked language modeling for future work because of computational constraints.

Abstract

from arXiv · show

The de-facto approach to many vision tasks is to start from pretrained visual representations, typically learned via supervised training on ImageNet. Recent methods have explored unsupervised pretraining to scale to vast quantities of unlabeled images. In contrast, we aim to learn high-quality visual representations from fewer images. To this end, we revisit supervised pretraining, and seek data-efficient alternatives to classification-based pretraining. We propose VirTex -- a pretraining approach using semantically dense captions to learn visual representations. We train convolutional networks from scratch on COCO Captions, and transfer them to downstream recognition tasks including image classification, object detection, and instance segmentation. On all tasks, VirTex yields features that match or exceed those learned on ImageNet -- supervised or unsupervised -- despite using up to ten times fewer images.

1. Introduction

VirTex revisits supervised pretraining by using semantically dense image captions to learn transferable visual representations from fewer images. It trains visual and language models jointly, then transfers the visual features to recognition tasks, matching or exceeding ImageNet-based methods with up to 10× fewer images.

  • Motivation: VirTex addresses the question of whether high-quality visual representations can be learned efficiently with fewer images than large-scale unsupervised pretraining.The approach revisits supervised pretraining and seeks an alternative to traditional classification pretraining that uses each image more efficiently.
  • Approach: VirTex jointly trains a ConvNet and Transformer from scratch to generate natural-language captions, then transfers the learned visual features to downstream recognition tasks.The transferred backbone is used for tasks such as object detection.
  • Motivation: Captions provide denser semantic supervision than contrastive learning or classification by describing objects, attributes, relationships, and actions.This semantic density motivates using textual features to learn visual features with fewer images.
  • Motivation: Natural-language annotations can simplify data collection because they do not require an explicit category ontology and can be written by non-experts.Weakly aligned image-text pairs can also be collected from internet images.
  • Results: VirTex matches or exceeds supervised and unsupervised ImageNet pretraining across image classification, object detection, instance segmentation, and low-shot recognition while using up to 10× fewer images.Models are trained from scratch on COCO Captions and evaluated on downstream recognition tasks.

2. Related Work

Related work scales visual pretraining through web data, unlabeled-image objectives, or multimodal pipelines. VirTex instead places vision tasks downstream of image-caption pretraining and trains its model from scratch.

  • Alternative data sources: Weakly supervised methods scale pretraining with large numbers of web images carrying noisy labels or tags.Examples use YFCC-100M or JFT-300M to move beyond ImageNet supervision.
  • Self-supervised learning: Self-supervised methods learn from unlabeled images using handcrafted pretext tasks, generative modeling, clustering, or contrastive objectives.Recent contrastive approaches encourage similar features for differently transformed views of one image.
  • Self-supervised learning: These self-supervised methods rely on low-level visual cues, whereas VirTex uses textual annotations to support semantic understanding and can exploit additional metadata.The comparison concerns the type of supervision used to learn visual representations.
  • Vision-and-language pretraining: Vision-and-language pretraining commonly builds on ImageNet-pretrained CNNs, detector-derived region features, pretrained language models, and multimodal transformers.In that pipeline, vision-and-language tasks remain downstream from visual representations initially learned on ImageNet.
  • VirTex: VirTex reverses that ordering by pretraining through image captioning and placing vision tasks downstream from vision-and-language pretraining.Compared with concurrent work, VirTex trains the entire model from scratch rather than relying on pretrained BERT features.

3. Method

VirTex learns visual features by jointly training a convolutional backbone with bidirectional captioning Transformers on image-caption pairs. After pretraining, it discards the textual head and transfers the visual backbone to downstream recognition tasks.

  • Pretraining objective: VirTex maps image-caption pairs to transferable visual representations by training a visual backbone and textual captioning head jointly.The visual backbone extracts image features, while the textual head predicts caption tokens.
  • Textual head: Bicaptioning uses independent forward and backward language models to predict caption tokens left-to-right and right-to-left.The two directions provide bidirectional language supervision.
  • Training: All model components are randomly initialized and optimized end-to-end to maximize the log-likelihood of the correct caption tokens.The learned parameters include the visual backbone and both directional language models.
  • Visual backbone: The visual backbone processes raw pixels into a spatial feature grid used for caption prediction during pretraining and for linear evaluation or fine-tuning downstream.The experiments use a ResNet-50 backbone producing a 7 × 7 grid of 2048-dimensional features.
  • Design choice: Directional language models are used instead of masked language models because preliminary experiments found masked modeling slower to converge and less sample-efficient.The paper leaves masked language modeling to future work due to computational constraints.
  • Textual head: The textual head uses Transformers with masked self-attention over caption tokens and attention between token and image features.The architecture includes residual connections, dropout, and layer normalization around its operations.
  • Training details: COCO Captions training uses 118K images with five captions each, standard image augmentation, and caption-consistent left-right swapping.Training uses distributed optimization for 500K iterations.

4. Experiments

VirTex is evaluated against supervised, self-supervised, and alternative captioning or annotation-based pretraining methods across classification, transfer, and ablation settings. Results generally show stronger data efficiency, while some ablations reveal task- and architecture-dependent effects.

  • Evaluation setup: VirTex uses COCO Captions for pretraining and transfers learned features to six downstream vision tasks using frozen backbones or fine-tuning.The evaluation covers image classification, object detection, instance segmentation, and low-shot or fine-grained recognition settings.
  • Annotation cost efficiency: VirTex achieves the best performance-versus-cost tradeoff among COCO methods using self-supervision, labels, masks, or captions.The comparison estimates annotation costs in worker hours and reports VirTex outperforming all evaluated methods.
  • Data efficiency: 88.7 mAP: VirTex-100% outperforms IN-sup-100% at VOC07 while using 118K rather than 1.28M images.With equal caption-training budget, spreading captions across more images also improves results: VirTex-50% reaches 79.4 mAP versus 69.3 for VirTex-10%.
  • Data efficiency: 53.8 vs. 53.6: VirTex-100% slightly exceeds IN-sup-10% on ImageNet-1k classification using 118K versus 128K images.The paper notes that this comparison is unfair because IN-sup models use the downstream dataset during training.
  • Ablations: Bicaptioning outperforms forward captioning, token classification, and masked language modeling, while sequential language modeling improves visual features.Masked language modeling performs worse, possibly because of poor sample efficiency, and may require longer training schedules.
  • Ablations: Larger transformers generally improve downstream performance, whereas very deep transformers show slight degradation consistent with overfitting.Wider and deeper visual backbones improve VOC07 results, but trends are opposite on ImageNet, which the authors attribute to optimization issues.
  • Fine-tuning transfer: VirTex matches or exceeds ImageNet-supervised and MoCo-IN pretraining across four fine-tuning tasks despite using 10× fewer pretraining images.It significantly outperforms methods using similar or more pretraining images, with particularly strong improvements on LVIS.

5. Conclusion

VirTex learns transferable visual representations from textual annotations and remains competitive with ImageNet-based supervised and self-supervised methods. The approach also offers a pathway toward web-scale image-text pretraining.

  • VirTex learns visual representations from textual annotations and is competitive with supervised classification and self-supervised learning on ImageNet.
  • Future work could transfer both VirTex’s visual backbone and textual head to tasks beyond downstream vision.
  • Captions could support scaling VirTex to web-scale image-text pairs, which are larger but noisier than COCO.

Appendix A. Additional Experiments

The evaluation protocol follows prior work on pretraining visual representations, with differences reported where applicable.

  • The experiments use an evaluation protocol consistent with prior work on pretraining visual representations.

A.1. Image Classification with Linear Models

The appendix documents evaluation details, data-efficiency controls, and comparisons between COCO-derived and ImageNet-style images. It also reports that larger backbones generally improve downstream performance.

  • Image Classification with Linear Models: VOC uses resizing to 256 pixels, a 224 × 224 center crop, and ImageNet color normalization.
  • Image Classification with Linear Models: SVM evaluation uses four cost values—C ∈ {0.01, 0.1, 1.0, 10.0}—selected from initial observations for faster evaluation.
  • Image Classification with Linear Models: VirTex uses learning rate 0.3, while ImageNet-supervised models use 0.1; annotation costs are estimated from COCO and nocaps reports.
  • Image Classification with Linear Models: Caption collection is estimated at 39.2 seconds per caption for 118K × 5 COCO captions.
  • Image Classification with Linear Models: ImageNet subsets preserve class distributions, whereas VirTex samples COCO Captions without class-label balancing, potentially advantaging ImageNet models.
  • Image Classification with Linear Models: ImageNet-supervised models follow torchvision ResNet-50 training with SGD, momentum 0.9, weight decay 10^-4, and 90 epochs.
  • Image Classification with Linear Models: COCO contains approximately 2.9 object classes and 5.7 instances per image, unlike mostly single-object ImageNet images.
  • Image Classification with Linear Models: An iconic-image COCO dataset yields 79.1 VOC07 mAP, versus 88.7 for VirTex, showing scene composition does not entirely explain VirTex’s data-efficiency.

A.2. Ablations

Bicaptioning is more sample-efficient than masked language modeling for learning transferable visual representations. Masked language modeling performs worse and converges more slowly in the reported ablations.

  • Masked language modeling performs worse than other pretraining tasks on downstream linear classification performance.
  • Masked language modeling converges more slowly than bicaptioning on VOC07 mAP, indicating poorer sample efficiency.

A.3. Fine-tuning Tasks for Transfer

The appendix documents reproducible Detectron2 fine-tuning configurations for COCO, PASCAL VOC, and LVIS, including an additional LVIS v0.5 evaluation where VirTex significantly outperforms all baselines.

  • Fine-tuning configurations: Config files specify the downstream fine-tuning setup for COCO, PASCAL VOC, and LVIS in Detectron2 format.The configurations are provided to exactly replicate the reported downstream evaluations.
  • LVIS v0.5: LVIS v0.5 evaluation uses separate train and validation splits, with 1230 classes and a 0.0 test score threshold.These settings differ from the LVIS v1.0 evaluation and follow the original LVIS v0.5 baselines.
  • LVIS v0.5: VirTex significantly outperforms all baseline methods on the LVIS v0.5 split.The appendix reports this trend as similar to the evaluation on the LVIS v1.0 split.

A.4. Selecting Best Checkpoint by VOC07 mAP

Captioning quality improves alongside downstream performance but is not strongly correlated with it, so the authors select the best checkpoint using VOC07 classification mAP.

  • Correlation: Captioning performance has a positive but imprecise correlation with downstream vision-task performance.The appendix uses this observation to motivate checkpoint selection by a downstream metric rather than captioning quality alone.
  • Checkpoint selection: The best checkpoints occur at different iterations: 496K by VOC07 mAP (88.7) and 492K by CIDEr (105.8).These values show why the two validation criteria do not identify the same checkpoint.
  • Checkpoint selection: The authors select the best checkpoint based on PASCAL VOC linear classification performance.They use VOC07 classification mAP as a representative downstream task because it is fast and simple to evaluate.

Appendix B. Decoder Attention Visualizations for Caption Predictions

The appendix visualizes how VirTex’s transformer decoder attends to image regions while generating caption words, using beam-searched caption predictions.

  • Attention visualizations: Figures 9 and 10 overlay decoder attention weights on input images for predicted caption words and time steps.The attention masks are normalized to [0, 1] to improve visual contrast.
  • Decoding setup: Captions are decoded from the forward transformer of an L = 1, H = 512 VirTex model using beam search.This decoding setup is used for the qualitative attention examples.
  • Attention visualizations: The visualizations show attention separately for each time step of the predicted caption.The examples include captions describing kites, a truck, a laptop, and a cat in a fenced area.
Loading 2006.06666v3…