VirTex: Learning Visual Representations from Textual Annotations

1. Introduction

• ImageNet classification pretraining has enabled broad downstream progress but is costly to scale; unsupervised methods instead use unlabeled images and have reached hundreds of millions or billions of images.The paper asks whether alternative supervised pretraining can learn high-quality representations with fewer images.Source: §1 · p. 1
• VirTex jointly trains a ConvNet and Transformer from scratch to generate image captions, then transfers the learned visual features to downstream recognition tasks.The approach is trained on COCO Captions.Source: §1 · pp. 1–2, Figure 1 · p. 1
• Captions describe multiple objects, attributes, relationships, and actions, providing a denser learning signal than contrastive learning or single-concept classification.This semantic density motivates using textual features to learn visual features with fewer images.Source: §1 · p. 1, Figure 2 · p. 2
• Textual annotations can simplify data collection because descriptions need no explicit category ontology and can be written by non-experts; weakly aligned image-text pairs are also available online.Classification labels typically require an ontology and crowdsourcing pipelines, while image-text sources include internet images.Source: §1 · pp. 1–2
• VirTex matches or exceeds supervised and unsupervised ImageNet pretraining on all evaluated downstream tasks, despite using up to 10× fewer images.Evaluation covers image classification, object detection, instance segmentation, and low-shot recognition.Source: §1 · p. 2

Related Work

• VirTex contrasts with weakly supervised methods that use vast quantities of low-quality web labels by focusing on fewer images with high-quality annotations.Prior work used 100M Flickr images with tags, JFT-300M, or up to 3.5B Instagram images with hashtags.Source: §2 · p. 2
• Self-supervised methods use hand-crafted pretext tasks, contrastive learning, or related objectives, whereas VirTex leverages textual annotations for semantic understanding.The passage notes these methods rely on low-level cues such as color and texture; VirTex can also leverage text metadata when scaled to internet images.Source: §2 · p. 2
• Vision-and-language pretraining transfers joint image-text representations to multimodal tasks, but its typical pipeline begins with ImageNet-pretrained visual features.The pipeline may also use detector features trained on Visual Genome, pretrained language models, and Conceptual Captions before downstream fine-tuning.Source: §2 · p. 2
• VirTex instead makes vision tasks downstream of image-captioning pretraining; relative to Sariyildiz et al., it trains from scratch and evaluates detection and segmentation.Sariyildiz et al. use pretrained BERT; Stroud et al. learn from paired textual metadata but operate and evaluate only on video tasks.Source: §2 · p. 3

Method

• Captions encode object presence, attributes, spatial relationships, and actions, providing rich semantic information that can support downstream visual recognition.Source: §3 · p. 3

Language Modeling

• VirTex uses directional models rather than masked language models because MLMs converged more slowly and predict only a subset of caption tokens, reducing sample efficiency.The authors leave MLMs for future work because of computational constraints.Source: §3 · p. 3

Visual Backbone

• The visual backbone is a ResNet-50 that maps a 224 × 224 image to a 7 × 7 grid of 2048-dimensional features; its pretraining projection is omitted downstream.Source: §3 · p. 3

Textual Head

• The textual head uses two Transformer language models with masked self-attention over caption tokens and attention to image features; the models share token embeddings.The forward and backward models have independent Transformer layers.Source: §3 · pp. 3–4

Method

• A visual backbone and textual head are trained end-to-end from scratch; the head predicts captions in both directions, then is discarded while the backbone transfers to recognition tasks.The forward model predicts left-to-right and the backward model right-to-left.Source: §3 · p. 3, Figure 3 · p. 4

4.1. Image Classification with Linear Models

• On VOC07, VirTex-100% reaches mAP 88.7 versus IN-sup-100% 87.6 using 118K versus 1.28M images, and VirTex outperforms IN-sup at similar image counts.With the same number of captions, VirTex-50% using one caption per image reaches mAP 79.4 versus 69.3 for VirTex-10% using five captions per image.Source: Figure 4 · p. 5, §4.1 · p. 6
• VirTex has the best performance-versus-cost tradeoff among COCO pretraining methods, supporting captions as more cost-efficient supervision than labels or masks.The comparison considers self-supervision, multi-label classification, instance segmentation, and VirTex caption training.Source: §4.1 · p. 5
• On ImageNet-1k classification, VirTex-100% outperforms IN-sup-10% (53.8 vs. 53.6) despite using 118K vs. 128K images; comparison is unfair because IN-sup models train for the downstream task using the downstream dataset.VirTex consistently outperforms IN-sup when both methods use fewer than 100K images.Source: §4.1 · p. 6

4.2. Ablations

• Bicaptioning beats forward captioning, and both outperform token classification; masked language modeling performs worse than all three.The authors suggest bidirectional modeling provides denser supervision, while sequential language structure helps visual features; MLM may suffer from poor sample efficiency.Source: Figure 5 · p. 6, §4.2 · p. 7
• Larger visual backbones improve VOC07 results but show opposite trends on ImageNet; wider and deeper textual transformers generally improve performance, with slight degradation at depth L = 4.The authors suggest the visual-backbone ImageNet trend may reflect an optimization issue, and that very deep transformers may overfit.Source: Figure 5 · p. 6, §4.2 · p. 7

4.4. Image Captioning

• VirTex models show modest COCO captioning performance, but their caption metrics surpass human performance despite weak correlation with human judgment.Attention visualizations show the model attending to relevant image regions when predicting tokens.Source: Figure 6 · p. 8, §4.4 · p. 8

4.3. Fine-tuning Tasks for Transfer

• Across four fine-tuning tasks, VirTex matches or exceeds ImageNet-supervised and MoCo-IN pretraining despite using 10× fewer images, and significantly improves LVIS performance.It also significantly outperforms methods using similar or more pretraining images; the authors attribute its LVIS gains to captions capturing long-tail visual concepts.Source: §4.3 · p. 8

• VirTex’s text-annotation-based visual representations are competitive with supervised-classification and self-supervised methods on ImageNet.Source: §5 · p. 8
• The work focuses solely on downstream vision tasks; future work could explore transferring both the visual backbone and textual head.Source: §5 · p. 8
• Captions offer a pathway to scaling VirTex to web-scale image-text pairs that are orders of magnitude larger, but noisier than COCO Captions.Source: §5 · p. 8

• The appendix adds implementation details for Section 4 experiments and follows prior evaluation protocols, reporting differences where applicable.Source: Appendix A · p. 13

A.1. Image Classification with Linear Models

• VOC images are resized so the shorter edge is 256 pixels, center-cropped to 224 × 224, and normalized with ImageNet RGB statistics.The same augmentation is applied to trainval and test images.Source: §A.1 · p. 13
• For VOC linear evaluation, the authors use per-class LinearSVC models and narrow the SVM cost sweep to C ∈ {0.01, 0.1, 1.0, 10.0} based on initial evaluations.They select the best SVM by 3-fold cross-validation; the stated LinearSVC setup uses an L2 penalty, squared-hinge loss, and class weights {1: 2, -1: 1}.Source: §A.1 · p. 13

Annotation Cost Efficiency

• Annotation-cost estimates use COCO-reported times for labels and masks and nocaps-reported median time per caption for captions.The caption estimate uses 39.2 seconds per caption for 118K × 5 COCO captions; mask estimates include label-collection time.Source: §A.1 · p. 13

Data Efficiency

• For data-efficiency experiments, ImageNet-supervised models use class-distribution-preserving subsets from 1% to 50%, while VirTex uses 10% to 100% of COCO Captions without class balancing.The authors note this may advantage ImageNet-supervised models; ImageNet training epochs remain fixed across subset sizes, while VirTex iterations scale with sampled-set size.Source: §A.1 · p. 13

Comparison: ImageNet vs. Cropped COCO

• 79.1 VOC07 mAP for ResNet-50 trained on 860K object-cropped COCO images, versus 88.7 for VirTex, shows VirTex’s data efficiency does not entirely stem from multi-object scene images.The crops expanded object bounding boxes by 0–30 pixels to mimic ImageNet-like images.Source: §A.1 · p. 13

Bicaptioning vs. Masked Language Modeling

• Masked Language Modeling performs worse than all other pretraining tasks on downstream linear classification, an issue attributed to its poor sample efficiency.The passage connects this sample-efficiency issue to the discussion in Section 3.Source: §A.2 · p. 14
• Masked LM converges more slowly than bicaptioning because it receives weaker supervision per caption, though a longer schedule may let it outperform bicaptioning.Both models use L = 1 and H = 2048 textual heads in the convergence comparison.Source: §A.2 · p. 14

Additional Evaluation: Backbone Ablations

• PASCAL VOC detection trends match VOC07 classification, supporting the conclusion that larger visual backbones can improve downstream performance despite opposite ImageNet-1k linear-classification trends.The authors attribute the opposing ImageNet trend to a possible optimization issue: ResNet-50 hyperparameters may not be optimal for other backbones.Source: §A.2 · p. 14

Fine-tuning configurations

• Detectron2 config files enable exact replication of downstream fine-tuning on COCO, PASCAL VOC, and LVIS, with parameters differing from base configs shown in Tables 5–7.The tables cover COCO instance segmentation, PASCAL VOC object detection, and LVIS instance segmentation.Source: §A.3 · p. 14, Table 5 · p. 14, Table 6 · p. 14

iNaturalist 2018 Fine-grained Classification

• For iNaturalist 2018 fine-grained classification, augmentation and weight initialization match ImageNet-1k linear classification, and class-balanced resampling is omitted following MoCo.The passage notes that iNaturalist has a long-tailed distribution like LVIS.Source: §A.3 · p. 14

LVIS v0.5 Instance Segmentation

• For LVIS v0.5, the config uses 1230 classes and a test score threshold of 0.0, following the original v0.5 baselines.The additional evaluation is reported for completeness because one MoCo baseline used the v0.5 splits.Source: §A.3 · p. 15
• VirTex significantly outperforms all baseline methods on LVIS v0.5, similar to its results on LVIS v1.0.The v0.5 evaluation uses LVIS v1.0 train and validation splits for the main evaluation, while one MoCo baseline used v0.5 splits.Source: §A.3 · p. 15

A.4. Selecting Best Checkpoint by VOC07 mAP

• Because captioning performance has a positive but imprecise correlation with downstream vision performance, VirTex selects its best checkpoint using VOC07 classification mAP.Source: §A.4 · p. 15, Figure 8 · p. 15
• 496K (VOC07 mAP 88.7) and 492K (CIDEr 105.8) were the respective best iterations, showing the metrics peak at different checkpoints.The metrics are for VirTex with a ResNet-50, L = 1, H = 2048 model.Source: §A.4 · p. 15, Figure 8 · p. 15
• PASCAL VOC linear classification serves as the representative downstream task for checkpoint selection because it is fast and simple to evaluate.Source: §A.4 · p. 15

Appendix B. Decoder Attention Visualizations for Caption Predictions

• Attention masks are normalized to [0, 1] to improve contrast and visibility.Source: Appendix B · p. 15
• Figure 9 shows decoder attention weights overlaid on input images at each time step of predicted captions from the forward transformer.Captions are decoded with beam search from an L = 1, H = 512 VirTex model.Source: Appendix B · p. 15, Figure 9 · p. 16
• Figure 10 visualizes decoder attention weights overlaid on the input image for a highlighted word in a caption.Captions are decoded with beam search from the forward transformer of the L = 1, H = 512 VirTex model.Source: Appendix B · p. 15, Figure 10 · p. 17