PaperlayerView exact revision on arXiv

Source-linked AI summary

Are we pretraining it right? Digging deeper into visio-linguistic pretraining

Amanpreet Singh, Vedanuj Goswami, Devi Parikh

arXiv:2004.08744v1cs.CVcs.CL

TL;DR

The choice of pretraining datasets has received less attention than architecture and objective design, leaving open how dataset domain similarity relates to downstream performance. The paper studies varied textual and visual domain matches across pretraining datasets and downstream tasks. It reports that closer-domain synthetic data can outperform a commonly used natural dataset, while suitable pretraining choices can yield near-state-of-the-art results without extra data or architectural changes.

  • Problem

    Pretraining dataset choice has received little attention, including whether textual and visual domain similarity to downstream tasks should guide selection.

    Source: §1 · p. 2

  • Method

    The authors empirically compare three pretraining datasets with varied domain similarities across four downstream tasks and generate a synthetic dataset closer to a downstream domain.

    Source: §1 · pp. 2–3, §5 · p. 14

  • Results

    Closer-domain synthetic data outperforms a commonly used natural dataset, and suitable pretraining choices achieve near-state-of-the-art performance without extra data or architectural changes.

    Source: §1 · p. 3, §5 · p. 14

  • Takeaways & Limitations

    The findings suggest that synthetic labelling of unlabelled data may help address reliance on paired labelled data in scaling visio-linguistic pretraining.

    Source: §1 · p. 3

    The findings are specific to the chosen pretraining and downstream datasets, although the authors say these cover a wide span of scenarios.

    Source: §2.2 · p. 6

Abstract

from arXiv · show

Numerous recent works have proposed pretraining generic visio-linguistic representations and then finetuning them for downstream vision and language tasks. While architecture and objective function design choices have received attention, the choice of pretraining datasets has received little attention. In this work, we question some of the default choices made in literature. For instance, we systematically study how varying similarity between the pretraining dataset domain (textual and visual) and the downstream domain affects performance. Surprisingly, we show that automatically generated data in a domain closer to the downstream task (e.g., VQA v2) is a better choice for pretraining than "natural" data but of a slightly different domain (e.g., Conceptual Captions). On the other hand, some seemingly reasonable choices of pretraining datasets were found to be entirely ineffective for some downstream tasks. This suggests that despite the numerous recent efforts, vision & language pretraining does not quite work "out of the box" yet. Overall, as a by-product of our study, we find that simple design choices in pretraining can help us achieve close to state-of-art results on downstream tasks without any architectural changes.

1 Introduction

Visio-linguistic pretraining has largely relied on intuitive dataset choices, despite limited study of how pretraining and downstream domains should match. The paper compares domain similarity across datasets and tasks, finding that a closer-domain synthetic dataset can outperform a commonly used natural dataset.

1 Introduction

  • The paper questions intuitive pretraining choices, including whether Conceptual Captions’ size or COCO Captions’ less noisy text makes either the better fit.It asks whether downstream domain should guide dataset selection and whether closer-domain automatically generated data can beat natural data from a different domain.Source: §1 · p. 2
  • The study varies textual and visual domain similarity across three pretraining datasets—VQA 2.0, COCO Captions, and Conceptual Captions—and four downstream tasks.The tasks are VQA, VizWiz, SNLI-VE, and MM-IMDB; VQA and COCO share images but differ in questions versus captions.Source: §1 · pp. 2–3
  • A synthetic dataset closer to the downstream domain achieves better performance than a more natural, commonly used dataset with a poorer domain match.The authors present this as a possible way to address the scalability bottleneck created by reliance on paired labelled data.Source: §1 · p. 3
  • The authors investigate how pretraining dataset domain affects downstream performance and how closer domain matching may improve it.They frame the work as empirical analysis of pretraining choices and representations, with the aim of achieving near state-of-the-art performance without extra data or architectural changes.Source: §1 · p. 3

2 Study Setup

The study compares pretraining datasets with different visual and textual domains across four downstream tasks, using two transformer architectures and masked pretraining objectives. Its setup also tests whether dataset scale and domain overlap matter for transfer.

2.1 Pretraining Datasets

  • COCO Captions, VQA 2.0, and Conceptual Captions vary in dataset characteristics and domain, enabling comparisons that isolate textual and visual domain differences.COCO and VQA 2.0 share images but differ in text; COCO and Conceptual Captions share caption text but differ in image sources.Source: §2.1 · pp. 3–4

2.2 Downstream Datasets

  • Four downstream datasets were selected to span visual and textual domain matches and mismatches, including small datasets intended to probe low-resource transfer.MM-IMDB matches neither pretraining domain, while VizWiz and MM-IMDB serve as small-data settings; the authors note that end-to-end fine-tuning can mask representation generalizability and limit findings to the chosen datasets.Source: §2.2 · pp. 5–6

2.3 Models

  • The study covers single-stream VisualBERT and dual-stream ViLBERT architectures, which differ in how they process and combine visual and textual inputs.ViLBERT uses separate visual and linguistic transformer streams connected by co-attention blocks.Source: §2.3 · p. 6

2.4 Pretraining Objectives

  • VisualBERT is trained with masked language modeling, while ViLBERT uses masked multimodal modeling, which combines masked text and image-region prediction objectives.For VQA-P pretraining, question-answer pairs provide the text input, with answers randomly sampled from human annotations.Source: §2.4 · pp. 6–7

2.5 Experimental Setup

  • Matching visual and textual domains align with maximum performance; CC-Small performs worst in most cases, and pretraining does not work on MM-IMDB.The table caption reports these comparisons across the downstream datasets.Source: Table 1 · p. 8
  • VQA-P and Conceptual Captions were clipped to COCO’s sample count for fair comparison, and Conceptual Captions was also tested from 10% of its data to full size.The smaller clipped subset is called CC-small.Source: §2.5 · p. 8

3 Life, the Universe and Pretraining

The experiments examine how pretraining-data choice and dataset size affect downstream performance, transferability, and scalability. They find that domain match matters, pretraining is not uniformly beneficial, and synthetic in-domain captions can outperform natural out-of-domain data while remaining below direct in-domain pretraining.

3.1 How should one choose the pretraining dataset?

  • On VQA-D, VizWiz, and SNLI-VE, pretraining datasets matching downstream visual or textual domains outperform CC-Small, while direct training works better on MM-IMDb.On SNLI-VE, COCO, which matches the visual domain, outperforms the other pretraining datasets.Source: §3.1 · p. 9
  • More data improves VQA-D performance, but VQA-P and COCO outperform CC at matched sizes, and full-scale CC still trails both.For VQA-P and COCO, gains diminish from 0.04M to 0.2M examples before improving again at 0.4M.Source: §3.1 · pp. 9–10, Figure 3 · p. 9

3.2 Does pretraining always help?

  • Pretraining can hurt or fail to help: MM-IMDB performs best with direct training, and frozen ViLBERT pretrained on COCO or CC-Small does not benefit downstream tasks.For SNLI-VE, CC-Small also fails to help when the model is fine-tuned.Source: §3.2 · pp. 10–11

3.3 Which pretrained representations are more transferable?

  • VQA-P pretraining yields more transferable representations for VQA-D, VizWiz, and SNLI-VE, while MM-IMDB results vary by model.The self-attention weights pretrained on VQA-P are closer to their VQA-D fine-tuned weights than those from other datasets.Source: §3.3 · p. 11, Figure 4 · p. 11

3.4 Pretraining Scalability and a Promising Alternative

  • Generated captions on CC images using a COCO-trained captioner make CCG closer to VQA-D; CCG outperforms CC on both model architectures but remains below VQA-P and COCO.Increasing CCG size further improves performance; surpassing COCO is a possibility the authors leave for future work.Source: Figure 3 · p. 9, §3.4 · p. 12
  • CCG-pretrained ViLBERT weights are closer to VQA-D fine-tuned weights than CC-pretrained weights in both L1 and angular distance.The comparison averages distances per layer.Source: §3.4 · p. 12, Figure 5 · p. 12

3.5 ViLBERT vs VisualBERT

  • VisualBERT consistently outperforms ViLBERT under the tested settings, despite ViLBERT's extra masked-image loss and double the parameters.The authors note that they used fewer loss terms than the original papers because additional terms did not help in their settings.Source: §3.5 · p. 13

3.7 Does all this help in pushing the SoTA?

  • 77.57% on SNLI-VE and 68.04% micro-F1 on MM-IMDB are state-of-the-art among published works; VQA 2.0 and VizWiz results are near state of the art.The reported single-model results use no extra data or ensembles.Source: §3.7 · p. 13

4 Related work

Related work describes the development of self-supervised pretraining in NLP, vision, and vision-language tasks. It notes that vision-language pretraining depends on labeled image-caption data, creating a scalability bottleneck that this paper addresses with synthetic data and design choices.

Related work

  • NLP pretraining commonly masks words, while vision methods use varied prediction or classification pretext tasks and vision-language models adapt shared architectures to multiple downstream tasks.Vision-language models use either interacting transformer streams or a shared projected feature space, with objectives including masked image-caption modeling, image-caption matching, and masked object-feature modeling.Source: §4 · pp. 13–14
  • Vision-language pretraining relies on labeled image-caption pairs, limiting large-scale self-supervised learning because aligned data are sparse.The paper presents synthetic data as a way to address this bottleneck.Source: §4 · p. 14

5 Conclusion

The study finds that pretraining-data source domains matter and that vision-language pretraining does not work out of the box. It reports that careful choices can yield near-state-of-the-art results without extra data, and synthetic labeling can scale pretraining.

Conclusion

  • The study's empirical analysis finds that pretraining source domain matters and that appropriate pretraining choices can produce near-state-of-the-art performance without extra data.It also presents synthetic labeling of unlabelled data as a step toward scaling pretraining despite the labeled-caption bottleneck.Source: §5 · p. 14

A Details of Experimental Setup

The appendix lists hyperparameters for different pretraining setups and notes that downstream fine-tuning reproduction details will be released with the code.

A Details of Experimental Setup

B Pretraining

Pretraining validation losses vary across datasets and objectives, while larger CC subsets and longer pretraining improve validation performance.

B Pretraining

  • COCO and CC achieve the lowest and highest validation losses, respectively, across same-size datasets trained for the same iterations.Both models achieve approximately the lowest MLM loss using COCO; ViLBERT’s lowest MIM loss is on CC-Small, only slightly better than COCO and VQA-P.Source: B Pretraining · p. 19, Table B.1 · p. 20
  • Increasing CC dataset size consistently improves validation performance when training for the same number of epochs.The comparison uses 10%, 50%, and 100% subsets with 30 epochs; longer pretraining is also indicated by the difference between 88K and 12K iterations.Source: Table B.1 · p. 20, B Pretraining · p. 20

C L1 and Angular Distances

The appendix compares how self-attention weights change between pretraining and fine-tuning, finding closer distances for pretrained and in-domain models in the reported comparisons.

C L1 and Angular Distances

D Synthetic data pretraining

The section evaluates generated CC pretraining on VizWiz and SNLI-VE, reporting competitive or better performance than CC pretraining across the two tasks.

D Synthetic data pretraining

  • CCG gives VisualBERT better downstream performance on VizWiz and ViLBERT performance close to CC, while SNLI-VE results are comparable but not better.The authors attribute the SNLI-VE outcome to CCG captions being closer in domain to COCO/VQA-D than to SNLI-VE.Source: D Synthetic data pretraining · p. 22, Table D.1 · p. 22

E Conceptual Captions and COCO Domain Comparison

The comparison finds that COCO and Conceptual Captions have contrasting visual domains, based on object counts and category distributions. COCO images contain more detected objects on average, while CC images are often focused on a few objects.

  • The top Visual Genome category distributions differ starkly between COCO and CC, supporting the authors’ conclusion that their visual domains contrast.The comparison uses random samples of 100 validation images from each dataset.Source: E Conceptual Captions and COCO Domain… · p. 22, Figure E.1 · p. 23
  • COCO averages 44.46 objects per image versus 31.95 for CC, which the authors associate with COCO’s broader image diversity and CC’s few-object focus.The authors describe CC images as frequently Wikipedia-style and focused on only a few objects.Source: E Conceptual Captions and COCO Domain… · p. 22
  • To compare object instances, the authors use a Visual Genome-trained detector to extract 100 proposals per image and ignore background objects.The detector was trained on 1,600 Visual Genome object classes; CC has no object bounding-box annotations.Source: E Conceptual Captions and COCO Domain… · p. 22

Related papers

Loading 2004.08744v1…