PaperlayerView exact revision on arXiv

Source-linked AI summary

WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learning

Krishna Srinivasan, Karthik Raman, Jiecao Chen, Michael Bendersky, Marc Najork

arXiv:2103.01913v2cs.CVcs.CLcs.IR

TL;DR

Multimodal models need larger, higher-quality, and more multilingual image-text datasets than existing public resources provide. WIT addresses this gap by extracting and filtering Wikipedia-associated image-text data, yielding broad coverage and strong alignment, while its specificity makes transfer to terse caption benchmarks challenging.

  • Problem

    Existing public image-text datasets are much smaller than unimodal resources and provide limited language coverage, constraining multilingual multimodal research.

  • Method

    WIT extracts multiple image-associated texts from Wikipedia articles and Wikimedia links, then applies rigorous filtering to retain high-quality image-text associations.

  • Results

    WIT contains over 37.6 million image-text sets spanning 11.5 million unique images and 108 languages, with 98.5% of sampled associations judged favorably.

  • Takeaways & Limitations

    WIT supports multimodal pretraining, image-text retrieval fine-tuning, and cross-lingual representation building as a rich multilingual resource.

  • Takeaways & Limitations

    WIT-trained models perform extremely well on Wikipedia-based evaluations but underperform on Flickr30K, Multi30k, and MS-COCO, partly because WIT texts are specific and verbose rather than terse captions.

Abstract

from arXiv · show

The milestone improvements brought about by deep representation learning and pre-training techniques have led to large performance gains across downstream NLP, IR and Vision tasks. Multimodal modeling techniques aim to leverage large high-quality visio-linguistic datasets for learning complementary information (across image and text modalities). In this paper, we introduce the Wikipedia-based Image Text (WIT) Dataset (https://github.com/google-research-datasets/wit) to better facilitate multimodal, multilingual learning. WIT is composed of a curated set of 37.6 million entity rich image-text examples with 11.5 million unique images across 108 Wikipedia languages. Its size enables WIT to be used as a pretraining dataset for multimodal models, as we show when applied to downstream tasks such as image-text retrieval. WIT has four main and unique advantages. First, WIT is the largest multimodal dataset by the number of image-text examples by 3x (at the time of writing). Second, WIT is massively multilingual (first of its kind) with coverage over 100+ languages (each of which has at least 12K examples) and provides cross-lingual texts for many images. Third, WIT represents a more diverse set of concepts and real world entities relative to what previous datasets cover. Lastly, WIT provides a very challenging real-world test set, as we empirically illustrate using an image-text retrieval task as an example.

1 INTRODUCTION

Multimodal datasets lag behind unimodal datasets in scale and language coverage, limiting multilingual multimodal research. WIT addresses this gap with a large, curated, multilingual image-text dataset and demonstrates its value for retrieval and evaluation.

  • Motivation: Existing image-text datasets are much smaller and mostly English compared with large text-only and image-only resources.Flickr has 30K examples and Conceptual Captions has 3.3M, while multilingual coverage is also limited.
  • Contribution: WIT contains over 37.6 million image-text sets spanning 11.5 million unique images across 108 languages.Each language has at least 12K examples, and 53 languages have over 100K image-text pairs.
  • Contribution: WIT extracts multiple image-associated texts from Wikipedia articles and Wikimedia links, retaining only high-quality image-text associations.The extraction includes fields such as reference descriptions and associated metadata.
  • Quality: 98.5% of randomly sampled WIT image-text associations received favorable judgments in an evaluation covering nearly 4,400 examples and 13K judgments across seven languages.The dataset leverages Wikipedia’s editing, verification, and correction mechanisms and was checked through human annotation.
  • Evaluation: WIT supports image-text retrieval experiments in both zero-shot and finetuned settings and provides a challenging real-world evaluation set.Its texts and images are grounded in diverse real-world entities and attributes.

2 RELATED WORK

Prior visio-linguistic datasets trade off human-caption quality, scale, filtering complexity, and language coverage. WIT combines extraction with Wikipedia’s curated multilingual repository to address these limitations.

  • Visio-Linguistic Datasets: Flickr30K and MS-COCO use crowdsourced captions that help ensure quality but remain insufficient for significant real-world improvements.Flickr30K contains about 30K images, while MS-COCO contains about 330K examples.
  • Visio-Linguistic Datasets: Conceptual Captions scales to about 3.3M image-text pairs through web-crawled alt text but requires complex filtering to maintain quality.This extraction paradigm is difficult to extend and has limited coverage and diversity.
  • WIT: WIT combines an extractive approach with a clean, curated multilingual repository containing images, illustrations, and detailed text descriptions.The repository is Wikipedia and its accompanying Wikimedia images.
  • VL Models: Multimodal models such as ViLBERT, UNITER, VL-BERT, VisualBERT, ImageBERT, B2T2, and Unicoder-VL commonly pretrain on CC, MS-COCO, or similar datasets.These models use objectives including image-text matching and word-region alignment.

3 WIT: WIKIPEDIA IMAGE TEXT DATASET

WIT combines Wikipedia’s curated, multilingual content with multiple image-associated texts and contextual metadata, then applies filtering to produce a large, diverse image-text dataset.

  • Wikipedia Crawl Data: WIT extracts images, associated texts, and contextual metadata from Wikipedia and Wikimedia across roughly 124M content pages.The extraction yielded about 150M image-text-context tuples before subsequent filtering.
  • The Texts used in WIT: Reference, attribution, and alt-text descriptions provide complementary image-associated text fields, while page and section text supply additional context.Reference descriptions are topical and relevant; attribution text can be multilingual, whereas alt-text is often uninformative.
  • Filtering: WIT cleans low-information text and filters images for dimensions, meaningful associations, permissible licenses, duplication, and inappropriate content.The quality filters eliminate less than 0.2% of all tuples for inappropriate content, while other filters remove generic or noisy examples.
  • Analyzing the WIT Data: The final dataset contains over 37M tuples spanning 108 languages and 11.5 million unique images.WIT also provides multiple texts per image, with 19.4M tuples containing at least two of the three primary text types.
  • Analyzing the WIT Data: WIT provides nearly 32M unique image-text pairs, 50M+ cross-lingual pairs from 3.1M images, and roughly 120M contextual texts.Nearly half of the 100+ languages contain 100K+ unique image-text tuples and 100K+ unique images.
  • Human Annotator Validation: Human evaluation used three questions and a three-point scale to assess description quality and contextual relevance.The process covered nearly 4,400 image-text examples and 13K judgments across seven languages; most examples were judged helpful.

4 MULTIMODAL EXPERIMENTS WITH WIT

The experiments use a dual-encoder retrieval model to test WIT as both a pretraining resource and an evaluation dataset across several multilingual and multimodal benchmarks.

  • Multimodal Experiments with WIT: The study evaluates WIT as a pretraining dataset and as an evaluation set for image-text retrieval.The downstream evaluation is performed without finetuning in the zero-shot setting described later.
  • Model: The model has separate text and image encoders whose final embeddings are combined using cosine similarity.The text input is represented as a bag of words, while the image tower uses an image embedding procedure inspired by prior work.
  • Evaluation: Experiments evaluate Flickr30K, Multi30K, MS-COCO, and WIT test sets, including English-only and non-English WIT subsets.WIT-pretrained models use the entire multilingual training set, and Conceptual Captions is used as a pretraining comparison.
  • Evaluation: Recall@K with K = 1, 3, 5 is used to measure image-text retrieval performance.Training uses batch size 128, evaluation uses batch size 1000, and optimization uses SGD with Momentum.

4.2 Evaluating a zero-shot pretrained model

Zero-shot WIT-pretrained models perform strongly on WIT’s English and non-English test sets, while performance depends on the evaluation collection and language distribution.

  • Evaluating a zero-shot pretrained model: WIT models perform well on both English and non-English WIT test sets without finetuning.Concatenating reference and attribution descriptions consistently gives the strongest performance among the tested WIT text configurations.
  • Evaluating a zero-shot pretrained model: Conceptual Captions-pretrained models lag behind WIT-pretrained models on WIT test sets, including the English-only set.This comparison indicates that WIT’s advantage is not limited to its non-English evaluation subset.
  • Evaluating a zero-shot pretrained model: On English Flickr30K and MS-COCO collections, the multilingual WIT model trails the English Conceptual Captions model.The gap is smaller than the difference observed between WIT and Conceptual Captions on held-out WIT test sets.

4.3 Understanding multilingual performance

The multilingual evaluation compares WIT-based models on Czech, German, and French subsets of Multi30K and WIT test data.

  • Understanding multilingual performance: Both models struggle on Multi30K, whereas the WIT model performs strongly on the held-out WIT test set.The evaluation uses Czech, German, and French test sets and corresponding WIT subsets.

4.4 Evaluation On (Image, Wiki Page Title) Retrieval Task

WIT is evaluated on a Wikipedia-based retrieval task that uses page titles to identify images, with both zero-shot and finetuned settings.

  • The task identifies images appearing on a Wikipedia page using only the page title as text input.Evaluation uses the held-out WIT test split.
  • Evaluation includes both zero-shot pretrained-model and finetuned-model settings.
  • The held-out WIT test split supports evaluation of retrieval grounded in Wikipedia pages.

4.5 Discussion

WIT-based models perform strongly on Wikipedia-derived evaluations but less well on conventional image-text collections. The discussion attributes this gap to differences in vocabulary, entity diversity, and text specificity.

  • 5x+ gains are observed on evaluation sets based on Wikipedia data, whereas performance is weaker on Flickr30K/Multi30k and MS-COCO.
  • 72% of WIT unigrams occur 3 times or less, compared with 43% for CC, indicating greater textual vocabulary diversity.
  • WIT and conventional evaluation collections show a massive vocabulary and concept-coverage gap under Jensen-Shannon Divergence analysis.The gap remains sizeable even for WIT’s English-only slice.
  • At least 80% of approximately 4.5M image entities occur 3 times or less, showing limited repetition in WIT’s image data.This corresponds to approximately 3.68M entities.
  • WIT text is often verbose and specific, whereas test collections commonly use terse captions containing common words and objects.The mismatch may be exacerbated by bag-of-words representations.
  • Hypernymization could remove names, locations, and times, but applying it consistently across more than 100 languages remains challenging.

5 FUTURE WORK

Future work extends WIT beyond the initial image-text retrieval experiments, including stronger multimodal architectures and additional multilingual tasks.

  • The initial study uses a simple dual encoder for image-text retrieval.
  • WIT could replace or supplement existing pretraining datasets for cross-attention multimodal transformer models.
  • The authors identify opportunities to formulate a range of new internationalization tasks with WIT.

6 CONCLUSION

The paper presents WIT as a large, multilingual, context-rich dataset for multimodal research. Its quality, alignment, and utility are supported by analysis and retrieval experiments, while its diversity exposes shortcomings in existing datasets.

  • WIT is described as the largest, multilingual, multimodal, context-rich dataset at the time of writing.
  • Texts associated with images and surrounding contexts are extracted from more than 100 languages, producing a rich and diverse dataset.
  • Detailed analysis and quality evaluation validate WIT’s high quality and strong image-text alignment.
  • Experiments demonstrate WIT as both a pretraining and finetuning resource for image-text retrieval models.
  • The analysis uncovers shortcomings of existing datasets and positions WIT as a resource for multilingual, multimodal research.

A APPENDIX

The appendix includes an annotated WIT image-text example for Half Dome, Yosemite, alongside multiple language-statistics tables.

  • Figure 5 is titled “WIT Image-Text Example with All Text Annotations.”
  • Table 12 presents a full WIT example illustration for Half Dome, Yosemite.
Loading 2103.01913v2…