PaperlayerView exact revision on arXiv

Source-linked AI summary

OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents

Hugo Laurençon, Lucile Saulnier, Léo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Siddharth Karamcheti, Alexander M. Rush, Douwe Kiela, Matthieu Cord, Victor Sanh

arXiv:2306.16527v2cs.IRcs.CV

TL;DR

Large multimodal models benefit from naturally interleaved image–text documents, but prior training datasets were unavailable and poorly documented. OBELICS provides an open, filtered web-scale collection and trains IDEFICS to demonstrate its viability, with competitive benchmark performance and stronger efficiency than image-text-pair training in a reported experiment.

  • Problem

    Prior multimodal-document results were difficult to reproduce because their training datasets were unavailable and their creation processes and compositions were insufficiently documented.

  • Method

    The paper constructs OBELICS from Common Crawl by filtering and deduplicating web content while preserving text–image interleaving, then trains IDEFICS models on multimodal documents.

  • Results

    IDEFICS is on par with Flamingo on 4 of 8 evaluation tasks, while IDEFICS-9B achieves 56.5 aggregated performance versus 55.8 for OpenFlamingo-9B.

  • Takeaways & Limitations

    OBELICS provides an open alternative to closed multimodal web-document datasets for training competitive open models.

Abstract

from arXiv · show

Large multimodal models trained on natural documents, which interleave images and text, outperform models trained on image-text pairs on various multimodal benchmarks. However, the datasets used to train these models have not been released, and the collection process has not been fully specified. We introduce the OBELICS dataset, an open web-scale filtered dataset of interleaved image-text documents comprising 141 million web pages extracted from Common Crawl, 353 million associated images, and 115 billion text tokens. We describe the dataset creation process, present comprehensive filtering rules, and provide an analysis of the dataset's content. To show the viability of OBELICS, we train vision and language models of 9 and 80 billion parameters named IDEFICS, and obtain competitive performance on different multimodal benchmarks. We release our dataset, models and code.

1 Introduction

OBELICS addresses the limited reproducibility of multimodal-document research by openly providing a large curated collection of web pages that preserve text–image context. The dataset supports IDEFICS models trained on interleaved multimodal documents.

  • Motivation: Multimodal documents interleave text paragraphs and images, and models trained on them outperform models trained solely on image-text pairs on various benchmarks.They can also generate long, coherent text about multiple images.
  • Motivation: Existing multimodal-document results were difficult to replicate because their datasets were unavailable and their creation processes and compositions were poorly documented.This motivated openly accessible, high-quality collections of multimodal web documents.
  • Motivation: Image-text-pair datasets often rely on brief, non-grammatical alt-text and remove images from their natural page context.OBELICS instead preserves long-form text interleaved with page images.
  • Contribution: OBELICS contains 141 million multimodal English web documents, 353 million associated images, and 115 billion tokens.The dataset is openly accessible and curated at web scale.
  • Contribution: The authors train IDEFICS, an 80-billion-parameter multimodal model, to demonstrate OBELICS’s viability against large-scale multimodal models such as Flamingo.The paper describes dataset creation, filtering, curation, content, and limitations.

2 Related Works

Related work spans large image-text-pair datasets, open text corpora, and multimodal web-document datasets. OBELICS builds on these efforts while targeting the reproducibility and transparency gap around large multimodal-document collections.

  • Image-text pair datasets: Large image-text datasets such as LAION, Conceptual Captions, ALIGN, COYO, and DataComp are commonly created through web crawling and alt-text extraction.They support contrastive multimodal encoders and other vision-language systems.
  • Web document datasets: Open web-text datasets including c4, ROOTS, The Pile, and OSCAR provide increasingly large collections for language-model training.LLaMA used 1.4T tokens from openly accessible English web content.
  • Multimodal web document datasets: OBELICS extends open curated web-document datasets by extracting images alongside text, changing the resulting collection from text-only documents to multimodal documents.This positions OBELICS between open web-text corpora and prior multimodal-document datasets.
  • Multimodal web document datasets: Flamingo and KOSMOS-1 were trained on large multimodal web-document collections, but those datasets were not publicly available and were sparsely documented.The limited information covered dataset content, filtering strategies, and resulting document quality.
  • Multimodal web document datasets: mmc4 made a multimodal web-document dataset accessible, containing 103 million documents and 585 million images.OBELICS uses more recent documents and describes more comprehensive filtering based on HTML DOM trees.

3 Creation of the Multimodal Web Document Dataset

OBELICS is constructed from recent Common Crawl data through language, quality, HTML, multimodal, and deduplication filters. The pipeline preserves webpage structure while reducing undesirable content and produces a large interleaved document collection.

  • Collection and preprocessing: The pipeline begins with 25 recent Common Crawl dumps, extracts main text, and removes documents with insufficient quality.The initial processing yields 41.2 billion documents before further filtering.
  • Collection and preprocessing: FastText language filtering, MinHash deduplication, repetition filtering, and logistic-regression quality classification reduce the corpus to 1.1 billion documents with HTML sources.The FastText classifier removes 63.6% of documents.
  • HTML preprocessing: HTML DOM preprocessing identifies relevant text and image nodes, removes generic material such as spam and advertisements, and produces simplified HTML files more than ten times smaller.The rules are based on manual inspection of HTML-node types.
  • Multimodal extraction: Texts and image links are extracted from simplified HTML into documents that interleave modalities while preserving DOM-defined line breaks and visual rendering.This retains the original webpage structure in the multimodal representation.
  • Filtering and deduplication: Node-, paragraph-, and document-level filters remove low-quality images, undesirable paragraphs, and documents with unsuitable image counts or text.Additional measures address consent, pornographic content, URL repetition, document duplication, and cross-document paragraph duplication.
  • Filtering and deduplication: 141 million documents and 353 million images remain after filtering and deduplication, including 298 million unique images.Stricter thresholds produce fewer documents but not necessarily higher quality.

4 Analysis of OBELICS

The analysis characterizes OBELICS’s scale, image distribution, text quality, topic diversity, and image–text relevance. It also reports manual quality checks and comparisons with existing datasets.

  • General statistics: OBELICS has the highest number of unique documents and total tokens among the alternatives compared, while containing a large number of images.The comparison notes that mmc4 has more images but substantially lower image uniqueness.
  • General statistics: 84.3% of OBELICS images are unique, compared with 60.6% in mmc4.The authors attribute mmc4’s lower uniqueness partly to duplicated spam or generic content and uneven image counts across documents.
  • General statistics: OBELICS images have a median of 1 per document, while documents have a median of 677 tokens.Although documents can contain up to 30 images, the heatmap is truncated at 6 images for visual clarity.
  • Text quality: OBELICS text has significantly lower average perplexity than c4, mmc4, and OSCAR, with a distribution close to The Pile.Lower perplexity indicates greater resemblance to Wikipedia documents under the evaluation language model.
  • Content diversity: LDA analysis examines OBELICS topic diversity at both 20-topic and 200-topic granularities.The analysis reports topic distributions, estimated proportions, and frequently associated words.
  • Quality and relevance: Manual inspection found that 90% of documents had images clearly related to their text, while 7% contained content not captured by cleaning filters.Photo images appeared in 80% of documents, graphic images in 29%, and structured-text images in 5%.

5 Validating the Viability of OBELICS

The authors validate OBELICS by training IDEFICS models and comparing multimodal-document training with image-text-pair training and existing models. OBELICS-based training reaches comparable or better performance while using fewer images.

  • Models trained on OBELICS outperform models trained on image-text pairs across various multimodal benchmarks.
  • Model details: IDEFICS uses frozen LLaMA and OpenClip backbones, learnable cross-attention Transformer blocks, Perceiver pooling, and next-token prediction.Multimodal documents are represented as sequences of interleaved text paragraphs and images, while image-text pairs are packed with captions.
  • Training on different mixture of data: Training on multimodal web documents reaches the same performance with an order of magnitude fewer images than training on image-text pairs.The comparison uses datasets whose images come from Common Crawl and highlights the role of longer text contexts.
  • Models trained on OBELICS achieve competitive performance at different scales: IDEFICS is on par with Flamingo on several multimodal benchmarks and achieves aggregated performance of 56.5 versus 55.8 for OpenFlamingo-9B.At the 9-billion-parameter scale, IDEFICS remains behind Flamingo-9B but outperforms OpenFlamingo-9B trained on mmc4.

6 Conclusion

OBELICS is presented as an open, filtered collection of interleaved multimodal web documents designed to support open-source large multimodal models. The authors train IDEFICS to demonstrate that OBELICS can serve as an alternative to closed datasets.

  • OBELICS is an open web-scale collection of filtered interleaved multimodal web documents based on Common Crawl snapshots.
  • The collection process balances scale with removal of undesirable texts and images while addressing data consent and pornographic content.
  • Training IDEFICS on OBELICS demonstrates a viable alternative to closed datasets for open-source large multimodal models.
  • Open multimodal-document datasets with scale, quality, and source diversity can support training competitive open models.

Checklist

The checklist records that the paper addresses reproducibility, asset licensing, consent, personally identifiable information, offensive content, and potential negative societal impacts. It also documents trade-offs inherent in releasing a crawled-web dataset.

  • Societal impacts: The release is described as a trade-off between risks from crawled web pages and future work enabled by a large, filtered dataset.The cited risks include images with faces, potential PII in texts, and offensive content.
  • Consent: Explicit consent from website authors was considered prohibitive at this scale, so the authors remove opted-out images instead.They acknowledge that this opt-out strategy cannot be exhaustive.
  • Reproducibility: The authors state that code, data, model-release plans, training details, and compute information are provided or addressed in the paper or appendices.
  • Data risks: The dataset uses publicly accessible websites and may contain offensive, insulting, or threatening texts and images prevalent on the web.
  • Data risks: The authors removed pornographic content and low-quality texts as much as possible but did not take additional intentional measures to remove personal information.Manual inspection of 250 random samples found no obvious personally identifiable information, although some is likely present.

A.1.1 Collecting of a Large Number of HTML Files

OBELICS begins with recent Common Crawl dumps, extracts and filters English web documents, deduplicates repeated content, and classifies pages for human-written quality. These stages reduce the initial crawl to 1.1 billion documents before multimodal filtering.

  • The collection starts from the 25 most recent Common Crawl dumps, spanning February 2020 to January/February 2023.Modified readability-lxml extracts main text and discards pages with excessively high perplexity, yielding 41.2 billion documents.
  • Selection of English content: FastText filtering removes 63.6% of extracted documents to identify non-English content.
  • Early text deduplication: MinHash deduplication uses 16 hashes calculated over 5-grams, while repeated-paragraph and repeated n-gram content is also removed.
  • Quality classification: A logistic regression classifier with hashed token frequencies retains pages likely to contain human-written text using a 0.5 curated-corpus probability threshold.
  • The preceding collection and filtering stages produce 1.1 billion documents with their HTML sources from Common Crawl WARC files.

A.1.2 Simplifying HTML Files

OBELICS simplifies raw HTML by parsing DOM trees, removing structural and stylistic noise, and deleting nodes associated with generic or difficult-to-linearize content. These rules reduce file size while retaining relevant textual and visual material.

  • DOM preprocessing: Simplified HTML is produced with selectolax, which parses raw HTML and creates DOM trees for efficient extraction and filtering.The preprocessing prioritizes simplifying HTML before extracting textual and visual elements.
  • DOM preprocessing: Cleaning converts line-break tags, condenses repeated whitespace, removes comments, and recursively eliminates empty leaves and unnecessary nesting.
  • Tag normalization: Styling-only tags are unwrapped so displayed text remains while formatting markup is removed.For example, <i>example</i> becomes example.
  • Node selection: The pipeline retains structural and media tags but removes nodes associated with logos, generic content, navigation, spam, and difficult-to-linearize text.Examples of removed noisy content include header and li elements, while p, h, img, figure, and related tags are retained.
  • Targeted removal: Nodes identified through footer, header, navigation, date, site-info, or more-link markers are specifically removed to suppress navigation, article dates, and topic-shift content.
  • Outcome: More than 10× average HTML-size reduction is achieved while preserving interesting content.

A.1.3 Extracting Multimodal Web Documents

The extraction stage converts simplified HTML into structured web documents composed of interleaved text and images. It preserves DOM order and formatting while downloading images at scale through a distributed process.

  • Document construction: The extraction objective is to transform simplified HTML into a sequence of interleaved texts and images.
  • Structure preservation: Texts and image links are extracted in their DOM order, preserving separations, line breaks, line feeds, and the page’s formatting.
  • Image downloading: 3.6 billion images are targeted for download with img2dataset, and 55%—approximately 2 billion—are successfully downloaded.
  • Image downloading: 20 virtual machines complete the distributed image-download operation within a few days.

A.1.4 Filtering Multimodal Web Documents

OBELICS filters multimodal documents at node, paragraph, and document levels to remove low-quality or irrelevant content. The process yields 365 million web documents and 1.4 billion potentially duplicated images after filtering.

  • Filtering levels: Filtering operates first on image nodes and text paragraphs, then on complete documents, with criteria that can modify or remove content.Text filters are adapted from Laurençon et al. (2022).
  • Image filtering: Images are discarded when their formats, dimensions, aspect ratios, or URLs indicate low-quality, disproportionate, logo, or generic content.Removed URL substrings include logo, button, icon, plugin, and widget.
  • Text filtering: Paragraph filters remove text with fewer than 4 words, excessive repetition or special characters, low stop-word or punctuation ratios, flagged words, high perplexity, or generic spam sentences.KenLM models trained on Wikipedia are used to identify excessively high perplexity.
  • Text filtering: Node- and paragraph-level filters retain only high-quality and relevant images and paragraphs for subsequent processing.
  • Document filtering: Documents with no images or more than 30 images are removed because excessive image counts often indicate unrelated or spam content.Document-level text filters use stricter cutoff values than paragraph-level filters.
  • Outcome: 365 million web documents and 1.4 billion images remain after filtering, with images potentially duplicated across documents.

A.1.5 Additional Filtering and Deduplication Steps

The topic analysis reveals a broad mix of cultural, political, scientific, technological, and everyday-life content, with substantial variation in topic prevalence.

  • The dataset spans topics including gaming, music, celebrity culture, environmental conservation, physical sciences, politics, education, technology, and travel.
  • Several specialized topics are relatively rare, including Surfing/Beach Culture at 0.02%, Fantasy/Mythology at 0.03%, and European Culture at 0.04%.
  • Political topics include Political Campaigns at 1.25%, US Legislation at 0.75%, and Financial Regulations at 0.57%.
Loading 2306.16527v2…