OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents

Motivation

• Naturally occurring multimodal documents outperform image-text-pair training on various benchmarks and support long, coherent text generation about multiple images.These documents consist of text paragraphs interleaved with images, such as web pages containing images.Source: §1 · p. 1

Problem

• Prior results have not been replicable because the datasets behind them are unavailable and their creation processes and compositions are poorly documented.This motivates large-scale collections of high-quality multimodal web documents for developing the next generation of models.Source: §1 · p. 1

Motivation

• Image-text pairs rely mainly on brief, approximate, often ungrammatical alt-text and remove images from their natural page context and relationships with other content.Figure 1 contrasts short or non-grammatical image alt-text with OBELICS extraction that interleaves long-form text and page images.Source: §1 · p. 2, Figure 1 · p. 2

Contribution

• OBELICS contains 141 million English multimodal web documents, 353 million associated images, and 115 billion text tokens, with text and images preserved in document context.The authors describe its creation, filtering, curation, content, and limitations, and release the dataset, models, and code.Source: §1 · p. 2
• IDEFICS, an 80 billion parameter multimodal model trained using OBELICS, achieves competitive performance against large-scale multimodal models such as Flamingo (Alayrac et al., 2022).The model is presented as evidence that the openly accessible dataset can support viable large multimodal systems.Source: §1 · p. 2

Image-text pairs datasets

• Image-text pair datasets contain billions of web-crawled image-text pairs, but OBELICS extracts images and the textual context in which they appear instead of associated alt-text.These datasets support contrastive multimodal encoders and image generation models.Source: §2 · p. 2

Web document datasets

• OBELICS extends large curated web-document datasets such as c4, ROOTS, Pile, OSCAR, and LLaMA’s 1.4T-token corpus by additionally extracting images, enabling models with vision capabilities.The related web-scale datasets were introduced to meet the need for increasingly larger training corpora.Source: §2 · pp. 2–3

Multimodal web document datasets

• Flamingo and KOSMOS-1 were trained on tens of millions of multimodal web documents, but their datasets are unavailable and insufficiently documented, hindering further research.Flamingo used 43 million multimodal web documents, while KOSMOS-1 used 71 million.Source: §2 · p. 3
• Compared with mmc4, OBELICS uses newer February 2020–February 2023 documents, more comprehensive HTML-DOM filtering, and additional image-, document-, and paragraph-level deduplication.mmc4 contains 103 million multimodal web documents and 585 million images, whereas its image-document matching uses CLIP similarities and a bipartite assignment problem.Source: §2 · p. 3

3.1 Collecting a Large Number of HTML Files

• The pipeline filters extracted text for non-English content with FastText, removes 63.6% of documents, deduplicates with MinHash and repeated-paragraph or n-gram filters, then leaves 1.1 billion documents.A logistic regression classifier with hashed token frequencies uses curated datasets like Wikipedia or OpenWebText as positive examples and Common Crawl samples as negative ones.Source: §3.1 · p. 4

3.2 Simplifying HTML Files

• Simplifying HTML with DOM-specific rules makes files more than ten times smaller while removing generic text, spam, ads, boilerplate, logos, and other generic images.The preprocessing prioritizes retaining information useful for extracting textual and visual elements.Source: §3.2 · p. 4

3.3 Extracting Multimodal Web Documents

• The simplified pages are converted into structured documents that interleave text and images while preserving DOM order, line breaks, formatting, and visual rendering.The process extracts text and image links and obtains 3.6 billion image links, successfully downloading 55% of them.Source: §3.3 · p. 4

3.4 Filtering Multimodal Web Documents

• Filtering operates at image-node and text-paragraph levels before document-level retention, leaving 365 million web documents and 1.4 billion images.Image filters remove unsuitable dimensions and logo-like URLs, while text filters target short, repetitive, malformed, inappropriate, boilerplate, and machine-generated content.Source: §3.4 · pp. 4–5

3.5 Responsible Filtering and Deduplication

• Responsible filtering removes opted-out and pornographic images, deduplicates repeated images and documents, and eliminates approximately 15% of text through within-domain paragraph deduplication.Images appearing more than ten times are removed, while limited cross-document duplication is retained to preserve differing contextual information.Source: §3.5 · pp. 5–6
• The final dataset contains 141 million documents and 353 million images, of which 298 million are unique, after filtering and deduplication.Stricter filtering produces fewer multimodal documents but not higher quality, so the authors recommend random subsets for smaller datasets.Source: §3.5 · p. 6

4.1 General Statistics

• OBELICS has the highest number of unique documents and total tokens among the largest alternatives while containing a huge number of images.Table 1 compares OBELICS with the largest existing alternatives, including mmc4-ff.Source: §4.1 · p. 6, Table 1 · p. 6
• OBELICS has a median of 1 image and 677 tokens per document, and its text has lower average perplexity than c4, mmc4, and OSCAR while closely matching The Pile.The dataset limits documents to 30 images; lower perplexity indicates greater resemblance to Wikipedia documents.Source: §4.1 · pp. 6–7, Figure 4 · p. 7, Figure 5 · p. 7

4.2 Topic Modeling

• LDA reveals topics ranging from Politics to Health and Music, while news sites are the most represented domains.The analysis uses 20-topic and 200-topic LDA results for high-level and granular views of dataset content.Source: §4.2 · p. 7

4.3 Qualitative Assessment of Dataset Samples

• In a manual sample, 90% of documents had images clearly related to the text, while 80% contained photos, 29% graphics, and 5% structured-text images.The inspection also found 7% of documents with content missed by cleaning filters, no NSFW images, no obvious PII except public or news-mentioned people, 3% with watermarks, and 2% with logos.Source: §4.3 · p. 7

Model details

• The models closely follow Flamingo, combining frozen LLaMA and OpenCLIP backbones with learnable cross-attention blocks and Perceiver pooling for interleaved documents or packed image-text pairs.Training uses standard next-token prediction and evaluates visual question answering, visual dialogs, hateful-speech detection, captioning, and OCR benchmarks.Source: §5 · p. 8

Training on different mixture of data

• Multimodal web-document training reaches the same performance with an order of magnitude fewer images than image-text-pair training and performs better on average.Image-text pairs retain a slight advantage on captioning, classification, and OCR, while combining both dataset types improves performance for a given number of images, tokens, or compute.Source: §5 · p. 8, Figure 6 · p. 8

Models trained on OBELICS achieve competitive performance at different scales

• The evaluation trains 80B- and 9B-parameter IDEFICS models on mixtures including LAION image-text pairs, captioning datasets, OBELICS, and Wikipedia-derived multimodal documents.These models are compared with same-size OpenFlamingo v2 and Flamingo models trained on similar mixtures.Source: §5 · p. 10
• 56.5 versus 55.8 aggregated performance shows IDEFICS-9B outperforming OpenFlamingo-9B, while IDEFICS matches or exceeds Flamingo on 4 of 8 tasks with 32 in-context examples.IDEFICS is also reported as often on par with Flamingo across multimodal benchmarks, though it remains behind Flamingo-9B at the 9B scale.Source: Table 2 · p. 9, §5 · p. 10

• OBELICS is an open web-scale collection of filtered interleaved multimodal web documents based on Common Crawl snapshots.Source: §6 · p. 10
• Training IDEFICS on OBELICS shows that multimodal documents can support a viable alternative to closed datasets.Source: §6 · p. 10
• The collection and filtering process balances scale with removal of undesirable texts and images while addressing data consent and pornographic content concerns.Source: §6 · p. 10
• Open multimodal-document datasets combining scale, quality, and diverse sources can support training competitive open models.Source: §6 · p. 10

Ethics and limitations

• The authors removed opted-out images to respect content creators, while acknowledging that this strategy cannot be exhaustive and inviting further opt-outs.Explicit consent from website authors was considered prohibitive given the dataset’s scale.Source: Checklist · p. 19

Checklist scope

• The work uses open-source libraries, cites their creators, discusses consent and data risks, and includes no theoretical results or human-subjects research.Crowdsourcing and participant compensation are marked not applicable.Source: Checklist · pp. 19–20

Reproducibility

• The authors will release the model and training code, document training details, and report compute resources, while error bars are marked not applicable.Training details are provided in Appendix A.4, and compute information is provided in an appendix.Source: Checklist · p. 19

Ethics and limitations

• The dataset may contain offensive, insulting, threatening, or personally identifiable content despite filtering and manual inspection, because it derives from publicly accessible websites.Pornographic content and low-quality texts were removed as much as possible, but no additional intentional personal-information removal was performed.Source: Checklist · pp. 19–20

A.1.1 Collecting of a Large Number of HTML Files

• 41.2 billion documents are obtained from 25 recent Common Crawl dumps spanning February 2020 to January/February 2023.Modified readability-lxml extracts main text, while pages with excessively high text perplexity are discarded.Source: §A.1.1 · p. 21
• 63.6% of documents are filtered out as non-English using a FastText classifier applied to extracted text.Source: §A.1.1 · p. 21
• Early deduplication uses 16-hash MinHash over 5-grams rather than URL matching, and removes documents with substantial repeated paragraphs or n-grams.URL-based deduplication is avoided because website content can change across Common Crawl snapshots.Source: §A.1.1 · p. 21
• A logistic regression classifier retains pages likely to contain human-written text, using hashed token frequencies and a 0.5 curated-corpus probability threshold.Wikipedia and OpenWebText provide positive examples, while Common Crawl samples provide negative examples.Source: §A.1.1 · p. 21
• The quality classifier is trained to distinguish curated human-written text from Common Crawl material before documents are retained.Curated datasets serve as positive examples and sampled Common Crawl documents as negative examples.Source: §A.1.1 · p. 21
• 1.1 billion documents and their HTML sources remain after applying the collection, language, deduplication, and quality filters.The retained documents come from associated Common Crawl WARC files.Source: §A.1.1 · p. 21

A.1.2 Simplifying HTML Files

• selectolax parses raw HTML into DOM trees, making subsequent text and image extraction more efficient.The preprocessing prioritizes simplified HTML because the original content supports filtering undesirable text and images.Source: §A.1.2 · p. 21
• DOM cleaning normalizes line breaks and spaces, removes comments, and recursively eliminates empty or unnecessarily nested nodes.These operations are repeated after node removal to further simplify the hierarchy.Source: §A.1.2 · p. 21
• Styling tags are unwrapped, while manually selected structural and media tags are retained and other nodes are removed as logos, spam, navigation, or difficult-to-linearize content.The retained set includes tags such as p, h, img, figure, iframe, and video; tags such as header, li, and table are removed.Source: §A.1.2 · p. 22
• Nodes associated with headers, footers, navigation, menus, and article dates are removed, while more-link nodes are replaced with an end-of-document marker for conversion to EOS during training.The more-link rule is treated as indicating a topic shift resembling the start of a new document.Source: §A.1.2 · p. 22
• More than 10 HTML-size reduction on average is achieved while preserving interesting content.The reduction follows the combined parsing, cleaning, unwrapping, removal, and node-modification steps.Source: §A.1.2 · p. 22

A.1.3 Extracting Multimodal Web Documents

• The pipeline transforms simplified HTML files into structured documents that interleave text and images.Texts and image links are extracted as an ordered sequence from the web page.Source: §A.1.3 · p. 22
• Texts and image links retain their DOM order, while HTML separations and line breaks preserve the original formatting and visual rendering.Source: §A.1.3 · p. 22
• Approximately 2 billion of 3.6 billion attempted images—55%—were downloaded with img2dataset across 20 virtual machines within a few days.The distributed setup enabled completion of the large-scale download operation within a few days.Source: §A.1.3 · p. 22

Filtering strategy

• The pipeline applies node-level image and paragraph-level text filters before deciding at document level which processed documents to retain or discard.Most text filters used at both stages were adapted from Laurençon et al. (2022).Source: §A.1.4 · pp. 22–23

Node-level image filtering

• Image filtering removes unsupported formats, extreme sizes or aspect ratios, and URLs containing logo, button, icon, plugin, or widget substrings.The criteria target small, oversized, disproportionate, generic, or likely irrelevant images, as in Zhu et al. (2023).Source: §A.1.4 · p. 23

Paragraph-level text filtering

• Paragraph filtering removes short, repetitive, special-character-heavy, stop-word-sparse, poorly punctuated, flagged-word-heavy, high-perplexity, generic-spam, and low-common-word text.KenLM models trained on Wikipedia and common words extracted from 10 million OSCAR documents support these filters; cutoff values are given in Table 3.Source: §A.1.4 · p. 23

Document-level filtering

• Document-level filtering removes documents without images or with more than 30 images, while applying the same text metrics with stricter cutoffs enabled by document-level precision.Documents with many images are more likely to contain unrelated images and spam.Source: §A.1.4 · p. 23

Filtering outcome

• 365 million web documents and 1.4 billion images remained after the filtering pipeline.Images may be duplicated across documents at this stage.Source: §A.1.4 · p. 23

A.1.5 Additional Filtering and Deduplication Steps

• Image deduplication retains occasional duplicates because differing surrounding contexts may benefit model training, while removing within-document duplicates and images appearing more than 10 times.Frequently repeated images are predominantly advertisements, browser icons, and similar elements.Source: §A.1.5 · p. 24
• Filtering removes NSFW documents, highly repeated images, duplicate documents, copied image sets, and repetitive same-domain paragraphs.NSFW filtering removes approximately 1% of documents, URL-based image filtering removes images appearing more than 10 times, and paragraph deduplication eliminates approximately 15% of web-document text.Source: §A.1.5 · pp. 24–25
• The final dataset contains 141 million documents and 353 million images, including 298 million unique images.Stricter filtering produces fewer multimodal documents without necessarily improving quality, so users seeking smaller subsets are advised to sample randomly.Source: §A.1.5 · p. 25

A.2.1 Examples of Multimodal Web Documents

• Example documents include ordinary multimodal webpages as well as an undesirable document with many images, little text, and unclear image relationships.The supplied examples are drawn from birding, legal, and entertainment webpages.Source: Figure 7 · p. 26, Figure 8 · p. 27, Figure 9 · p. 28

A.2.5 Topic Modeling with 200 Topics

• The 200-topic LDA analysis identifies diverse themes, including travel destinations (0.94%), personal experiences (2.07%), sports and games (1.29%), and statistics (0.99%).Other topics include healthcare, legal matters, international politics, scientific research, agriculture, and collectibles.Source: §A.2.5 · pp. 42–43, Table 6 · p. 44

A.3 Ethical discussion

• OBELICS incorporates ethical review and content filtering, but retains a small proportion of unsuitable material and inherits web-corpus risks including bias, demographic under-representation, and inspection difficulty.An interactive visualization supports browsing 11M documents, while red-teaming found that OBELICS-trained models associated men more often than women with terms such as “financial” and “software”.Source: §A.3 · p. 45

A.4.1 Architecture Details

• IDEFICS combines pretrained image and language backbones with Perceiver blocks and Transformer cross-attention inserted every 4 language-model layers.The pretrained backbones are frozen, and layer normalization on projected queries and keys improved training stability in early experiments.Source: §A.4.1 · p. 45

A.4.2 Training Details

• Training uses mixtures of web documents and image-text pairs, with IDEFICS receiving longer sequences and larger batches; IDEFICS and the 9B LAION+OBELICS model each required exactly three restarts after unrecoverable loss spikes.The mixtures use OBELICS and Wikipedia alongside LAION and PMD, with repeated higher-quality datasets and sampling probabilities controlling source frequency.Source: §A.4.2 · p. 46

A.4.3 Compute Details

• The 9B models use 32 or 64 80GB A100 GPUs for approximately 6 days, while IDEFICS and IDEFICS-9B use 512 and 128 GPUs for about 14 days each.The compute infrastructure is hosted on an AWS cluster in Oregon.Source: §A.4.3 · p. 46

A.4.4 Evaluation

• Evaluation matches Flamingo’s benchmark splits and uses in-context learning with prompt selection; classification ranks label probabilities, while captioning and VQA use open-ended greedy decoding.VisDial ranks candidates, and selected stop words truncate generated responses; the LAION-only 9B model keeps only the first answer word for some single-word VQA tasks.Source: §A.4.4 · pp. 46–48

A.4.5 Additional Experimental Results

• VQA performance declines after extended LAION-only training, while image-text pairs outperform multimodal documents on image captioning; OBELICS mixtures are compared in Figure 11.The authors hypothesize that image-text pairs support fast image-text associations but not the complex reasoning required for VQA, whereas matched captioning favors LAION-only training.Source: §A.4.5 · p. 49, Figure 11 · p. 50

A.5 License and Author Statement

• The dataset is released under CC-BY with disclosure requirements, while users bear responsibility for complying with source licenses, rights, laws, regulations, and ethical guidelines.The authors disclaim liability for copyright, privacy, sensitive-information, or other violations and state that the dataset will be maintained on Hugging Face Hub.Source: §A.5 · p. 51