Source-linked AI summary
Multimodal C4: An Open, Billion-scale Corpus of Images Interleaved with Text
Wanrong Zhu, Jack Hessel, Anas Awadalla, Samir Yitzhak Gadre, Jesse Dodge, Alex Fang, Youngjae Yu, Ludwig Schmidt, William Yang Wang, Yejin Choi
TL;DR
Large-scale publicly available corpora of interleaved images and text were missing despite their importance for multimodal in-context learning. The paper introduces Multimodal C4, built from C4, and reports that its interleaved sequences improve multimodal in-context learning.
Problem
Large-scale publicly available corpora of interleaved images and text were unavailable, although prior experiments suggest they support performant multimodal in-context learning.
Method
The paper constructs Multimodal C4 from public webpages in the cleaned English C4 corpus, retrieving downloadable images and retaining documents with valid images.
Results
Models trained on Multimodal C4 image/text sequences perform multimodal in-context learning more effectively than models trained on single image/caption pairs.
Takeaways & Limitations
Multimodal C4 provides a public corpus for studying and developing multimodal in-context learning with interleaved image/text sequences.
Takeaways & Limitations
Models trained on Multimodal C4 inherit risks from the underlying C4 corpus, despite existing auditing efforts on its text-only version.
Abstract
from arXiv · showhide
In-context vision and language models like Flamingo support arbitrarily interleaved sequences of images and text as input. This format not only enables few-shot learning via interleaving independent supervised (image, text) examples, but also, more complex prompts involving interaction between images, e.g., "What do image A and image B have in common?" To support this interface, pretraining occurs over web corpora that similarly contain interleaved images+text. To date, however, large-scale data of this form have not been publicly available. We release Multimodal C4, an augmentation of the popular text-only C4 corpus with images interleaved. We use a linear assignment algorithm to place images into longer bodies of text using CLIP features, a process that we show outperforms alternatives. Multimodal C4 spans everyday topics like cooking, travel, technology, etc. A manual inspection of a random sample of documents shows that a vast majority (88%) of images are topically relevant, and that linear assignment frequently selects individual sentences specifically well-aligned with each image (80%). After filtering NSFW images, ads, etc., the resulting corpus consists of 101.2M documents with 571M images interleaved in 43B English tokens.
1 Introduction
Multimodal C4 addresses the lack of a public, large-scale corpus of interleaved images and text by extending C4 with aligned image placements. The paper reports broad topical coverage, effective filtering and alignment, and promising few-shot multimodal learning results.
- Motivation: Large-scale interleaved image-text pretraining data had not previously been made publicly available.Such sequences support multimodal in-context learning, including few-shot examples and prompts involving multiple images.
- Contribution: Multimodal C4 is a public, billion-scale dataset of interleaved image/text sequences constructed from the cleaned English C4 corpus.The corpus extends a widely used text-only web dataset with multimodal content.
- Method: Images are assigned to sentences by solving a bipartite linear assignment problem, with CLIP ViT-L/14 estimating image-sentence compatibility.Each sentence receives at most one image, and the assignment process is applied across more than 100M documents.
- Corpus properties: The corpus covers everyday topics such as cooking and travel, while filtering removes NSFW content and advertisements with high accuracy.The authors also report that the resulting images are relevant to their associated documents and often align with individual sentences.
- Applications: Initial ablations show that training on mmc4 sequences enables few-shot, in-context adaptation to image-captioning datasets.This application motivates interleaved rather than captions-only multimodal pretraining.
2 Related Dataset Work
Prior public multimodal pretraining datasets largely pair images with literal descriptions, whereas web content also expresses broader image-text relationships. Multimodal C4 is positioned as a public corpus for capturing these interleaved relationships while supporting more transparent examination of the data.
- Existing datasets: Most public million- or billion-scale multimodal pretraining datasets pair images with literal descriptions.Examples include LAION-2B, CC-12M and YFCC100M.
- Dataset gap: Web images can relate to text in ways beyond literal description, including placement within longer bodies of text.Existing datasets with multiple images per snippet or longer text do not fully provide the targeted interleaved format.
- Transparency: The public release of mmc4 is intended to facilitate more transparent and critical examination of interleaved corpora than privately held training sets.The authors note that models trained on mmc4 inherit risks from the corpus.
3 Data Curation Process
Multimodal C4 extends the cleaned English C4 corpus with downloadable, filtered images and interleaves them with sentences through image-text alignment. The curation pipeline removes duplicates, unsuitable resolutions and aspect ratios, NSFW content, and offers privacy- and precision-oriented subsets.
- Initial data collection: 365M documents and 156B tokens in C4 provide the web-scale text foundation for Multimodal C4.C4 was selected because it is widely adopted, web-based, and already underwent language identification, deduplication, and placeholder-text filtering.
- Gathering images: 115M documents and 1.37B images remained after retrieving webpage images, restricting formats and URL tokens, resizing images, and removing documents without valid downloads.Images were limited to PNG/JPEG/JPG files and resized to a maximum dimension of 800px.
- De-duplication+small resolution: Image deduplication removes within-document duplicates, highly duplicated images, images smaller than 150px, and extreme aspect ratios associated with banner-like ads.A phash-based detector identifies visually similar images; a threshold of 5 removes roughly 10M additional images.
- Discarding NSFW images: 97.4% accuracy was achieved by the NSFW classifier on its test set, while a manual sample found zero NSFW images among 3.7K images marked safe.Images with predicted NSFW probability over 0.1 were discarded, removing approximately 10% of remaining images.
- Aligning images and sentences: Linear assignment places images into sentences without relying on raw HTML positions, while zero-shot CLIP ViT-L/14 produces better within-document alignment than prior fine-tuned methods.The assignment treats each document as a bipartite problem, with at most one image assigned to each sentence.
4 Exploring mmc4
Exploration of mmc4 shows broad topical coverage, highly clustered image sources, and generally strong image-document and image-sentence relevance. The linear assignment method spreads images across more sentences while only slightly reducing average CLIP similarity.
- Sources of documents & images: 101.2M documents span 6.0M domains, while 571.4M images span 4.9M domains and are more concentrated among frequent hosts.The top 10% of domains account for 77% of documents but 89% of images.
- Image-text similarity: Linear assignment raises the mean percentage of sentences with an associated image from 22% to 34%, while average CLIP similarity decreases from 24.5 to 24.0.Compared with assigning each image to its maximally similar sentence, linear assignment distributes images more evenly throughout documents.
- Topic-based assessment: A 22K-document topic analysis finds clusters covering cooking, communities, travel, music, and art, with associated image clusters generally matching qualitative expectations.The analysis uses LDA with 30 topics and examines images most associated with each topic.
- Manual verification of image relevance+properties: Manual assessment finds 87.7% of examined images topically related to their documents and 80.4% well-aligned with their assigned sentences.The evaluation sampled 200 documents containing 836 images.
5 OpenFlamingo: An Early Application of mmc4
OpenFlamingo experiments test whether training on mmc4’s interleaved image-text sequences improves multimodal in-context captioning. The results favor interleaved-sequence training over caption-only training, while highlighting sensitivity to sequence flattening choices.
- Caveats: Future work should evaluate how alternative flattening schemes affect downstream performance because the reported method is only one possible approach.The experiments also used a preliminary v1 of mmc4-core.
- OpenFlamingo experiments: Five image sequences during training can suffice for OpenFlamingo models to generalize to 32-shot inference.This finding is reported as similar to prior Flamingo experiments.
- OpenFlamingo experiments: A model trained on 5M captions and 2.5M mmc4 sequences exceeds a zero-shot model trained on 15M LAION-2B captions.The result provides evidence that interleaved training supports adaptation to MSCOCO-style captions.
6 Conclusion
The paper releases mmc4, a large corpus of interleaved images and text, and reports initial evidence that it improves multimodal in-context learning. It identifies broader evaluation, scaling, and instruction-tuning questions for future work.
- Conclusion: mmc4 contains 100M+ documents, 571M images, and 43B English tokens derived from c4.The corpus is publicly introduced as an interleaved image-text resource.
- Conclusion: Models trained on mmc4 image-text sequences perform multimodal in-context learning more effectively than models trained on single image-caption pairs.This is reported as an initial experimental result.
- Future work: Future evaluations should test whether models reason flexibly across images and text or mainly reproduce interleaved independent examples.The paper presents this as an open empirical question.
- Future work: Future work should examine whether large-scale interleaved corpora bottleneck in-context vision-language learning and whether single-modal pretraining can remove that bottleneck.The paper frames both possibilities as unresolved.
- Future work: Instruction tuning is proposed as a complementary direction for multimodal in-context learning.The paper distinguishes it from interleaving independent supervised image-text examples.
B Full Set of LDA Topics
This appendix reports the complete topic inventory produced by the paper’s 30-topic LDA model on mmc4 documents.
- Full Set of LDA Topics: Table 4 contains the full set of topics for the k=30 LDA model introduced in Section 4.The topic model was trained on a random sample of mmc4 documents, with topic frequencies averaged over corpus documents.
C Most Frequent Top-Level Domains
The paper provides top-level domain frequency statistics for mmc4 and mmc4-core, alongside c4-en reference statistics.
- Most Frequent Top-Level Domains: Tables 5 and 6 list the 50 most frequent top-level domains for documents and images in mmc4 and mmc4-core.The domain statistics are discussed in Section 4.
- Most Frequent Top-Level Domains: Figure 8 ranks c4-en’s top 50 document domains by frequency for reference.The ranking follows the same ordering used for the results in Figure 6.
D.1 Images w/ Watermarks/Ads/Logos
Figure 9 presents manually labeled images involving watermarks, logos, and advertisements, while noting that logo- and advertisement-associated images can be difficult to distinguish.
- Figure 9a shows sample images containing watermarks in various forms.
- Figure 9b and Figure 9c show images associated with logos and advertisements, respectively.
- The dissimilarity between logo-associated and advertisement-related images is relatively modest.Advertisement-related images may contain promotional language or persuasive expressions, but may also feature only logos.
D.2 Interleaved Document
The appendix documents mmc4 through domain-frequency tables and examples of documents interleaving sentences with assigned images and CLIP similarity scores.
- Tables 7 and 8 display interleaved document examples containing sentences, accompanying images, and CLIP ViT/L-14 image-text similarity scores.
- Tables 5 and 6 list the top-50 top-level domains for documents and images in mmc4 and mmc4-core.
- Figure 9 provides manually labeled examples of images with watermarks and images related to logos or advertisements.
- Table 7 includes two logo-related images with “NELO” that are relevant to the document content and therefore excluded from the advertisement category.