Source-linked AI summary
LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs
Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, Aran Komatsuzaki
TL;DR
Multimodal models need large-scale image-text data, yet sufficiently large datasets were not publicly available. The paper releases LAION-400M with 400 million CLIP-filtered pairs, embeddings, and kNN indices, and demonstrates successful DALL-E training on a subset.
Problem
Sufficiently large publicly available image-text datasets for training multimodal language-vision models from scratch were unavailable.
Method
The authors construct and release LAION-400M from Common Crawl-derived image-text pairs, providing CLIP filtering, embeddings, kNN indices, and processing tools.
Results
Despite using approximately 7.2 million images for one epoch, DALL-E training showed fast convergence and generated samples of sufficient quality.
Takeaways & Limitations
The open dataset enables broader community access to large-scale language-vision model training and research.
Abstract
from arXiv · showhide
Multi-modal language-vision models trained on hundreds of millions of image-text pairs (e.g. CLIP, DALL-E) gained a recent surge, showing remarkable capability to perform zero- or few-shot learning and transfer even in absence of per-sample labels on target image data. Despite this trend, to date there has been no publicly available datasets of sufficient scale for training such models from scratch. To address this issue, in a community effort we build and release for public LAION-400M, a dataset with CLIP-filtered 400 million image-text pairs, their CLIP embeddings and kNN indices that allow efficient similarity search.
1 Introduction
Large-scale multimodal models can transfer to new datasets without per-sample target labels, but comparable publicly available image-text datasets have been lacking. LAION-400M addresses this gap by releasing 400 million CLIP-filtered image-text pairs with embeddings and search indices.
- Large-scale multimodal models demonstrate strong transfer to novel datasets without per-sample labels.
- Publicly released datasets had not matched the scale of the largest image-text datasets, which ranged from 400 million to around a billion pairs.
- LAION-400M releases 400 million CLIP-filtered image-text pairs, their CLIP embeddings, and kNN indices.
2 Dataset and Methods
LAION-400M combines Common Crawl processing, filtering, metadata, embeddings, search indices, and tools for acquiring and processing hundreds of millions of image-text pairs.
- The project releases 400 million image-URL/metadata pairs, CLIP embedding/text pairs, kNN indices, img2dataset, and a web demo.
- Acquisition: Acquisition separates distributed Common Crawl processing, which finds matching URLs and captions, from lighter single-node post-processing that produces the final dataset.
- Acquisition: Common Crawl WAT files are parsed for HTML IMG tags with alt text, and raw images are downloaded asynchronously.
- Filtering: Filtering removes short alt text, small images, duplicates, low image-text cosine similarity, and illegal contents detected with CLIP embeddings.
- img2dataset: img2dataset downloads, resizes, and stores images and captions, processing 100 million images in 20 hours on one specified node.
3 Analysis & Results
The dataset supports similarity search and image-size-based subset selection, while a DALL-E proof of concept shows successful training progress from only a small subset.
- Web demo and similarity search: The web demo searches images and texts from query images or text using CLIP embeddings and precomputed kNN indices.
- Image sizes: Image-size distributions enable subsets with high-resolution images and resolutions suited to particular training purposes.
- Training DALL-E model: Despite training for one epoch on approximately 7.2 million images, the DALL-E model showed fast convergence across varied categories.
- Training DALL-E model: Generated samples had sufficient quality to provide evidence of successful DALL-E training progress.
4 Conclusion
The authors release LAION-400M as an open alternative to proprietary large-scale datasets and demonstrate that a subset can train a DALL-E model with sufficient sample quality.
- LAION-400M closes the gap to proprietary large-scale datasets used to train models such as DALL-E and CLIP.
- A subset of LAION-400M trains a DALL-E model and produces samples of sufficient quality.
- The dataset opens large-scale language-vision model training and research to the broader community.