PaperlayerView exact revision on arXiv

Source-linked AI summary

LAION-5B: An open large-scale dataset for training next generation image-text models

Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, Jenia Jitsev

arXiv:2210.08402v1cs.CVcs.AIcs.LG

TL;DR

Large-scale image-text models depend on billions of training pairs, but comparable datasets were not publicly available. LAION-5B assembles and releases over 5.8 billion CLIP-filtered pairs, and subsets support CLIP models that match or approach the performance of models trained on closed data across diverse evaluations.

  • Problem

    Comparable large-scale image-text training datasets were not publicly available, limiting broader research on language-vision models.

  • Method

    The paper constructs LAION-5B from Common Crawl images and alt-text, filtering pairs with CLIP into English, multilingual, and language-neutral subsets.

  • Results

    Models trained on LAION-400M match or approach OpenAI CLIP performance across zero-shot classification, distribution shifts, VTAB, retrieval, and fine-tuning.

  • Takeaways & Limitations

    LAION-5B broadens access to large-scale, multilingual training and research on language-vision models previously dependent on proprietary datasets.

  • Takeaways & Limitations

    Training-set overlap with downstream test sets may inflate accuracy and overstate the true generalization of models trained on LAION-5B.

Abstract

from arXiv · show

Groundbreaking language-vision architectures like CLIP and DALL-E proved the utility of training on large amounts of noisy image-text data, without relying on expensive accurate labels used in standard vision unimodal supervised learning. The resulting models showed capabilities of strong text-guided image generation and transfer to downstream tasks, while performing remarkably at zero-shot classification with noteworthy out-of-distribution robustness. Since then, large-scale language-vision models like ALIGN, BASIC, GLIDE, Flamingo and Imagen made further improvements. Studying the training and capabilities of such models requires datasets containing billions of image-text pairs. Until now, no datasets of this size have been made openly available for the broader research community. To address this problem and democratize research on large-scale multi-modal models, we present LAION-5B - a dataset consisting of 5.85 billion CLIP-filtered image-text pairs, of which 2.32B contain English language. We show successful replication and fine-tuning of foundational models like CLIP, GLIDE and Stable Diffusion using the dataset, and discuss further experiments enabled with an openly available dataset of this scale. Additionally we provide several nearest neighbor indices, an improved web-interface for dataset exploration and subset generation, and detection scores for watermark, NSFW, and toxic content detection. Announcement page https://laion.ai/laion-5b-a-new-era-of-open-large-scale-multi-modal-datasets/

1 Introduction

The introduction presents large-scale image-text data as central to recent vision-language advances, while noting that comparable pre-training datasets remain unavailable publicly. LAION-5B addresses this gap with a public dataset exceeding 5.8 billion examples and validates its suitability for training large image-text models.

  • Motivation: CLIP improved ImageNet zero-shot top-1 accuracy from 11.5% to 76.2%.The introduction attributes this gain to contrastive objectives combined with large neural networks.
  • Motivation: Recent image-text advances depend on pre-training datasets containing hundreds of millions or billions of image-text pairs, including 400 million for CLIP and 6.6 billion for BASIC.The introduction identifies these datasets as a critical ingredient in the new generation of image-text models.
  • Problem: These large pre-training datasets and resulting models were generally unavailable to the wider research community, concentrating research in a small number of industrial labs.The introduction links this limited access to reduced transparency and impeded research progress.
  • Contribution: LAION-5B is introduced as a public image-text dataset containing over 5.8 billion examples, constructed from Common Crawl and filtered with an existing CLIP model.The described dataset includes 2.32 billion English image-text examples; the passage continues with additional dataset components beyond the supplied excerpt.
  • Validation: The authors train CLIP on a 400-million-example LAION-5B subset to compare its performance with OpenAI’s publicly released CLIP models.They motivate this validation because OpenAI’s models are the largest publicly released image-text models.
  • Limitations: The authors describe LAION-5B as an initial release rather than a finished data product and release the software stack used to assemble it.They state that widespread curation exceeds the scope of a single paper because of the immense size of current image-text pre-training datasets.

2 Related Work

Prior work established large-scale contrastive vision-language learning and text-guided image generation, while dataset efforts expanded from carefully annotated corpora to increasingly large web-harvested image-text collections. However, multilingual resources remained comparatively scarce before LAION-5B.

  • Vision-Language Models: CLIP embedded images and text in a shared representation space through contrastive learning, enabling strong zero-shot image classification and robustness across distribution shifts.ALIGN, BASIC, and LiT subsequently increased training scale, batch size, or used frozen image representations with contrastive fine-tuning; Flamingo introduced in-context learning for vision-language models.
  • Text-Guided Image Generation: DALL-E demonstrated diverse text-prompted image generation, while GLIDE, DALL-E 2, Imagen, Parti, and Stable Diffusion improved visual fidelity and prompt correspondence.
  • Image-Text Datasets: Human-annotated datasets such as MS-COCO and Visual Genome provided high-quality labels but were limited to 330K and 5M examples, respectively.YFCC-100M substantially increased scale with about 99 million images and one million videos, but retained only user-generated metadata without additional annotations.
  • Image-Text Datasets: Web-derived datasets progressively combined cleaning or relaxed filtering with larger scale, reaching 1.8 billion image-text pairs in ALT200M and ALIGN, while RedCaps used Reddit-provided captions.CC3M began from web images and alt-text and applied additional cleaning; CC12M later relaxed the filtering protocol to increase dataset size.
  • Multilingual Datasets: Non-English image-text datasets were uncommon, and the largest multilingual dataset before LAION-5B contained around 36 million Wikipedia Image Text samples.Researchers also translated English captioning datasets into Farsi, Korean, and Japanese; LAION-5B provided roughly two orders of magnitude more multilingual samples.

3 Collection Methodology

LAION-5B was assembled from Common Crawl by extracting alt-text image pairs, downloading images at scale, and applying staged content filtering. The pipeline used language detection and CLIP similarity thresholds, while tagging potentially harmful content rather than removing all offensive samples.

  • Web page filtering: The pipeline starts from Common Crawl snapshots, which contain about 300 TiB of data covering around 3 billion web pages, and targets images with associated HTML alt-text.Alt-text is extracted from IMG tags in Common Crawl WAT metadata files to form image-text pairs.
  • Pipeline overview: Three components structure dataset assembly: distributed web-page filtering, distributed image-text downloading, and content filtering.The acquisition pipeline tracks downloads and applies distributed inference before retaining qualifying pairs.
  • Web page filtering: Alt-text undergoes CLD3 language detection into English, another language, or no detected language.The no-language category includes language-agnostic short text such as product and place names.
  • Post-processing: Post-processing removes pairs with less than 5 characters of text, less than 5 KB of image data, or potentially malicious, large, or redundant images.CLIP cosine similarity is then computed using ViT-B/32 models, with multilingual CLIP applied to non-English languages.
  • Post-processing: 90% of the original 50 billion images were removed by CLIP thresholds, leaving just short of 6 billion examples.English pairs below 0.28 and other-language pairs below 0.26 cosine similarity were removed.
  • Content filtering and limitations: The dataset tags potentially offensive content instead of removing it wholesale, allowing users to choose whether to include such samples.Q16 and a specialized NSFW classifier identify inappropriate concepts, and the classifiers and related implementations are open-sourced.

4 Dataset Composition

LAION-5B is released in three language-based subsets totaling 5.85 billion image-text pairs, including English, multilingual, and undetected-language samples. The dataset also provides metadata for filtering and analysis, including similarity, NSFW, and watermark scores.

  • 2.32 billion image-text pairs form the English subset, called LAION-2B-en or LAION-2B when the language is clear from context.
  • 2.26 billion image-text pairs span over 100 other languages, led by Russian, French, German, Spanish, and Chinese.These languages account for 10.6%, 7.4%, 6.6%, 6.6%, and 6.3% of the multilingual subset, respectively.
  • 1.27 billion samples have no clearly detected language, with inspected examples often showing products or places and captions combining clear semantics with possible SEO or product-tag noise.
  • Metadata files in Apache Parquet include identifiers, image URLs, text, dimensions, image-text cosine similarity, and NSFW and watermark detector outputs.Each detector provides one score between 0 and 1.
  • 3% of images were detected as NSFW and can be filtered out using the NSFW tag.

5 Experiments Validating LAION-5B

Experiments validate LAION-5B’s utility for large-scale vision-language training through curated subsets, CLIP reproduction studies, scaling analyses, and text-to-image generation. The results include strong performance across classification, robustness, downstream-task, and generative-model evaluations.

  • Subdataset Generation: LAION-5B enabled curated subsets including LAION-High-Resolution, a 170M superresolution dataset, and LAION-Aesthetic, a 120M aesthetic-image dataset.Researchers also used LAION-5B and LAION-400M for facial representation learning and invasive species mitigation.
  • CLIP Reproduction and Improvements: The authors reproduced CLIP at multiple scales on LAION-400M using ViT-B/32, ViT-B/16, ViT-B/16+, and ViT-L/14.Training used 128–400 NVIDIA A100 GPUs on the JUWELS Booster supercomputer, following CLIP’s hyperparameter choices.
  • CLIP Reproduction and Improvements: LAION-trained models were compared with original CLIP on ImageNet, distribution-shift datasets, and VTAB+, which averages top-1 accuracy over 35 tasks.The evaluation used predefined class prompts, averaged text embeddings, and image-text cosine similarity for zero-shot classification.
  • Scaling Effects: Accuracy on VTAB+ improved with total compute for models trained on LAION-400M and LAION-2B-en.The authors present this relationship as evidence that zero-shot performance improves with scale, while noting that future work should examine possible saturation.
  • Scaling Effects: At 34B images seen, ViT-L/14 trained on LAION-2B-en reached 75.4%, compared with 73.9% for LAION-400M.These experiments used ViT-B/32 and ViT-L/14 across LAION-5B and LAION-400M subsets while varying training compute, supporting a significant dataset-scale effect given sufficient compute.
  • Image Generation: Fine-tuning OpenAI’s GLIDE on LAION-5B produced LAIONIDE, whose generated samples were compared with original GLIDE through an interactive web demo.The paper also directs readers to appendix experiments covering GLIDE and Stable Diffusion.

6 Technical Limitations

LAION-5B has technical limitations stemming from possible data overlap, noisy or uninformative alt-text, and biases or errors introduced by automated CLIP filtering. These limitations are presented as starting points for future dataset analysis and improvement, although overlap is not considered a serious threat to result validity.

  • LAION-5B’s scale makes it infeasible to investigate every dataset aspect in one publication, so these limitations are starting points for future analysis and improvement.
  • Data Overlap: Data overlap with downstream test sets could inflate accuracies and overstate the true generalization of LAION-5B-trained models.
  • Data Overlap: Nevertheless, the authors do not view test-set overlap as a serious validity threat, citing few substantial differences in related CLIP evaluations and likely non-overlap for ObjectNet.
  • Other text sources: Alt-text can be SEO spam, incoherent keywords, or corrupted text, making annotations less informative or useless for training.
  • Other text sources: Generated text formed by concatenating class names may provide a meaningful alternative when natural-language captions are corrupted.
  • Filtering with CLIP: CLIP filtering is scalable and reduces the need for human control, but it also introduces CLIP’s flaws and model biases.
  • Filtering with CLIP: Small-scale CLIP ViT-B/32 filtering may retain weakly connected pairs and remove high-quality pairs that stronger models could preserve.

7 Safety and Ethical Discussion

LAION-5B’s openness enables systematic study of model capabilities, dataset-induced biases, and safety issues that closed datasets make difficult or impossible. The authors frame it as a research artefact requiring careful, transparent investigation and further curation, with explicit attention to inappropriate content and privacy.

  • Motivation and risks: Closed large-scale datasets make systematic investigation of model training and behavior very hard or even impossible.Vision-language models may perform worse for certain user groups or reproduce discriminatory behavior.
  • Opportunities for safety research: LAION-5B enables studies of how uncurated datasets affect model biases and safety, supporting automated curation methods that alleviate these issues.The dataset is presented as a basis for investigating undesired safety outcomes under different usage conditions.
  • Mitigation tools: Improved inappropriate-content tagging and a watermark filter can improve the safety and quality of text-to-image models trained on LAION-5B.These measures were motivated by problems identified after LAION-400M’s release and investigations of unfiltered data.
  • Open development: The dataset is a starting point rather than a final endpoint, and the authors advocate broad research producing open and transparent datasets and training procedures.They reject a non-transparent, closed-door approach to dataset and model development.
  • Responsible use: The authors advocate academic use only, careful investigation and reporting of downstream biases and newly detected behavior, and attention to privacy issues from Common Crawl links.Users are encouraged to use the provided tools and contribute findings to the dataset repository; privacy measures are discussed in Appendix Sec. G.1.

8 Conclusion · Appendix (LAION-5B: An open large-scale dataset for training next generation image-text models)

LAION-5B expands the scale of openly available image-text datasets to over 5 billion pairs, enabling stronger zero-shot transfer and robustness. Experiments show that subsets support competitive CLIP training and quality GLIDE fine-tuning, broadening access to multilingual language-vision research.

  • 8 Conclusion: Over 5 billion image-text pairs make LAION-5B a larger updated open dataset for training and studying state-of-the-art language-vision models.The dataset further pushes the scale of open training data.
  • 8 Conclusion: This scale produces strong increases in zero-shot transfer and robustness.The passage attributes both improvements to the dataset’s scale.
  • 8 Conclusion: A subset of LAION-5B trains SOTA CLIP models across various scales.These models match the strong zero-shot and robustness performance of original models trained on closed curated data.
  • 8 Conclusion: These trained CLIP models match the original models’ strong zero-shot and robustness performance.The original models were trained on closed curated data.
  • 8 Conclusion: LAION-5B subsets can fine-tune generative models like GLIDE.The resulting GLIDE samples are described as good quality.
  • 8 Conclusion: The dataset enables multi-language large-scale training and research in language-vision models.The authors state that these opportunities were previously restricted to those having access to comparable resources.

A Datasheet for LAION-5B dataset … C.5 Specialized NSFW image content tagging

LAION-5B is an open, web-scale image-text dataset comprising 5.8 billion pairs with multilingual metadata, model-derived similarity and safety signals, and substantial practical and ethical limitations. Its preparation combines Common Crawl filtering, distributed embedding and indexing tools, and specialized NSFW tagging, while the datasheet restricts uncurated use to research.

  • A.2 Composition: 5.8 billion image-text pairs include URLs, captions, dimensions, language, CLIP/MCLIP similarity, watermark probability, and NSFW probability.The dataset contains 2.3 billion English, 2.2 billion multilingual, and 1.2 billion unknown-language samples.
  • A.3 Collection Process: LAION-5B is filtered from over 50 billion Common Crawl images, retaining 5.8 billion image URL–alt-text pairings without hard class labels.English samples require cosine similarity above 0.28, while multilingual and unknown-language samples require at least 0.26.
  • Q14 Is the dataset self-contained, or does it link to or otherwise rely on external: The dataset contains near duplicates, depends on unstable web links, and can be downloaded in 7 days using 10 nodes.The authors provide img2dataset for efficient URL-based downloading but cannot guarantee sample existence or archive the current state.
  • Q15 Does the dataset contain data that might be considered confidential (e.g., data: Sensitive, offensive, and potentially identifying content may occur because the dataset was scraped from the web, and safety tags are explicitly imperfect.People may be identifiable through face recognition or associated text, while the dataset provides no subpopulation markers.
  • Q42 Is there anything about the composition of the dataset or the way it was collected: The datasheet reports no direct notification or guaranteed consent for depicted individuals, and recommends responsible research use rather than production deployment.The dataset mirrors broader societal biases, including subpopulation depiction, violence, and nudity, and should not support decisions about people.
  • A.5 Uses: LAION-5B supports representation learning, image-to-text generation, captioning, safety research, and low-resource language research, and is distributed as open source through Hugging Face.LAION owns the metadata and release as CC-BY-4.0 but does not own the copyright of the images or text.
  • C.5 Specialized NSFW image content tagging: NSFW tagging used an open CLIP/L-14-based MLP trained on over 682K images across drawing, hentai, neutral, porn, and sexy classes.The pipeline first considered an EfficientNetV2 classifier, then selected the simpler MLP and avoided commercial solutions to keep implementations open source.

C.6 Further inappropriate content tagging · C.7 Watermark and safety inference

The paper documents potentially inappropriate concepts in LAION-5B and provides user-facing safety and watermark filtering. These tags are integrated into the exploration interface to support removal of unsafe or watermarked examples.

  • C.6 Further inappropriate content tagging: Q16 documentation identified a broad range of potentially inappropriate concepts in LAION-5B.Frequent concepts included human, people, man, woman, porn, bondage, kinky, and bdsm.
  • C.6 Further inappropriate content tagging: The identified concepts frequently concerned humans and pornographic content.Examples included the concepts human, people, man, woman, porn, bondage, kinky, and bdsm.
  • C.6 Further inappropriate content tagging: Further filtering depends highly on the use-case and users’ opinions.
  • C.6 Further inappropriate content tagging: The watermark test set used manually defined annotation criteria for watermark and non-watermark images.Figure 6 shows annotation examples for both categories.
  • C.7 Watermark and safety inference: The authors collected training and test sets to enable removal of unsafe and watermarked examples.The training set was augmented with KNN-retrieved examples, while the test set represented the dataset distribution and was manually annotated.
  • C.7 Watermark and safety inference: The safety and watermark tags gave users the ability to remove unsafe and watermarked examples.The filtering functionality was exposed through the dataset exploration interface.
  • C.7 Watermark and safety inference: Inference was performed using the embedding-reader module.
  • C.7 Watermark and safety inference: Safety tags were integrated into the UI and filtered out almost all unsafe results.This provided confidence that training a generative model on the data would not produce unexpectedly unsafe images.

D Dataset Samples and Statistics … E.2 Distributed Training and InfoNCE Loss

The paper characterizes LAION-5B through representative samples, caption and language statistics, then details CLIP reproduction and distributed InfoNCE training. The distributed formulation reduces memory complexity from quadratic to linear in the global batch size.

  • D Dataset Samples and Statistics: The language classifier shows low confidence for names, identifying numbers, and short-form text.Improving the language classifier is identified as an important future direction.
  • D Dataset Samples and Statistics: Random examples cover four SFW samples from each of the three randomly shuffled LAION-5B subsets.Each sample includes its image and associated caption.
  • D Dataset Samples and Statistics: 56% of the multilingual dataset is composed of the 10 most frequent languages.Caption lengths have similar, right-skewed frequency distributions across subsets.
  • E Further Experimental Details and Results on CLIP reproduction: CLIP reproduction experiments use the LAION 400M and 2B-en subsets and evaluate zero-shot classification with VTAB+ and retrieval.The paper documents experimental results for both evaluation settings.
  • E.1 Training Details: Models are trained with PyTorch distributed data parallelism on multiple NVIDIA A100 GPUs using AdamW, InfoNCE loss, warmup, cosine decay, and weight decay 0.2.AdamW uses β1 = 0.9 and β2 = 0.98 for all models.
  • E.2 Distributed Training and InfoNCE Loss: Distributed InfoNCE training gathers image and text embeddings across GPU workers so every worker can use other workers’ embeddings as negatives.This communication enables correct global loss and gradient computation for all positive and negative sample pairs.
  • E.2 Distributed Training and InfoNCE Loss: For global batch size N = 32768, naive InfoNCE requires an 8 GB float32 N × N matrix, whereas sharding reduces complexity from O(N^2) to O(nN).The reformulation reduces scaling overhead from gigabytes to megabytes while preserving correct global gradients.

E.3 Detailed Results & Further Analysis · F Overview of Experiments and Results on Generative Models

E.3 finds that larger LAION training scales generally improve transfer and retrieval, while specialized, adversarial, imbalanced, and structured tasks expose important weaknesses. F briefly summarizes generative-model experiments using GLIDE and Stable Diffusion trained on LAION-5B subsets.

  • E.3 Detailed Results & Further Analysis: LAION-2B-en generally matches or outperforms LAION-400M, except on some specialized-domain and structured datasets.The reported exceptions include Diabetic Retinopathy and PatchCamelyon.
  • E.3 Detailed Results & Further Analysis: 34B samples seen and larger datasets consistently improve ImageNet-1k zero-shot transfer when the training compute is sufficient.Experiments vary LAION-80M, LAION-400M, and LAION-2B across 3B, 13B, and 34B samples seen.
  • E.3 Detailed Results & Further Analysis: Few-shot linear-probe evaluations across seven transfer datasets show similar transfer to original OpenAI CLIP models and clear effects of model or data scale.The benchmarks include ImageNet, Food101, Cars, CIFAR-10 & 100, DTD, and SUN397.
  • E.3 Detailed Results & Further Analysis: 24.3% is the observed ViT-L/14 difference between CLIP WIT and LAION models on ImageNet-A, whose adversarial design limits generalizability.The authors do not explain the discrepancy and note that ImageNet-A was selected to be adversarial for an ImageNet-1k-pretrained ResNet-50.
  • E.3 Detailed Results & Further Analysis: Diabetic Retinopathy accuracy ranges from 3% to 73.3% for CLIP WIT models and from 7.4% to 24.2% for LAION models.The dataset is highly imbalanced, with the majority class comprising 74% of samples; prompt tuning may alleviate low performance.
  • E.3 Detailed Results & Further Analysis: VTAB structured tasks remain difficult because they involve counting, depth prediction, or position/angle prediction, making improvement an open research question.The tasks listed are CLEVR, DSPRITES, SmallNORB, DMLAB, and KITTI.
  • E.3 Detailed Results & Further Analysis: LAION models consistently improve over CLIP WIT on all MSCOCO 5K retrieval metrics, while Flickr30k results are similar or better except for ViT-B/16 image retrieval.The authors leave open whether the retrieval advantage is general or dataset-specific.
  • F Overview of Experiments and Results on Generative Models: Generative-model experiments train GLIDE and Stable Diffusion using subsets of LAION-5B.This section provides an overview of the experiments and results rather than specific findings in the supplied passage.

F.1 GLIDE … G.2 Potential Biases Induced by CLIP Filtering

The paper demonstrates how LAION-5B supports finetuning and training of large-scale generative models, while openly exposing privacy risks and biases from web-scale data collection and CLIP-based filtering. It also describes mitigations and auditing opportunities for these safety and ethics concerns.

  • F.1 GLIDE: OpenAI’s filtered GLIDE checkpoints cannot generate human imagery, so the authors finetune them on LAION-5B to restore this capability.They finetune the released GLIDE 64 pixel base checkpoint from OpenAI.
  • F.1 GLIDE: LAIONIDE-v2 trains on roughly 25 million shuffled LAION-2B pairs after filtering NSFW or likely-NSFW samples, non-English captions, extreme aspect ratios, small images, and some slurs.The slur search is explicitly described as far from comprehensive.
  • F.2 Stable Diffusion: Stable Diffusion is trained across LAION-5B subsets using staged resolutions and conditioning dropout, including 237,000 steps at 256x256 and 515,000 steps at 512x512.The schedule also includes 194,000 steps at 512x512 and 390,000 steps at 512x512 with 10% text-conditioning dropout.
  • G.1 Privacy: Because LAION-5B contains links to publicly available Internet content, it may include personal information such as faces, medical images, and other personal-related material.LAION’s CLIP retrieval tools can help users locate whether crawled links point to their personal data.
  • G Further Discussion on Safety and Ethics: LAION tools can help users discover problematic personal or copyrighted content and initiate removal through host providers or LAION’s contact form.The paper presents this access as a way to increase awareness of risks in large-scale crawled datasets and their potential third-party exploitation.
  • G.1 Privacy: Privacy concerns may be mitigated by obfuscating or generating personal facial attributes, enabling open-data training sets without private facial data while preserving face representations.The paper identifies generation-based anonymization as especially applicable to LAION-5B.
  • G.2 Potential Biases Induced by CLIP Filtering: CLIP filtering is difficult to audit because its underlying training dataset is unknown, while cosine similarity can misguide selection and assign higher similarity to malignant than benign captions.The paper frames LAION-5B’s open availability as a starting point for auditing contrastive image-text models.
Loading 2210.08402v1…