Source-linked AI summary
On the De-duplication of LAION-2B
Ryan Webster, Julien Rabin, Loic Simon, Frederic Jurie
TL;DR
The paper addresses de-duplication at LAION-2B scale and the potential for extreme compression to make dataset exploration more accessible. It presents a compact CLIP-feature approach and reports roughly 700 million duplicated images with 91% precision.
Problem
Method
Results
Roughly 700 million LAION-2B images were identified as duplicated, with 91% precision.
Source: §1 · p. 1Takeaways & Limitations
The released de-duplicated dataset, representative sets, histograms, and SNIP indices are intended to support dataset usability and transparency.
Source: §7 · p. 6Compressing image and text modalities independently risks losing their alignment, and tested hybrid encoders performed worse than the baseline or subsequent methods.
Source: §3 · p. 2
Abstract
from arXiv · showhide
Generative models, such as DALL-E, Midjourney, and Stable Diffusion, have societal implications that extend beyond the field of computer science. These models require large image databases like LAION-2B, which contain two billion images. At this scale, manual inspection is difficult and automated analysis is challenging. In addition, recent studies show that duplicated images pose copyright problems for models trained on LAION2B, which hinders its usability. This paper proposes an algorithmic chain that runs with modest compute, that compresses CLIP features to enable efficient duplicate detection, even for vast image volumes. Our approach demonstrates that roughly 700 million images, or about 30\%, of LAION-2B's images are likely duplicated. Our method also provides the histograms of duplication on this dataset, which we use to reveal more examples of verbatim copies by Stable Diffusion and further justify the approach. The current version of the de-duplicated set will be distributed online.
1. Introduction
Public datasets such as LAION-2B bring billions of CLIP-refined image-caption pairs to generative-model development. This work introduces feature compression and a de-duplication pipeline, reporting roughly 700 million duplicated images and using duplication patterns to find further Stable Diffusion copies.
1. Introduction
- LAION-5B contains billions of CLIP-refined image-caption pairs, and LAION-2B-en is a useful subset for users working mainly with English captions.Source: §1 · p. 1
- The paper studies multimodal retrieval on LAION-2B and makes the SNIP code available online.Source: §1 · pp. 1–2
- SNIP’s contrastive feature-compression loss better retains feature semantics for multimodal tasks while maintaining competitive retrieval performance.SNIP stands for Subset Nearest Neighbor CLIP compression.Source: §1 · p. 1
- Roughly 700 million LAION-2B images are duplicated, with 91% precision in the reported de-duplication.The authors also provide duplication histograms.Source: §1 · p. 1
- Synthesizing only the most duplicated images revealed further Stable Diffusion verbatim copies with significantly fewer resources than previous demonstrations.Source: §1 · p. 2
2. Related work
Prior work established CLIP’s multimodal capabilities and documented copying and overfitting concerns in generative models. Existing de-duplication approaches and available indices pose challenges for compact, rapid searches over LAION-2B, motivating exploration of extreme compression.
2. Related work
- CLIP uses contrastive loss to align image and text representations, supporting zero-shot and transfer tasks and applications such as text-to-image conditioning.OpenCLIP reproduced the original CLIP results, and released models surpassed its zero-shot ImageNet performance.Source: §2 · p. 2
- Carlini et al. [5] showed Stable Diffusion can reproduce training images exactly, and prior work linked such copying to duplicated training images [19].The paper also identifies highly duplicated images that were not overfit or were only loosely overfit.Source: §2 · p. 2
- Prior de-duplication methods and learned descriptors were not suited to compact, efficient cleaning of LAION-2B at massive scale.The available LAION indices could take 50–100 ms per query when memory mapped, too slow for billions-scale de-duplication.Source: §2 · p. 2
- The authors frame extreme compression as a potential way to make dataset exploration more accessible to the community.Source: §2 · p. 2
3. CLIP Feature Compression
The section compares reconstruction-based and contrastive approaches to CLIP feature compression. Its proposed neighbor-aware loss preserves local feature distances for retrieval, while the compression pipeline also uses approximate indexing.
MSE baseline
- The MSE autoencoding baseline compresses CLIP features through an image encoder and decoder, with latent output dimension controlling the compression rate.Source: §3 · p. 2
Contrastive Compression
- Independent compression of image and text descriptors risks losing their alignment, and tested hybrid encoders performed worse than the baseline or subsequent methods.Source: §3 · p. 2
- The contrastive approach applies CLIP loss to latent image and text features; this latent loss performed better overall than alternatives and was used in all experiments.Source: §3 · p. 2
- SNIP adds an image-neighbor contrastive term to CLIP compression, finding neighbors within sliding 10M-sample chunks to preserve distance properties for retrieval.Neighbor search within each chunk is exhaustive over pairwise L2 distances.Source: §3 · pp. 2–3
Compression results
- At higher compression ratios, MSE-based PCA and MSE compression lose ImageNet-class semantics relative to SNIP-based losses.Source: Figure 1 · p. 3
Approximate Search
- IVF searches exhaustively within the τ closest centroid lists, while Product Quantization splits vectors into M sub-vectors; larger M increases compression.The sub-vectors are quantized with k-means.Source: §3 · p. 3
4. Image Similarity Search
The section compares compressed descriptors and quantized indices for image and multimodal retrieval. SNIP combines image-search performance similar to MSE with stronger multimodal performance than MSE-based descriptors.
4. Image Similarity Search
- The retrieval benchmarks use exhaustive search on raw features as ground truth and compare it with approximate k-nearest-neighbor results from the index.A synthetic ground truth is generated from a small set of brute-force nearest neighbors.Source: §4 · p. 3
- Multimodal evaluation reconstructs database vectors from the quantized index before searching, and compares the results with AutoFaiss.The comparison uses AutoFaiss in a more extreme compression regime.Source: §4 · p. 4
- For zero-shot ImageNet, LMSE indices are less efficient than LSNIP, and SNIP also outperforms AutoFaiss on text-to-image search at the tested compression ratios.AutoFaiss performs worst at these compression ratios; the authors report a sharp performance drop above a certain ratio across tasks.Source: §4 · p. 4
- On image similarity search, MSE and SNIP perform similarly, while CLIP-only loss networks perform worse; SNIP performs better on multimodal tasks.The authors characterize SNIP as a best-of-both-worlds descriptor; Figure 2 shows image-feature-only retrieval fronts for L400M on CPU and GPU and L2B on CPU.Source: §4 · p. 4, Figure 2 · p. 4
5. De-duplicating LAION 2B
The authors apply compact-index pipelines to find many LAION-2B duplicates quickly, aiming for decent rather than perfect precision. They use an adaptive threshold and report that deduplication ran on a 32 GB RAM machine in several days.
5. De-duplicating LAION 2B
- Duplicates among retrieved nearest neighbors are identified with an adaptive threshold based on the query’s distance to its quantized version, rather than an absolute dADC cutoff.The method accounts for duplicates that may have non-low absolute dADC but share the same dADC; the threshold is applied only to retrieved k-NNs.Source: §5 · pp. 4–5
- The LAION-2B deduplication ran on a 32 GB RAM machine and took several days, with precision estimated from 100,000 labeled duplicated images and raw-CLIP ground truth.The raw-feature threshold was chosen conservatively to label near duplicates after inspection.Source: §5 · p. 5
- Figure 3 pairs duplicated database images with Stable Diffusion outputs; its top and bottom examples were duplicated more than 5,000 and 2,000 times, respectively.The figure presents examples of verbatim copies found by synthesizing the most-copied L2B images.Source: Figure 3 · p. 5
- The dataset-wide algorithm can also produce representative sets that group duplicated images together.The authors loop over the entire dataset using the algorithm with any index.Source: §5 · p. 5
6. Investigating Verbatim Copies in Stable Diffusion
The authors probe Stable Diffusion for verbatim training-image copies by prompting it with images among LAION-2B’s most duplicated. They report finding numerous copies after manually reviewing generated samples, using a small targeted set rather than synthesizing at the scale of prior work.
6. Investigating Verbatim Copies in Stable Diffusion
- For 10,000 samples, raw ViT-L-14 CLIP features marked 9,000 as near duplicates; the resulting label imbalance made precision high and suggests the method will miss some duplicates overall.The authors chose an adaptive threshold of 1%.Source: Figure 4 · p. 5
- Table 1’s precision values are not directly comparable across its ViT-L-14 four-GPU shards and unsharded ViT-H-14 CPU setup because different raw-CLIP thresholds were used.The caption says image-space comparisons will be provided in future versions.Source: Table 1 · p. 5
- The probe sampled 100 images from the most-duplicated long tail of L2B, unlike prior work [5], which synthesized nearly 175 million images and yielded 94 images.The authors generated nine random samples per chosen image prompt.Source: §6 · pp. 5–6
- Reviewing several hundred generated samples from prompts for the selected images revealed numerous verbatim copies of training data.The authors also found highly duplicated images that were not verbatim copies, and near-identical cases between those categories.Source: §6 · p. 6
7. Conclusions
The conclusion presents a modest-resource, decent-precision pipeline for de-duplicating LAION-2B-en, with downloadable deduplicated data, representative sets, histograms, and SNIP indices intended to support dataset exploration and transparency.
7. Conclusions
- The pipeline avoids pre-training or recomputing billions of CLIP features and is presented as an accessible tool for dataset transparency amid copyright concerns about duplicates.The authors say code to download the deduplicated dataset, representative sets, duplication histograms, and SNIP indices will be released.Source: §7 · p. 6