Source-linked AI summary
SemDeDup: Data-efficient learning at web-scale through semantic deduplication
Amro Abbas, Kushal Tirumala, Dániel Simig, Surya Ganguli, Ari S. Morcos
TL;DR
Web-scale datasets contain semantic redundancy that exact-duplicate filters miss, making data-efficient training difficult. SemDeDup uses pretrained embeddings and tractable clustering to remove semantic duplicates, reducing LAION data by 50% with minimal performance loss while improving speed and out-of-distribution performance, and providing gains on C4.
Problem
Web-scale datasets are largely uncurated beyond exact-duplicate filtering, leaving semantic redundancy that can make training inefficient.
Method
SemDeDup uses pretrained foundation-model embeddings and clustering to identify and remove semantically similar but nonidentical examples efficiently.
Results
SemDeDup removes 50% of LAION data with minimal performance loss and 2x faster learning, while improving out-of-distribution performance and achieving 15% efficiency gains on C4.
Takeaways & Limitations
Removing semantic duplicates can improve learning speed and data efficiency across largely uncurated LAION and partially curated C4 datasets.
Takeaways & Limitations
SemDeDup does not capture many forms of semantic redundancy or address bad and misleading data, leaving room for further dataset reductions.
Abstract
from arXiv · showhide
Progress in machine learning has been driven in large part by massive increases in data. However, large web-scale datasets such as LAION are largely uncurated beyond searches for exact duplicates, potentially leaving much redundancy. Here, we introduce SemDeDup, a method which leverages embeddings from pre-trained models to identify and remove semantic duplicates: data pairs which are semantically similar, but not exactly identical. Removing semantic duplicates preserves performance and speeds up learning. Analyzing a subset of LAION, we show that SemDeDup can remove 50% of the data with minimal performance loss, effectively halving training time. Moreover, performance increases out of distribution. Also, analyzing language models trained on C4, a partially curated dataset, we show that SemDeDup improves over prior approaches while providing efficiency gains. SemDeDup provides an example of how simple ways of leveraging quality embeddings can be used to make models learn faster with less data.
1 Introduction
Web-scale self-supervised datasets contain redundancy, while scaling data increasingly yields diminishing returns. SemDeDup uses pretrained-model embeddings to remove semantic duplicates, reducing LAION data by 50% with minimal performance loss and faster learning.
- Motivation: Large web datasets such as LAION are generally unfiltered and require extremely long training durations.LAION contains 5 billion image/text pairs, and multimodal models train for many epochs on such datasets.
- Motivation: Diminishing marginal returns make ever-increasing data requirements unsustainable for improving test performance.Test error generally follows a power law with data quantity, while many models continue improving after tens of dataset passes.
- Problem: The approach targets semantic duplicates, which preserve largely identical information despite differing in pixels, tokens, margins, crops, or wording.Semantically redundant data are a related category involving distinct underlying objects with overlapping information.
- Problem: 50% of LAION examples contain at least one semantic duplicate detectable by SemDeDup.These examples are semantically similar but difficult to identify using simple input-space deduplication.
- Results: 50% smaller LAION training sets achieve nearly the same performance 2x faster, with improved out-of-distribution performance.The semantically nonredundant subset preserves performance while reducing training time by half.
- Results: On C4, SemDeDup beats prior state-of-the-art deduplication while providing 15% efficiency gains and sometimes improving performance.C4 is a partially curated text corpus, so its gains differ from those observed on LAION.
2 Related Work
Prior deduplication work largely removes exact or approximate textual duplicates, while data-efficiency methods include coresets and pruning criteria. SemDeDup instead offers a simple, tractable approach for semantic deduplication at uncurated web scale.
- Deduplication: Prior language and vision deduplication methods have focused mainly on exact duplicates and approximate n-gram overlap.C4 was deduplicated using repeated three-sentence spans and later MinHash-based document overlap.
- Coresets: Coresets seek small training subsets retaining full-data test accuracy, but many algorithms are computationally prohibitive at web scale.This motivates methods balancing data reduction with computational tractability.
- SemDeDup: SemDeDup develops an exceedingly simple and tractable algorithm achieving computational and data efficiency at scale.Its contribution differs from traditional coreset methods primarily in scalability to web-scale data.
- Positioning: Earlier embedding-based pruning examined relatively small, curated datasets, whereas SemDeDup studies highly uncurated web-scale data.The paper identifies semantic deduplication as a potentially important initial step in self-supervised data pruning.
3 SemDeDup
SemDeDup identifies semantic duplicates by comparing pretrained-model embeddings rather than raw inputs. Clustering makes this comparison tractable for web-scale datasets by restricting similarity computations within clusters.
- Embedding space: SemDeDup embeds each data point with a large pretrained foundation model to obtain a semantically meaningful distance metric.The method uses CLIP for images and OPT for natural-language data.
- Embedding space: Semantic duplicates can remain far apart in pixel or token space, so input-space distance is insufficient for finding them.Embedding-space comparison is designed to capture semantic similarity that exact or surface-level matching misses.
- Clustering: Naive deduplication requires O(n^2) comparisons, making all-pairs search impractical for web-scale datasets.LAION-440M would require approximately 1.9x10^17 similarity computations.
- Clustering: K-means clustering reduces SemDeDup’s complexity from O(n^2) to O(n^2/k) by comparing points within clusters.For LAION-440M, comparisons fall to approximately 4.6x10^12, a 5-order-of-magnitude improvement.
4 SemDeDup on LAION
SemDeDup reveals substantial semantic redundancy in LAION-440M and removes it using embedding-based deduplication, preserving or improving performance while accelerating training. The method also improves out-of-distribution results on multiple tasks.
- Semantic redundancy: 50% of LAION-440M images have at least one semantic duplicate at ϵ = 0.03.At ϵ = 0.00095, 30% of images already have a semantic duplicate.
- Semantic redundancy: Semantic duplicates include distorted versions of the same image, such as different crops, margins, aspect ratios, or color filters.At higher ϵ, SemDeDup also identifies semantically redundant images from different sources depicting similar concepts.
- In-distribution performance: 37% of LAION-440M can be removed without performance loss, while removing 50% causes less than 0.5% performance loss on ImageNet-1k.Random pruning produces much larger performance drops.
- In-distribution performance: Average zero-shot performance improves across 24 tasks after semantic deduplication, including when substantially less data is retained.The figure reports improvement down to 63% of the pre-training data.
- Out-of-distribution performance: Removing 37% of the data improves average performance across six out-of-distribution datasets and matches baseline performance when 50% is removed.SemDeDup also outperforms random pruning on individual out-of-distribution robustness tasks.
- Training efficiency: Deduplicated models reach convergence in substantially fewer training iterations.This reduction in training points enables faster training while retaining near-baseline performance.
5 SemDeDup on Natural Language
SemDeDup is applied to C4 by clustering document embeddings from a pre-trained OPT model and removing semantic duplicates. It outperforms random pruning across language-model validation measures and can match NearDup with greater data reduction.
- Method: Documents are represented by last-token embeddings from a pre-trained 125M OPT model and clustered with K = 11000.SemDeDup is then applied to the clustered embeddings, with comparisons against random pruning and NearDup.
- Results: SemDeDup significantly outperforms random pruning on prompts_with_answers perplexity and average opt_valid performance.The evaluation uses two independent validation sets.
- Results: SemDeDup beats random pruning on every opt_valid validation set.This result is reported for the individual validation-set breakdown.
- Efficiency: 10–15% less compute is sufficient to match the full-dataset single-epoch performance when smaller SemDeDup-pruned datasets are trained for multiple epochs.The efficiency gain is larger at higher pruning percentages.
- Comparison with baselines: SemDeDup outperforms random pruning on the C4 validation set, while its performance is comparable to NearDup under 4% pruning.NearDup and SemDeDup are comparable in this setting because only a small fraction of the dataset is removed.
- What is pruned: At low ϵ, SemDeDup identifies templated text with only a few words changed, which evades exact-string deduplication.These documents contain highly redundant information despite differing in surface form.
6 Analysis of hyperparameter choices
The analysis finds SemDeDup’s results are robust to several design choices, including cluster count, embedding model, and duplicate-retention strategy. Continuing training on pruned data can improve performance while reducing iterations and compute cost.
- Compute efficiency from extended training: Training on the 80% pruned dataset reaches baseline perplexity on prompts_with_answer in 95.0% of baseline training, saving 5.0% compute.The figure reports means and standard deviations across three random training seeds.
- Number of k-means clusters for SemDeDup: The exact choice of k has a very small impact on ImageNet zeroshot accuracy when keeping 40% of LAION440M.The tested values were k = 70,000, 50,000, and 10,000.
- Foundation model for embeddings: Using embeddings from a foundation model pretrained on a different private dataset has no impact on performance when training OpenCLIP on 40% of LAION440M.
- Choosing which semantic duplicates to keep: The strategy for choosing which duplicate to retain has negligible effect on ImageNet zero-shot accuracy.The compared strategies retain examples with low centroid similarity, random examples, or high centroid similarity.
- Training on deduplicated data for more iterations: Training on 50% of LAION440M until 75% of baseline iterations outperforms the full-data baseline on ImageNet, 24-dataset average accuracy, and six out-of-distribution datasets.
- Choosing the deduplication threshold ϵ: The deduplication threshold ϵ is manually tuned on 10% of clusters, whose dataset-size estimates provide a good approximation of the final size.The relationship between ϵ and deduplicated dataset size is semi-linear for LAION and C4.
7 Compute cost of running SemDeDup
Running SemDeDup adds little computational overhead relative to training and can produce substantial overall savings when datasets are pruned. The cost can also be amortized when the deduplicated dataset supports many downstream training runs.
- Deduplication overhead: SemDeDup’s deduplication overhead does not exceed 1% of LAION440M training cost in GPU hours.For example, training on 50% of the data saves 50% of training cost while deduplication requires only 1% of that cost.
- Amortization: The deduplication cost can be amortized by semantically deduplicating a web-scale dataset once and distributing the smaller foundation dataset to many downstream users.
8 Discussion
SemDeDup removes semantic duplicates using pretrained embeddings and improves learning efficiency and out-of-distribution performance, but it does not solve all data-pruning problems. Its scope is limited by unaddressed redundancy, misleading data, domain-specific embedding availability, ignored captions in LAION, and uncertain scaling behavior on C4.
- Discussion: SemDeDup improves learning speed and out-of-distribution performance while providing efficiency gains of up to 50% on LAION and 15% on C4.
- Limitations: The method captures only some semantic redundancy and does not address bad or misleading data, leaving room for further dataset reductions.
- Limitations: SemDeDup requires a pretrained embedding model relevant to the domain, which may be unavailable for entirely novel domains.The authors expect this limitation to affect only a small fraction of practical use cases.
- Limitations: LAION deduplication used image data but ignored captions, so caption information may reveal additional semantic duplicates.
- Limitations: C4 shows more modest gains because it is partially curated and contains fewer duplicates, while the reported models were small relative to the best models.The authors note that results may change with scale.
- Open questions: The optimal data-pruning policy under tractability and performance constraints remains an extremely difficult open question.
A.1 Number of k-means Clusters for SemDeDup
SemDeDup is robust to the k-means cluster count: deduplicated datasets remain nearly identical across tested k values, while duplicate-detection efficiency remains high. The clustering step also makes web-scale deduplication computationally tractable.
- 97% intersection is retained between datasets deduplicated at 72% size using different k values.Only 3% of examples are replaced when k changes among 10,000, 25,000, 50,000, and 70,000.
- The deduplicated dataset-size range across k values is only 0.003% at ϵ=0.00095 and 2% at ϵ=0.26.
- More than 94% of duplicates are detected at 63% data retained, and 89% at 40%, for k=50,000.
B CLIP Zeroshot Evaluation
CLIP models trained on SemDeDup-pruned LAION data outperform the baseline across most zero-shot evaluation tasks. At 63% of the data, the method wins on 19 of 30 evaluated tasks.
- 19 of 30 tasks favor SemDeDup over the baseline when CLIP is trained on 63% of LAION-440M.The evaluation includes 24 standard datasets and 6 out-of-distribution datasets.
- SemDeDup models outperform random pruning across the reported CLIP zero-shot evaluations.
- The CLIP evaluation uses the training parameters summarized in Table A3.
C LAION-233M De-duplication
Across image and language-model evaluations, SemDeDup removes substantial redundancy while preserving or improving performance relative to random pruning. Its benefits include stronger out-of-distribution results and faster convergence, with effects examined across multiple dataset sizes and model settings.
- 63.61% ImageNet top-1 accuracy is achieved by SemDeDup on 55% of LAION-233M, versus 61.3% for random pruning and 64.62% for the full dataset.
- The appendix compares performance across 24-dataset zero-shot evaluation, out-of-distribution evaluation, and multiple OPT perplexity settings.
- SemDeDup outperforms random pruning on all six out-of-distribution datasets across every retained-data fraction.
- SemDeDup removes increasingly more examples from larger clusters as the deduplication threshold ϵ increases.
- SemDeDup consistently outperforms random pruning at lower data fractions for both 125M and 1.3B OPT models.