PaperlayerView exact revision on arXiv

Source-linked AI summary

Data Filtering Networks

Alex Fang, Albin Madappally Jose, Amit Jain, Ludwig Schmidt, Alexander Toshev, Vaishaal Shankar

arXiv:2309.17425v3cs.AIcs.LG

TL;DR

The paper asks how to replace ad-hoc filtering of massive uncurated training pools with a learned approach. It trains data filtering networks to induce large image-text datasets, finding that DFN-5B supports a ViT-H/14 reaching 84.4% zero-shot ImageNet accuracy and that DFN-induced datasets improve across several tasks. The authors release DFN-2B and a public-data recipe, while noting that direct optimization of dataset quality remains unresolved.

  • Problem

    Dataset curation for large-scale pre-training remains ad-hoc, and existing filtering can depend on pretrained models while offering limited guidance for improving dataset quality.

  • Method

    The paper learns neural data filtering networks from image-text data and uses them to filter large uncurated pools into pre-training datasets, applying standard model-improvement techniques to the DFNs.

  • Results

    DFN-induced datasets achieve strong compute-efficient CLIP performance across tasks, including 84.4% zero-shot ImageNet accuracy for ViT-H/14 trained on DFN-5B.

  • Takeaways & Limitations

    DFNs provide a public-data recipe for constructing billion-scale image-text datasets and enable the released DFN-2B dataset for further research.

  • Takeaways & Limitations

    The paper does not yet know how to optimize directly for dataset quality and therefore uses weak proxies such as alignment.

Abstract

from arXiv · show

Large training sets have become a cornerstone of machine learning and are the foundation for recent advances in language modeling and multimodal learning. While data curation for pre-training is often still ad-hoc, one common paradigm is to first collect a massive pool of data from the Web and then filter this candidate pool down to an actual training set via various heuristics. In this work, we study the problem of learning a data filtering network (DFN) for this second step of filtering a large uncurated dataset. Our key finding is that the quality of a network for filtering is distinct from its performance on downstream tasks: for instance, a model that performs well on ImageNet can yield worse training sets than a model with low ImageNet accuracy that is trained on a small amount of high-quality data. Based on our insights, we construct new data filtering networks that induce state-of-the-art image-text datasets. Specifically, our best performing dataset DFN-5B enables us to train state-of-the-art CLIP models for their compute budgets: among other improvements on a variety of tasks, a ViT-H trained on our dataset achieves 84.4% zero-shot transfer accuracy on ImageNet, out-performing models trained on other datasets such as LAION-2B, DataComp-1B, or OpenAI's WIT. In order to facilitate further research in dataset design, we also release a new 2 billion example dataset DFN-2B and show that high performance data filtering networks can be trained from scratch using only publicly available data.

1 Introduction

The paper treats dataset filtering as a research problem analogous to model design and introduces data filtering networks (DFNs) for constructing large, high-quality pre-training datasets. DFN-induced datasets achieve strong compute-efficient CLIP performance, including 84.4% ImageNet accuracy for ViT-H/14.

  • 1 Introduction: Dataset construction remains less systematically studied and guided than model scaling, despite datasets driving progress in machine learning.Large training sets are often unreleased or reproduced through one-off efforts, limiting iterative refinement.
  • 1 Introduction: DFNs are neural networks that filter large uncurated image-text pools into billion-scale pre-training datasets without relying on domain-specific heuristics.The authors assume a large uncurated dataset exists and focus on its filtering phase.
  • 1 Introduction: A small contrastive image-text model trained only on high-quality data can construct state-of-the-art datasets.The paper characterizes filtering-network properties across supervision signals and training-data quality.
  • 1 Introduction: 81.4% ImageNet zero-shot accuracy is achieved by ViT-L/14 trained on DFN-2B, exceeding the previous best ViT-L trained on DataComp-1B by over 2 percentage points.The model saw 12.8B examples during training.
  • 1 Introduction: 84.4% ImageNet zero-shot accuracy is achieved by ViT-H/14 trained on DFN-5B, while models trained on DFN-induced datasets improve across classification, retrieval, and visual question answering.DFN-2B and DFN-5B are induced by the same DFN; DFN training cost is omitted from the scaling plot and is less than 1/50 of total CLIP training cost.
  • 1 Introduction: The authors release DFN-2B and show that high-performance DFNs can be trained from scratch using only public data.The pipeline is presented as a recipe for constructing large datasets and supporting further dataset-design research.

2 Background and Related Work

CLIP established large-scale training on web-scraped image-text pairs, while subsequent dataset pipelines automated filtering at web scale using alignment scores and heuristics. Existing approaches provide useful benchmarks but remain dependent on pretrained models or costly, dataset-specific filtering steps.

  • 2.1 Contrastive Image Language Pre-training (CLIP): CLIP uses separate vision and text encoders with contrastive loss to bring matched image-text representations together and separate mismatched pairs.Larger CLIP vision-transformer variants generally exhibit better zero-shot generalization and transfer properties.
  • 2.2 Dataset Construction: Web-scale datasets such as LAION collect image-text pairs from Common Crawl and retain pairs whose image-text alignment exceeds a threshold.LAION uses OpenAI CLIP ViT-B/32 to compute cosine similarity for filtering.
  • 2.2 Dataset Construction: LAION-2B could only match OpenAI’s ImageNet zero-shot performance with a significantly larger compute budget despite being five times larger.The paper identifies pretrained-model dependence as a possible limitation of CLIP filtering.
  • 2.2 Dataset Construction: DataComp provides 12.8 billion Common Crawl image-text pairs and fixes training compute budgets for evaluating filtering methods.The benchmark supplies nested pools and standardized model configurations for different scales.
  • 2.2 Dataset Construction: DataComp-1B combines CLIP filtering with ImageNet-based clustering, improving on LAION-5B across benchmarks but retaining dependence on OpenAI CLIP and a costly ImageNet-specific step.This illustrates the trade-off between competitive dataset quality and pipeline complexity.
  • 2.2 Dataset Construction: Prior work also showed that keyword-based sampling and global balancing can construct competitive datasets, although these approaches rely on sampling heuristics.The present work focuses on improving model performance through dataset construction.

3 Data Filtering Networks

Data filtering networks are evaluated by the quality of the datasets they induce, not simply by their own downstream task performance. The section defines pointwise DFN filtering and compares CLIP-based and alternative filtering models using DataComp evaluations.

  • Definitions: DFNs filter individual samples from a larger data pool, enabling parallelized processing of large candidate datasets.Applying a DFN returns the samples accepted by the network; the pointwise restriction is intended to support efficient scaling.
  • Definitions: A filter dataset trains the DFN, an induced dataset is produced by filtering the pool, and an induced model trains only on that filtered dataset.
  • Definitions: A CLIP model used as a DFN filters image-text pairs, and its filtering performance is the benchmark performance of the induced model.The section commonly uses CLIP image-text models as DFNs and evaluates induced models with metrics such as ImageNet top-1.
  • Evaluation Setup: DFN quality is measured by the strength of the models it induces, using DataComp zero-shot evaluation across multiple pool scales and task categories.The evaluation includes medium, large, and xlarge pools, with ImageNet, distribution-shift, VTAB, and retrieval subsets.
  • Understanding Data Filtering Networks: ImageNet performance is uncorrelated with filtering performance: a model with 30% lower ImageNet performance than OpenAI CLIP can filter equally well.Filtering performance is measured by applying the model to DataComp medium and evaluating models trained on the resulting dataset.
  • Understanding Data Filtering Networks: Training-data quality determines filtering quality: replacing high-quality CC12M data with Common Crawl data produces DFNs whose induced datasets become only slightly better than unfiltered data.The experiment uses 10-million-sample mixtures for DFN training and a separate 128-million-sample DataComp medium pool for filtering.
  • Understanding Data Filtering Networks: Among tested alternatives, CLIP filters outperform binary classifiers and M3AE, with CLIP offering more flexible distributional assumptions and encouraging image-text alignment.The alternatives include ResNet and frozen-CLIP binary classifiers plus an M3AE reconstruction-loss filter trained on CC12M.

4 Creating Better Data Filtering Networks

The paper improves DFNs by training and fine-tuning CLIP-based filters, then applying them to large unfiltered pools to induce stronger datasets. These datasets achieve state-of-the-art results, improve compute efficiency, transfer beyond vision, and preserve robustness under distribution shift.

  • DFN design: DFNs are trained like standard models, beginning with high-quality data and using fine-tuning, weight ensembling, augmentation, and longer training to improve filtering.The pipeline applies model-design interventions to the filtering network rather than only to the final model.
  • Dataset-induced performance: 81.4% ImageNet zero-shot accuracy and 0.669 average performance across 38 DataComp datasets are achieved by ViT-L/14 trained on DFN-2B.This exceeds DC-1B by 2.2%, OpenAI WIT-400M by 5.9%, and LAION-2B by 8.3% on ImageNet zero-shot improvement.
  • Dataset-induced performance: 84.4% ImageNet zero-shot transfer accuracy and 0.710 average DataComp performance are achieved by ViT-H/14 trained on DFN-5B.The model uses 39B samples at 224 × 224 resolution and 5B samples at 378 × 378 resolution.
  • Efficiency: A ViT-L/14 trained on DFN-2B surpasses a ViT-G/14 trained on LAION-2B by 1.5% ImageNet zero-shot accuracy while using 16x less computational cost.A ViT-B/16 trained on DFN-2B also reaches competitive performance with OpenAI’s ViT-L/14 at a 4x computational cost reduction.
  • Using high-quality data: Training a DFN on HQITP-350M produces a better induced dataset than directly combining HQITP-350M with a dataset from a worse DFN.This supports using high-quality data to train the filtering model rather than directly training the end model.
  • Robustness: Fine-tuning a DFN on ImageNet improves ImageNet and all tested distribution shifts, whereas directly training on ImageNet does not improve ObjectNet, ImageNet-Sketch, or ImageNet-R.The paper attributes the robustness difference to filtering from a more distributionally diverse candidate pool.
  • Beyond vision: DFN-2B improves zero-shot VQA performance consistently across VQAv2, GQA, and OKVQA compared with the standard OpenAI ViT-L visual encoder.The DFN-2B encoder is competitive with a model using the standard encoder while improving across the reported VQA tasks.
  • Public reproducibility: A competitive DFN can be trained from scratch using publicly available Conceptual Captions 12M, Conceptual Captions 3M, and Shutterstock 15M data.This DFN matches OpenAI’s ViT-B/32 filtering performance at DataComp medium and large scales.

5 Discussion

DFNs offer a simple, scalable way to filter raw data during training, but important questions remain about how to optimize dataset quality directly. The paper currently relies on weak proxies such as alignment, whose applicability beyond image-text data is unclear.

  • Practicality: DFNs operate on individual samples and scale linearly with candidate-pool size, enabling filtering of much larger datasets.Their small size also permits integration into training procedures for larger networks with minimal marginal cost.
  • Practicality: DFNs can filter raw-data batches directly before training, reducing the need for complex preprocessing procedures.The pipeline is described as flexible to integrate into existing workflows.
  • Limitations: The paper does not yet know how to optimize directly for dataset quality and therefore uses weak proxies such as alignment.This is identified as an unanswered question for future work.
  • Limitations: The appropriate filtering proxy remains unclear for other domains, including speech, text, and video data.The paper frames this as an open question rather than a demonstrated failure of DFNs in those domains.

A Training Hyperparameters

The appendix states that DataComp hyperparameter settings are used for the medium, large, and xlarge pools.

  • DataComp hyperparameter settings are followed for the medium, large, and xlarge pools.

B DFN Hyperparameters

The appendix specifies separate training settings for ablation DFNs and the final DFNs inducing DC-2B.

  • Ablations: Ablation DFNs use DataComp large-scale hyperparameters with ViT-B/32 instead of ViT-B/16.
  • Final DFNs: Final DFNs inducing DC-2B train for 5.12B samples with batch size 16,384 and 2,000 warmup steps.

C Robustness of Using ImageNet at Filtering vs. Training Time

Table 8 examines robustness when ImageNet is used to fine-tune a data filtering network rather than directly for training. The experiment uses DataComp large scale with ViT-B/16 and 1.28B samples.

  • Table 8 evaluates fine-tuning a data filtering network on ImageNet.
  • The comparison concerns datasets induced by ImageNet fine-tuning versus direct ImageNet training.
  • The experiment was run at DataComp large scale with ViT-B/16 and 1.28B samples.

D Full Experimental Evaluation & Model Release

This section provides access to checkpoints and detailed evaluation results, including results for models in Table 3 across 38 DataComp evaluation datasets.

  • The paper provides links to model checkpoints.
  • Detailed evaluation results are provided for models in Table 3.
  • The evaluation covers each of the 38 DataComp evaluation datasets.

E Figures measuring average performance instead of ImageNet

The figures and accompanying table report average performance rather than only ImageNet performance. They organize comparisons across multiple datasets, training steps, and model types.

  • The listed datasets include CC12M+CC3M+SS15M, HQITP-135M, HQITP-350M, LAION-2B, OpenAI-WIT400M, CommonPool, and DataComp-1B.
  • The comparison lists training steps from 1B through 34B.
  • The listed model types are ViT-B/32 and ViT-L/14.
  • Figures 6 and 7 show average-accuracy versions of Figures 3 and 4.

F Log-Scale plot of Figure 1

This section presents a log-scale view of compute scaling for CLIP training across datasets and notes that high-quality data filtering networks can be produced entirely from scratch.

  • Figure 8 plots compute scaling behavior for training CLIP models on various datasets using a log scale.
  • Table 10 states that high-quality data filtering networks can be produced completely from scratch.
  • The from-scratch results do not use OpenAI CLIP models.
  • DFN training in Table 10 uses HQITP-135M, a subset of HQITP-350M used elsewhere in the paper.
Loading 2309.17425v3…