PaperlayerView exact revision on arXiv

Source-linked AI summary

A Decade's Battle on Dataset Bias: Are We There Yet?

Zhuang Liu, Kaiming He

arXiv:2403.08632v2cs.CVcs.LG

TL;DR

This paper revisits whether modern models can distinguish images from large, diverse datasets presumed to be less biased. It tests dataset classification and finds that neural networks do so with high accuracy, while further experiments indicate that the learned signal can generalize and transfer.

  • Problem

    The study asks whether dataset bias remains detectable in modern, large-scale datasets developed to be diverse and less biased.

    Source: §1 · p. 1

  • Method

    The authors revisit dataset classification by training neural networks to predict which of several large datasets an image comes from.

    Source: §1 · p. 1

  • Results

    More than 84% held-out validation accuracy on YCD, versus 33.3% for chance, and the finding is robust across dataset combinations and architecture generations.

    Source: §1 · p. 1

  • Takeaways & Limitations

    The results indicate that dataset bias persists in modern datasets and that its learned patterns may generalize and transfer, motivating further discussion of dataset representativeness.

    Source: §1 · p. 2, §7 · p. 11

    The concrete forms of bias captured by neural networks remain largely unclear.

    Source: §7 · p. 11

Abstract

from arXiv · show

We revisit the "dataset classification" experiment suggested by Torralba & Efros (2011) a decade ago, in the new era with large-scale, diverse, and hopefully less biased datasets as well as more capable neural network architectures. Surprisingly, we observe that modern neural networks can achieve excellent accuracy in classifying which dataset an image is from: e.g., we report 84.7% accuracy on held-out validation data for the three-way classification problem consisting of the YFCC, CC, and DataComp datasets. Our further experiments show that such a dataset classifier could learn semantic features that are generalizable and transferable, which cannot be explained by memorization. We hope our discovery will inspire the community to rethink issues involving dataset bias.

1 INTRODUCTION

This study revisits dataset bias by testing whether modern models can identify images from large, diverse datasets that are presumed less biased. Models do so with high held-out accuracy, and experiments indicate that the signal includes generalizable, transferable features rather than mere memorization.

1 INTRODUCTION

  • The renewed study is motivated by a tension: less biased data may help, while more capable models may better exploit dataset bias.The authors ask how representative current pretraining datasets are of the real world and whether greater dataset diversity could improve model generalizability.Source: §1 · pp. 1–2
  • Dataset classification revisits Torralba & Efros’s (2011) “Name That Dataset” experiment using YFCC, CC, and DataComp in a three-way task.The study describes these internet-collected datasets as among the largest, most diverse, and least curated datasets in the wild.Source: §1 · p. 1
  • More than 84% held-out validation accuracy on YCD, versus 33.3% for chance, shows that modern neural networks can identify the dataset of an image.The result is robust across dataset combinations and architecture generations, with over 80% accuracy in most cases.Source: §1 · p. 1
  • More samples and stronger augmentation can improve held-out dataset-classification accuracy, resembling generalization in semantic classification.The authors interpret this behavior as evidence that networks discover dataset-specific patterns; the learned representations also transfer to image classification.Source: §1 · p. 2
  • On pseudo-datasets uniformly sampled from one dataset, classification becomes intractable as more samples or stronger augmentation make memorization harder, and no transferability appears.These behaviors contrast with real dataset classification.Source: §1 · p. 2
  • With frozen self-supervised representations and only a dataset-label-trained linear classifier, dataset classification reaches 78% accuracy.The self-supervised model is pretrained on the union of datasets without dataset-identity labels.Source: §1 · p. 2
  • The authors conclude that dataset bias remains unresolved even in large, diverse, less curated datasets, and call for further discussion of its forms and implications.They note that the concrete forms of bias remain unclear, while some patterns may generalize and transfer and may not be readily noticed by people.Source: §1 · p. 2

2 A BRIEF HISTORY OF DATASETS

Computer vision datasets evolved from synthesized stimuli and isolated test images into task-defining training and validation sets. With transferable representations, ImageNet became a pretraining resource, and large diversified datasets increasingly served representation learning rather than a single defined task.

2 A BRIEF HISTORY OF DATASETS

  • Early vision research used synthesized stimuli such as lines, stripes, and blobs to examine models of human vision.The use of synthesized patterns continued in early computer vision.Source: §2 · p. 3
  • After photo-digitizing devices appeared, algorithms could be tested on one or a few real images, including the Cameraman image, used as a standard test image since 1978.The community gradually formed the concept of using data to evaluate computer vision algorithms.Source: §2 · p. 3
  • Machine learning clarified the dataset concept by adding training data for model optimization alongside validation data.Together, training and validation data define tasks such as MNIST’s 10-digit classification and Caltech-101’s 101-category classification.Source: §2 · p. 3
  • ImageNet exemplified increasingly challenging tasks with over one million images and 1,000 classes, including fine-grained animal categories.The passage notes that its scale and difficulty challenged both human recognition and the algorithms of its time.Source: §2 · p. 3
  • After the 2012 deep-learning revolution, researchers found that large-scale dataset representations could transfer to downstream tasks, prompting routine ImageNet pretraining.ImageNet became a pinhole into a universal visual world rather than a task of its own.Source: §2 · p. 3
  • Large, diversified datasets such as YFCC100M, CC12M, and DataComp-1B were built for pretraining, and many do not provide training/validation splits.Their construction does not always define a task of interest.Source: §2 · p. 3

3 ON DATASET BIAS

Research on dataset bias includes dataset classification, domain adaptation, replicated-data tests, and bias-mitigation methods. This paper distinguishes bias between datasets in real-world representativeness from social and stereotypical bias, which can occur within a dataset.

3 ON DATASET BIAS

  • Torralba & Efros studied dataset bias with hand-crafted features and SVMs, while Tommasi et al. used neural networks with linear classifiers on pretrained ConvNet features and smaller, simpler datasets.These works provide earlier dataset-classification settings for comparison with web-scale datasets.Source: §3 · pp. 3–4
  • Domain-adaptation methods distinguish features from different domains adversarially, whereas this study considers datasets presumed less distinguishable to people.The passage notes that domain-adaptation problems are known to have significant domain gaps.Source: §3 · p. 4
  • ImageNetV2 replicated the ImageNet validation-set collection protocol and found bias reflected in accuracy degradation.The bias was further analyzed in subsequent work.Source: §3 · p. 4
  • Benchmarks test generalization under biases such as corruptions and hazardous conditions, while other work explores multi-dataset training and test-time adaptation to mitigate bias.Adversarial learning and domain-independent training are also described as promising debiasing approaches.Source: §3 · p. 4
  • Dataset bias concerns whether concepts and objects represent the real world; social and stereotypical bias concerns fairness and can occur within one dataset.The two notions are related but emphasize different aspects; a dataset can have little social bias yet be unrepresentative of the world.Source: §3 · p. 4

4 DATASET CLASSIFICATION

The study defines dataset classification and tests it on large, diverse datasets collected from the internet. Modern networks perform strongly across dataset combinations and architectures, while training-data and augmentation trends suggest they learn generalizable patterns rather than simply memorizing images.

4 DATASET CLASSIFICATION

  • Dataset classification treats each dataset as a class in an N-way image-classification task evaluated on held-out images.The datasets were selected for large scale, diversity, and use in pre-training generalizable representations.Source: §4 · p. 4, §4.1 · p. 4

4.1 ON THE DATASETS WE USE

  • The selected internet datasets differ in sources and curation, leaving potential dataset-specific biases despite their intended diversity.The selection includes YFCC, CC, DataComp, WIT, LAION, and ImageNet.Source: §4.1 · p. 5

4.2 MAIN OBSERVATION

  • All 20 three-dataset combinations exceed 62% accuracy, 16 exceed 80%, and YFCC, CC, and ImageNet reach 92.7%.With all six datasets, accuracy is 69.2%.Source: Table 2 · p. 5, §4.2 · p. 6
  • Four of five architectures exceed 80% accuracy on YCD, while AlexNet reaches 77.8%.The authors report that no specific architectural component appears indispensable for dataset classification.Source: Table 3 · p. 6, §4.2 · p. 6
  • A ConvNeXt with 7K parameters achieves 72.4% accuracy on YCD, and models of different sizes achieve high accuracy.The authors report diminishing returns as model size increases.Source: §4.2 · pp. 6–7, Figure 2 · p. 6
  • More training images improve held-out accuracy, a trend the authors interpret as evidence of generalizable patterns rather than memorization.The trend is reported for both ConvNeXt and AlexNet.Source: Figure 3 · p. 6, §4.2 · p. 7
  • Stronger data augmentation consistently improves dataset-classification accuracy across training-set sizes, despite making memorization harder.The default setting uses random cropping, RandAug, MixUp, and CutMix.Source: §4.2 · p. 7, Table 4 · p. 7

5 ANALYSIS

The analysis tests whether dataset classification relies on low-level artifacts or memorization, and whether its learned features generalize or transfer. Results show that dataset-specific signals persist under corruption, differ from memorization on pseudo-datasets, and can support transfer, while cross-dataset generalization remains limited.

5.1 LOW-LEVEL SIGNATURES?

  • Strong dataset-classification accuracy persists after color jitter, noise, blur, or resolution reduction, suggesting models use more than low-level signatures.Each corruption is applied to both training and validation data; corruption lowers accuracy, especially as its degree increases.Source: §5.1 · p. 8, Table 5 · p. 8

5.2 MEMORIZATION OR GENERALIZATION?

  • On pseudo-datasets sampled without replacement from one source, harder settings impede training, and held-out accuracy remains near the 33% chance level.The authors interpret this pattern as memorization of individual images and random labels, unlike real dataset classification, where shared generalizable patterns are learned.Source: Table 6 · p. 8, §5.2 · p. 9

5.3 SELF-SUPERVISED LEARNING

  • MAE linear probing reaches 76.2% dataset-classification accuracy after ImageNet pre-training and 78.4% after pre-training on YCD.Only the linear classifier is supervised for dataset identity; the YCD-pre-trained MAE had no prior knowledge of that goal.Source: Table 7 · p. 9, §5.3 · p. 9

5.4 FEATURES LEARNED BY CLASSIFYING DATASETS

  • Dataset-classification features achieve nontrivial ImageNet-1K linear-probing accuracy, with accuracy increasing when more datasets are combined, but remain significantly worse than MAE and MoCo v3 features.The authors interpret the increase as evidence that discovering biases across more datasets can yield better features.Source: Table 8 · p. 9, §5.4 · p. 10

5.5 CROSS-DATASET GENERALIZATION

  • Combining all datasets yields the best overall result, yet each evaluation dataset has its lowest validation loss when pre-training uses that same dataset.The combined-dataset result controls the total image count by sampling one-sixth from each dataset.Source: Table 9 · p. 10, §5.5 · p. 10

6 USER STUDY

The user study asked volunteers to classify YFCC, CC, and DataComp images, with access to labeled training examples. Participants performed above chance but well below the neural network, and their reported cues varied in how useful they were for distinguishing datasets.

Settings and users

  • Twenty machine-learning researchers classified 100 held-out validation images each and could browse labeled training images without a time limit.Fourteen participants had computer-vision research experience.Source: §6 · pp. 10–11

User study results

  • Human accuracy averaged 45.4% (median 44%), versus 33.3% chance and the neural network’s 84.7%.Among 20 participants, 11 scored 40–45%, seven scored 45–50%, and two exceeded 50%.Source: §6 · p. 11, Figure 5 · p. 11

User-reported patterns

  • Participants named dataset-specific visual cues, including people and scenery for YFCC, cartoons and colorful images for CC, and white backgrounds for DataComp.The reported cues also included products, text, and cleaner backgrounds for DataComp.Source: §6 · p. 11
  • Some cues, such as DataComp’s white backgrounds, may help distinguish datasets, while “people” is not useful because all datasets include people.Fifteen participants described the task as difficult, and none called it easy.Source: §6 · p. 11

7 CONCLUSION

The paper concludes that modern neural networks can capture dataset bias even in large, diverse datasets, and that some captured patterns may generalize and transfer. The specific forms of those patterns remain largely unclear.

Conclusion

  • Dataset bias remains capturable by modern neural networks across models, dataset combinations, and other settings.The study revisits dataset classification with modern neural networks and large-scale datasets.Source: §7 · p. 11
  • The captured bias may include generalizable and transferable patterns that people do not readily notice.The concrete forms of the bias remain largely unclear.Source: §7 · p. 11
  • The authors hope further study of dataset bias will help build less-biased datasets in the future.Source: §7 · p. 11

A IMPLEMENTATION DETAILS

The implementation samples and preprocesses images for dataset classification, then trains models for a fixed sample budget. Linear-probing and cross-dataset experiments use settings based on MAE and MoCo v3, respectively.

Data and sampling

  • The default dataset-classification setup samples 1M images per dataset, using only train splits when datasets provide predefined splits.Image-text datasets contribute images only; LAION is filtered before use, and the sampled images are divided into training and validation sets.Source: A IMPLEMENTATION DETAILS · p. 16

Training

  • Training uses the same number of samples seen as a typical 300-epoch ImageNet-1K supervised run, regardless of training-set size.The batch size is held constant, so this budget also corresponds to the same number of iterations.Source: A IMPLEMENTATION DETAILS · p. 16, Table 10 · p. 16

Experiment settings

  • ViT-B linear probing follows MAE settings, while cross-dataset generalization uses MoCo v3 hyperparameters and a batch size of 1024.For Section 5.4, the study uses the epoch-250 checkpoint and sweeps learning rates and feature-extraction layers.Source: A IMPLEMENTATION DETAILS · p. 16

Image preprocessing

  • Inference resizes images to a 256-pixel short side and takes a 224×224 center crop, preventing direct use of resolution or aspect-ratio distributions as shortcuts.Training uses randomly augmented 224×224 crops.Source: A IMPLEMENTATION DETAILS · p. 16

B ADDITIONAL RESULTS

Additional results show fast convergence, strong discrimination between closely related datasets, and same-dataset advantages under MAE.

B ADDITIONAL RESULTS

  • ConvNeXt-T YCD training converges quickly to high accuracy during the initial phases.Figure 6 plots training loss and validation accuracy.Source: Figure 6 · p. 16, B ADDITIONAL RESULTS · p. 16
  • DataComp and LAION are highly confused in 6-way dataset classification; both source from Common Crawl and filter for caption alignment in CLIP space.The authors suggest their shared sourcing and filtering explain the confusion.Source: Figure 7 · p. 17, B ADDITIONAL RESULTS · p. 17
  • With only 8K training images from each, a classifier reached 81.8% accuracy classifying ImageNetV2 and ImageNet-1K’s validation set, substantially above 50%.Source: B ADDITIONAL RESULTS · p. 17
  • MAE pretraining shows a diagonal pattern of lowest validation loss for same-dataset evaluation, with smaller per-column loss differences than contrastive learning.Table 11 marks the lowest validation loss for each evaluation dataset.Source: Table 11 · p. 17, B ADDITIONAL RESULTS · p. 17

C LIMITATIONS

The paper leaves the concrete forms of learned bias unresolved and examines only six large-scale image datasets, excluding other popular datasets and video or language.

C LIMITATIONS

  • The exact forms of bias captured by neural networks remain unclear.The authors identify understanding and interpreting those forms as future research.Source: C LIMITATIONS · p. 17
  • The study covers a limited set of six large-scale image datasets.Source: C LIMITATIONS · p. 17
  • Other popular image datasets and domains such as video and language are outside the study's scope.Source: C LIMITATIONS · p. 17

Related papers

Loading 2403.08632v2…