PaperlayerView exact revision on arXiv

Source-linked AI summary

Do We Train on Test Data? Purging CIFAR of Near-Duplicates

Björn Barz, Joachim Denzler

arXiv:1902.00423v2cs.CV

TL;DR

Near-duplicates in CIFAR training and test sets raise questions about whether benchmark evaluations measure generalization to unseen images. The paper identifies duplicates, replaces them to create ciFAIR, and re-evaluates CNNs, finding a significant performance drop while model rankings change little.

  • Problem

    Near-duplicates across CIFAR training and test sets complicate evaluation of models’ ability to generalize to unseen data.

    Source: §1 · pp. 1–2

  • Method

    The authors identify duplicates in CIFAR-10 and CIFAR-100, replace them with same-domain images to create ciFAIR, and re-evaluate CNN architectures.

    Source: §1 · p. 2, §6 · p. 5

  • Results

    Re-evaluation on ciFAIR produced a significant performance drop, while the relative ranking of tested models did not change considerably.

    Source: §6 · p. 5

  • Takeaways & Limitations

    The authors conclude that test-set near-duplicates can substantially bias evaluation and development of image-recognition approaches.

    Source: §6 · p. 5

    Prior test-set duplicates might have prevented publication of less overparameterized approaches that could generalize as well but fail to memorize duplicates.

    Source: §6 · p. 5

Abstract

from arXiv · show

The CIFAR-10 and CIFAR-100 datasets are two of the most heavily benchmarked datasets in computer vision and are often used to evaluate novel methods and model architectures in the field of deep learning. However, we find that 3.3% and 10% of the images from the test sets of these datasets have duplicates in the training set. These duplicates are easily recognizable by memorization and may, hence, bias the comparison of image recognition techniques regarding their generalization capability. To eliminate this bias, we provide the "fair CIFAR" (ciFAIR) dataset, where we replaced all duplicates in the test sets with new images sampled from the same domain. We then re-evaluate the classification performance of various popular state-of-the-art CNN architectures on these new test sets to investigate whether recent research has overfitted to memorizing data instead of learning abstract concepts. We find a significant drop in classification accuracy of between 9% and 14% relative to the original performance on the duplicate-free test set. The ciFAIR dataset and pre-trained models are available at https://cvjena.github.io/cifair/, where we also maintain a leaderboard.

1. Introduction

The paper examines whether near-duplicates between training and test sets compromise CIFAR benchmark evaluation. It identifies duplicates, replaces them in new test sets, and re-evaluates CNNs.

1. Introduction

  • Near-duplicates can make model comparisons reflect memorization rather than generalization, especially when error-rate differences are only one or two percentage points.The significance of performance differences depends on the overlap between test and training data.Source: §1 · p. 2
  • 3.3% of CIFAR-10 and 10% of CIFAR-100 test images have near-duplicates in their respective training sets.The paper studies two heavily benchmarked computer-vision datasets.Source: §1 · p. 2
  • The authors replace duplicates with new images from Tiny Images and re-evaluate several state-of-the-art CNN architectures on the resulting test sets.The remaining test images are left untouched.Source: §1 · p. 2
  • Unlike Recht et al.’s CIFAR-10 resampling, this work also analyzes CIFAR-100, replaces only duplicates, and distinguishes three duplicate types.The authors also make the duplicate list, test sets, and pretrained models publicly available.Source: §1 · p. 2

2. The CIFAR Datasets

CIFAR-10 and CIFAR-100 are separate image-classification datasets sampled from the same source. Their small images and manageable size support fast CNN training.

2. The CIFAR Datasets

  • CIFAR-10 has 10 classes and CIFAR-100 has 100; each contains 50,000 training and 10,000 test images.The datasets’ classes and images do not overlap, although both were sampled from Tiny Images.Source: §2 · p. 2
  • Tiny Images contains approximately 80 million web-collected images queried across approximately 75,000 WordNet synsets, all resized to 32 × 32 pixels.Here, “tiny” refers to image resolution, not collection size.Source: §2 · p. 2
  • CIFAR’s low resolution and manageable size allow fast CNN training, helping establish the datasets as popular computer-vision benchmarks.Source: §2 · p. 2

3. Hunting Duplicates

The authors mine candidate duplicates using learned CNN features, manually classify the closest pairs, and find substantial overlap between CIFAR test and training images.

3.1. Mining Duplicate Candidates

  • A trained lightweight CNN produces L2-normalized global-average-pooling features; each test image is matched to its nearest training image by Euclidean distance.With L2-normalized features, Euclidean distance is equivalent to cosine distance.Source: §3.1 · pp. 2–3

3.2. Manual Annotation

  • Manual review avoids relying on distance thresholds alone, which the authors say would produce many false positives; annotators assign each pair to one of four classes.The GUI presents the image pair, its feature-space distance, and a pixel-wise difference image.Source: §3.2 · p. 3

3.3. Duplicate Statistics

  • 10% of CIFAR-100 test images and 3.25% of CIFAR-10 test images have duplicates, including training-set and within-test duplicates.Counts were 891 training-set and 104 within-test duplicates for CIFAR-100, and 286 and 39, respectively, for CIFAR-10.Source: §3.3 · p. 3
  • Most duplicates are near-duplicates; CIFAR-10 has no exact duplicates, unlike CIFAR-100.The authors suggest CIFAR-100’s basic duplicate-removal step may have been omitted during its creation.Source: §3.3 · p. 3
  • ResNet-110 error rates on duplicate-containing test images were 0% for CIFAR-10 and 2.9% for CIFAR-100, versus 5.3% and 26.1% on the full test sets.The authors say this contrast verifies that even near-duplicate and highly similar images can be classified far too easily by memorization.Source: §3.3 · p. 4

4. The Duplicate-Free ciFAIR Test Dataset

The authors replace identified duplicates in CIFAR test sets with candidate images from Tiny Images, the source of the original CIFAR datasets. They call the resulting duplicate-free test datasets ciFAIR-10 and ciFAIR-100.

4. The Duplicate-Free ciFAIR Test Dataset

  • Each replacement candidate was manually checked against its three nearest neighbors, and accepted only if it was not a duplicate of any of them.The selection GUI displayed the candidate and the neighbors from the existing training and test sets.Source: §4 · p. 4, Figure 5 · p. 4
  • For CIFAR-100, some replacement candidates were accepted despite line drawings, cartoons, or multiple instances of an object category, unlike CIFAR-10.The authors note that separate CIFAR-100 labeler instructions had not been published and that some original instructions appeared relaxed for that dataset.Source: §4 · p. 4
  • The modified datasets retain the original CIFAR training sets and use test sets from which identified duplicates have been replaced.The replacement images are sampled from Tiny Images, the source of the original CIFAR datasets.Source: §4 · p. 4
  • The datasets are called ciFAIR-10 and ciFAIR-100 (“fair CIFAR”); they consist of the original CIFAR training sets and modified test sets that are free of duplicates, and are available online.Source: §4 · p. 4

5. Re-evaluation of the State of the Art

The authors re-evaluate popular CNN architectures on the original CIFAR and modified ciFAIR test sets. Error rates rise on ciFAIR, while architecture rankings change little overall.

5. Re-evaluation of the State of the Art

  • The authors re-evaluate several popular CNN architectures on both CIFAR and ciFAIR test sets, training models themselves using original source code and authors’ instructions where available.They could not find pre-trained CIFAR models for the architectures, so their results do not exactly match those in the original papers.Source: §5 · p. 4
  • Error rates increase by 0.41 percentage points on CIFAR-10 and 2.73 percentage points on CIFAR-100 on average; the relative difference reaches 12%.Table 2 reports classification error rates on the original CIFAR and modified ciFAIR test sets.Source: §5 · p. 4, Table 2 · p. 5
  • Architecture rankings remain stable on CIFAR-100, while WRN and DenseNet swap positions on CIFAR-10.Source: §5 · p. 4

6. Conclusions

The authors identify CIFAR test-set duplicates and replace them with same-domain images in ciFAIR. Re-evaluated CNNs show a performance drop, but model rankings change little; the authors recommend ciFAIR for CIFAR evaluation.

6. Conclusions

  • The authors identify CIFAR test-set duplicates through manual annotation supported by image retrieval, then replace them with same-domain images in ciFAIR.Source: §6 · p. 5
  • Re-evaluating several state-of-the-art CNN models for image classification on this new test set led to a significant drop; even simple models such as ResNet-110 classified almost all near-duplicate images perfectly.Source: §6 · p. 5
  • Model rankings do not change considerably, but the authors say all tested models can memorize the training data.They suggest near-duplicates might have prevented publication of less-overparameterized approaches that could generalize as well but fail to memorize them.Source: §6 · p. 5
  • The authors encourage CIFAR researchers to evaluate models on ciFAIR as a better estimate of generalization to new data.Their community leaderboard accepts entries only when pre-trained models are provided so performance can be verified.Source: §6 · p. 5
Loading 1902.00423v2…