Source-linked AI summary
CINIC-10 is not ImageNet or CIFAR-10
Luke N. Darlow, Elliot J. Crowley, Antreas Antoniou, Amos J. Storkey
TL;DR
CIFAR-10 is widely used for initial benchmarking, but the report identifies a need for a larger, more challenging dataset with equal splits, while ImageNet is unwieldy. CINIC-10 extends CIFAR-10 with selected ImageNet images downsampled to 32×32; the added images are randomly distributed across the new train, validation, and test sets. The resulting dataset contains 270,000 images, including 60,000 from CIFAR-10, in three equal-sized splits with 9,000 images per class in each split.
Problem
CIFAR-10 is widely used for initial benchmarking, but the report identifies a need for a larger, more challenging dataset with equal splits, while ImageNet is unwieldy.
Source: §1 · p. 2Method
Results
The resulting dataset contains 270,000 images, including 60,000 from CIFAR-10, in three equal-sized splits with 9,000 images per class in each split.
Source: §2 · p. 3Takeaways & Limitations
CINIC-10 is presented as a larger, more challenging alternative to CIFAR-10 that is less difficult than ImageNet and supports same-class distribution-shift testing.
Source: §1 · p. 3, §6 · p. 10CINIC-10 examples can be noisier, less explicitly cropped and centred, and less class-distinct than CIFAR-10 examples.
Source: §4.2 · p. 5
Abstract
from arXiv · showhide
In this brief technical report we introduce the CINIC-10 dataset as a plug-in extended alternative for CIFAR-10. It was compiled by combining CIFAR-10 with images selected and downsampled from the ImageNet database. We present the approach to compiling the dataset, illustrate the example images for different classes, give pixel distributions for each part of the repository, and give some standard benchmarks for well known models. Details for download, usage, and compilation can be found in the associated github repository.
1 Motivation
CIFAR-10 is convenient for initial benchmarking, while ImageNet is computationally unwieldy; existing datasets also leave a gap in task difficulty and split design. CINIC-10 extends CIFAR-10 with downsampled ImageNet images to offer a larger, more challenging benchmark with equal splits and a same-class distribution-shift test.
1 Motivation
- ImageNet is unwieldy, with large images and over a million examples, while CIFAR-10 and CIFAR-100 are popular for initial neural-network benchmarking.A single ImageNet training run can take several days without abundant computational resources.Source: §1 · p. 2
- CIFAR-10 has 6,000 examples per class and CIFAR-100 has 600; a benchmark offering another difficulty milestone and equal train, validation, and test splits would be useful.The report notes that ImageNet-32 is more challenging than ImageNet because downsampling reduces image information.Source: §1 · p. 2
- CINIC-10 (CINIC-10 Is Not ImageNet or CIFAR-10) is presented to combat shortcomings of existing benchmarking datasets, extending CIFAR-10 through the addition of downsampled ImageNet images.Source: §1 · p. 3
- At 270,000 images, CINIC-10 has 4.5× as many images as CIFAR, and its CIFAR-sized images make it a drop-in alternative to CIFAR-10.Equal-sized train, validation, and test splits support fair assessment of generalization; train and validation can also be combined for a larger training set.Source: §1 · p. 3
- CINIC-10 combines CIFAR and ImageNet images that may not be identically distributed, enabling tests of CIFAR-trained models on ImageNet images from the same classes.This makes distribution shift an explicit challenge in the dataset.Source: §1 · p. 3
2 Construction
CINIC-10 is constructed by reformatting and repartitioning CIFAR-10, then adding selected, downsampled ImageNet images. The resulting 270,000-image dataset has three equal splits and 9,000 images per class in each split.
2 Construction
- The construction repartitions CIFAR-10 into three equal sets of 20,000 images, with 2,000 per class per set, while retaining the full original CIFAR-10 test set in CINIC-10's test set.The remaining CIFAR images are randomly split between train and validation, and filenames permit recovery of the original CIFAR-10 data.Source: §2 · p. 3
- Relevant synsets from the Fall 2011 ImageNet release were identified and collected, with mappings to CINIC-10 classes recorded in repository files.Some archive downloads failed, but the report says this was not exceedingly detrimental because only a subset of downloaded images was used.Source: §2 · p. 3
- ImageNet JPEGs were converted to 32×32 colour images with Pillow's Box algorithm, matching the approach used for ImageNet32x32 for consistency.The lowest observed class-relevant sample count was 21,939 for truck, so 21,000 samples were randomly selected from each synset-group and assigned across the new splits.Source: §2 · p. 3
- The resulting CINIC-10 contains 270,000 images: 60,000 from CIFAR-10 and the remainder from ImageNet, divided into three equal-sized train, validation, and test subsets.Each class in each subset contains 9,000 images.Source: §2 · p. 3
3 Download and Usage
The section points readers to the dataset repository and shows a PyTorch data-loader setup for using CINIC-10.
3 Download and Usage
- The dataset download is available through the associated repository and the University of Edinburgh DataShare.The repository contains further download details.Source: §3 · p. 4
- A PyTorch data loader uses torchvision's ImageFolder to load the CINIC-10 training directory.The example sets the directory path and provides channel-wise mean and standard deviation values for normalization.Source: §3 · p. 4
- The code sets the CINIC directory, specifies mean [0.47889522, 0.47227842, 0.43047404] and standard deviation [0.24205776, 0.23828046, 0.25874835], and begins a training DataLoader declaration.Source: §3 · p. 4
4 Analysis
The analysis presents color-intensity distributions for the CIFAR-10 and ImageNet contributions and examples from CINIC-10 classes.
4.1 Distribution
- CIFAR-10 and ImageNet contribution intensity distributions are very similar, but not identical.Figure 1 shows histograms across red, green, and blue channels, with dashed lines marking the means.Source: Figure 1 · p. 4, §4.1 · p. 4
4.2 Examples
- Randomly selected CINIC-10 examples are noisier, may be uncropped or off-center, and can be less class-distinct than CIFAR-10 examples.The deer class may include cows and goats; the examples include automobile and airplane panels.Source: §4.2 · p. 5, Figure 2 · p. 5, Figure 3 · p. 5
5 Benchmarks
The section reports CINIC-10 benchmark results and notes that the tested model definitions were copied from CIFAR-10 experiments.
5 Benchmarks
- Table 1 presents benchmark results on CINIC-10.Source: §5 · p. 5, Table 1 · p. 6
- The model definitions tested on CINIC-10 were copied from those applied to CIFAR-10.Source: §5 · p. 5
6 Conclusion
CINIC-10 augments CIFAR-10 with downsampled ImageNet images, creating a dataset larger and more challenging than CIFAR-10 but less difficult than ImageNet.
6 Conclusion
- CINIC-10 is larger and more challenging than CIFAR-10, but not as difficult as ImageNet.The dataset was compiled by augmenting CIFAR-10 with downsampled images sourced from ImageNet.Source: §6 · p. 10