PaperlayerView exact revision on arXiv

Source-linked AI summary

Are All Training Examples Created Equal? An Empirical Study

Kailas Vodrahalli, Ke Li, Jitendra Malik

arXiv:1811.12569v1cs.LGcs.CVstat.ML

TL;DR

The paper asks whether carefully selected subsets can replace some of the large image datasets used to train modern computer-vision models. It analyzes image importance with gradient-based measures and finds that redundancy differs across four datasets: MNIST has largely redundant examples, while most examples in CIFAR-10, CIFAR-100, and ImageNet are valuable.

  • Problem

    The paper examines whether a carefully selected subset can suffice when modern deep-learning models rely on large training datasets.

    Source: §1 · p. 1

  • Method

    The authors use individual-image loss-gradient magnitudes to assess importance and retrain networks on subsets selected by that measure.

    Source: §3.1 · p. 3

  • Results

    Most training examples are valuable in CIFAR-10, CIFAR-100, and ImageNet, while most MNIST examples are largely redundant.

    Source: §6 · p. 9

  • Takeaways & Limitations

    The results suggest that CIFAR-10, CIFAR-100, and ImageNet are difficult to reduce in size while maintaining performance, unlike MNIST, where maximum-gradient subsampling performs better.

    Source: §5 · p. 8, §6 · p. 9

    The empirical results are specific to the deep-learning models used, although the authors report evidence that some image-importance findings generalize across models.

    Source: §6 · p. 9

Abstract

from arXiv · show

Modern computer vision algorithms often rely on very large training datasets. However, it is conceivable that a carefully selected subsample of the dataset is sufficient for training. In this paper, we propose a gradient-based importance measure that we use to empirically analyze relative importance of training images in four datasets of varying complexity. We find that in some cases, a small subsample is indeed sufficient for training. For other datasets, however, the relative differences in importance are negligible. These results have important implications for active learning on deep networks. Additionally, our analysis method can be used as a general tool to better understand diversity of training examples in datasets.

1. Introduction

Because deep networks require millions of images and dataset collection and annotation can be expensive, the paper examines how important individual training images are and how dataset properties vary. It proposes simplicity and redundancy as comparison attributes, finding redundancy in MNIST but diversity in CIFAR-10, CIFAR-100, and ImageNet.

  • The analysis is motivated by networks’ need for millions of images and the expense of collecting and annotating large datasets.A common practice is pretraining on a large image dataset and then fine-tuning on a smaller application-specific dataset.Source: §1 · p. 1
  • The paper identifies a lack of analysis of dataset properties despite their importance to applications.Source: §1 · p. 1
  • It measures image importance using each training image’s loss-gradient magnitude after training, then retrains on selected subsets across four image-classification datasets.The datasets are MNIST, CIFAR-10, CIFAR-100, and ImageNet.Source: §1 · p. 1
  • The proposed simplicity and redundancy attributes distinguish redundant MNIST data from diverse CIFAR-10, CIFAR-100, and ImageNet data.Source: §1 · p. 1

2. Related Work

The related work connects image-importance analysis to active learning and data-distribution analysis. It contrasts oracle-guided sample selection with dataset analyses that examine distribution properties, while motivating a trained-model perspective on diversity.

2. Related Work

  • The paper positions relative image importance at the intersection of active learning’s informative-example selection and data-distribution analysis of data similarity.Source: §2 · pp. 1–2

2.1. Active Learning

  • Active learning uses an oracle to select data, can reduce labeling needs or training time, and has mixed evidence across datasets, models, and applications.Source: §2.1 · p. 2
  • Drawing on SVM support-vector intuition, the paper treats largest-gradient images as critical examples but fixes a large batch rather than querying data adaptively.Source: §2.1 · p. 2

2.2. Data Distribution Analysis

  • Dataset descriptions often report surface properties, but these do not fully represent relations to the true data manifold; prior work also studies dataset bias.Examples of surface properties include labels and image sizes; bias work considers dataset characteristics such as image viewpoints or geographic origin.Source: §2.2 · p. 2
  • Rather than relying only on basic image statistics or human-annotated qualities, the paper investigates diversity from a trained model’s perspective; minibatch gradient sums are reported to approximate gradient ordering.Source: §2.2 · p. 2

3. Methods

The analysis ranks training images by the magnitude of their individual loss gradients, then tests how gradient-based subsamples compare with random sampling. It evaluates this procedure across four image datasets and describes three gradient-based sampling strategies alongside a random baseline.

3. Methods

  • The study analyzes MNIST, CIFAR-10, CIFAR-100, and ImageNet using one to three standard network architectures per dataset, with 10% of data reserved for validation.The validation split is made before the analysis.Source: §3 · pp. 2–3

3.1. Gradient Magnitude as Importance Measure

  • Individual loss-gradient magnitudes provide a conservative signal for ranking images: larger norms tend to correspond to larger batch gradients, though false positives are possible.The authors use the triangle-inequality upper bound and report a linear correlation supporting it as an approximation for ordering gradient values.Source: §3.1 · pp. 3–4

3.2. Subsample Selection

  • For each dataset, the procedure trains on all training data, computes per-image gradients, then retrains from random initialization on a selected subset and records test accuracy.Validation data is used for early stopping during the full-data training; subset accuracy measures how well it represents the full dataset.Source: §3.2 · p. 5

3.3. Batch Selection

  • The three gradient-based strategies are Max-Gradient, Non-extreme Max-Gradient, and Gradient-CDF, compared with a random-subsampling baseline.Non-extreme Max-Gradient discards the top 5% of images before selecting; Gradient-CDF weights sampling probabilities by gradient magnitude and adds randomness.Source: §3.3 · p. 5

4. Experiments

Experiments compare gradient-based subsampling across four image datasets and several network settings. Results vary: MNIST can be approximated by a small subset, while gradient-based sampling is less consistently advantageous on CIFAR and ImageNet.

4. Experiments

  • The experiments cover MNIST, CIFAR-10, CIFAR-100, and ImageNet, reporting ℓ2 gradient norms over last-layer weights for the first three and weights for ImageNet.The study also considers VGG16 [21], AlexNet, and additional shallow networks for MNIST and CIFAR-10.Source: §4 · pp. 5–6

4.1. MNIST

  • At only 5% of training data, Max-Gradient overtakes the other MNIST sampling methods and is already nearly at full-data test accuracy.The largest-gradient images are described as difficult to classify, while other images become progressively easier to read.Source: §4.1 · p. 7

4.2. CIFAR-10

  • CIFAR-10's top-gradient images show lower class-label entropy at smaller k, while VGG16 and the CNN overlap more than the random baseline but remain model-specific.The greater label-distribution skew is offered as a possible contributor to decreased Max-Gradient test accuracy.Source: Figure 7 · p. 6, §4.2 · pp. 7–8
  • On CIFAR-10, gradient-based sampling can improve VGG16 test accuracy, but random sampling can do better, unlike the clearer benefit on MNIST.The largest-gradient images have limited overlap across models at small subsample sizes, and label entropy falls as fewer images are selected.Source: Figure 8 · p. 6, §4.2 · p. 8

4.3. CIFAR-100

  • On CIFAR-100, gradient-based sampling performs worse than random sampling, and the largest gradients have a markedly skewed class-label distribution.The skew may indicate that some classes are harder to classify than others.Source: Figure 8 · p. 6, §4.3 · p. 8

4.4. ImageNet

  • On ImageNet, Max-Gradient performs below Random and Gradient-CDF for AlexNet; VGG16 shows similar trends through 40% sampled data.The authors suggest this pattern may indicate similar gradient-based importance values between the two networks.Source: Figure 9 · p. 7, §4.4 · p. 8

5. Discussion

MNIST shows redundancy that can make subsampling effective, whereas CIFAR-10, CIFAR-100, and ImageNet show little redundancy and are harder to reduce without losing performance. Gradient-based sampling can also skew class representation, and its performance varies across datasets.

5. Discussion

  • On MNIST, maximum-gradient subsampling performs better, while random sampling beats Max-Gradient on CIFAR-10 and CIFAR-100 and closely matches Gradient-CDF.The authors interpret the CIFAR results as evidence that these datasets are diverse rather than redundant.Source: §5 · p. 8
  • Gradient-ranked sampling skews class distributions and can make generalization more difficult; Non-extreme Max-Gradient lowers CIFAR-100 accuracy but roughly matches Random on CIFAR-10.The authors suggest CIFAR-100's tenfold larger class count may exacerbate distribution skew.Source: §5 · p. 8
  • On ImageNet, Gradient-CDF closely matches Random in test accuracy, while Max-Gradient achieves significantly lower test accuracy.The authors note that top-gradient images appear more varied on ImageNet than on CIFAR-100, possibly reflecting greater diversity.Source: §5 · p. 8
  • MNIST contains redundant data, but CIFAR-10, CIFAR-100, and ImageNet contain little redundancy, despite CIFAR-10 and MNIST having similar class counts and image data sizes.The paper offers dataset collection and the larger underlying space of animals and vehicles than Arabic numerals as possible explanations.Source: §5 · pp. 8–9
  • It may be substantially more difficult to reduce training data while maintaining performance on CIFAR-10, CIFAR-100, and ImageNet than on MNIST, suggesting active learning may be more challenging there.Source: §5 · p. 9

6. Conclusion

The authors propose an empirical approach to analyze image-data diversity and find that most examples are valuable in CIFAR-10, CIFAR-100, and ImageNet, while many MNIST examples are redundant. Their results are tied to the deep-learning models tested, and they identify batch-based importance measures as a possible direction with computational tradeoffs.

6. Conclusion

  • The empirical analysis finds most training examples valuable in CIFAR-10, CIFAR-100, and ImageNet, but largely redundant in MNIST.The authors conclude that reducing data while maintaining performance is difficult for the three less-redundant datasets.Source: §6 · p. 9
  • The analysis results are specific to the deep-learning models used, rather than direct properties of the datasets, although evidence suggests some importance findings generalize between models.The authors report that some images appear important independent of the learning model.Source: §6 · p. 9
  • The authors propose exploring batch-evaluated importance metrics, which could encourage gradient orthogonality and class-label entropy but may require heuristics because subset optimization can be exponentially complex.They note that batch-based ideas have been used in active learning.Source: §6 · p. 9

Supplementary Material

Supplementary examples show that images with large gradient magnitudes can be correlated by pixel values or semantic category. CIFAR-10 figures illustrate these patterns across model architectures, while class-label entropy provides another indicator of category skew.

Supplementary Material

  • Large-gradient CIFAR-10 images show pixel-level white-background repetition for the linear model and recurring distant birds for the vanilla CNN, illustrating correlated examples.The figure organizes the selected images by architecture.Source: Figure 10 · p. 11, Supplementary Material · p. 11
  • Gradient correlations can occur at both semantic-category and pixel levels, with some categories or colors appearing more often among high-gradient images.These observations are presented as additional evidence supporting Gradient-CDF.Source: Supplementary Material · p. 11
  • Top-gradient CIFAR-10 samples show recurring cats and dogs across models, while low entropy at small CNN subsamples indicates that some classes contribute more high-gradient images.The entropy figure compares top-gradient label entropy with the uniformly distributed-label baseline.Source: Supplementary Material · p. 11, Figure 11 · p. 11, Figure 12 · p. 12
Loading 1811.12569v1…