Semantic Redundancies in Image-Classification Datasets: The 10% You Don't Need

Contributions

• At least 10% of ImageNet and CIFAR-10 can be removed without affecting test accuracy when models are trained from scratch on the remaining data.The authors present this as a counterexample to the claim that these datasets contain no redundancy.Source: §1 · p. 1

Motivation

• The paper studies dataset properties through redundancy, extending analysis across ImageNet, CIFAR-10, and CIFAR-100.This focus addresses a comparatively underexplored aspect of datasets themselves, alongside prior work on imbalance, simulation, and prototypical examples.Source: §1 · p. 1

Contributions

• The contribution is to demonstrate significant redundancy rather than introduce a new algorithm, while motivating more efficient training and data collection.The authors also connect the finding to active learning by describing an upper bound on the best performance achievable from a subset.Source: §1 · p. 1

Data selection

• Prior example-selection methods prioritize training examples or accelerate convergence, whereas this work seeks subsets whose removal does not harm generalization.This distinction separates redundancy identification from selecting informative data during learning.Source: §2 · p. 2

Dataset reduction

• Earlier dataset-reduction methods iteratively remove small sets until performance declines, but their need to create many subsets makes them impractical for modern datasets.The cited approach is presented as an early attempt to reduce training-set size.Source: §2 · p. 2

Active learning

• Active-learning and subset-selection approaches formulate selection through constrained optimization or uncertainty, but simple uncertainty strategies can outperform one cited deep-network method.These methods target informative selection rather than explicitly locating redundant groups.Source: §2 · p. 2

Prototypicality

• Prototypicality-based work reports improved object-detection performance from prioritizing high-value examples and best CIFAR-10 performance using nearly-the-most prototypical 10% of training data.These studies emphasize informative examples, not removal of semantically redundant data.Source: §2 · p. 2

Redundancy analysis

• Vodrahalli et al. [Vodrahalli et al., 2018] use gradient magnitude from a pretrained classifier to prioritize examples, but fail to find redundancies in CIFAR-10 and ImageNet.The paper contrasts this with explicitly analyzing relationships among samples.Source: §2 · p. 2

Semi-supervised learning

• The authors suggest that retaining the most informative labeled examples could provide insights for semi-supervised learning evaluated on ImageNet and CIFAR datasets.The related-work discussion notes that commonly used subsets are randomly selected and labels may be removed for evaluation.Source: §2 · p. 2

3.1 Motivation

• The method addresses a limitation of gradient-based importance by comparing samples with an explicit dissimilarity measure, allowing near-duplicates to be identified as redundant.The motivation is that high gradient magnitudes may not distinguish between near-duplicate samples.Source: §3.1 · p. 2

3.2 Algorithm

• Samples are represented by penultimate-layer embeddings from a model trained on the full dataset, and agglomerative clustering groups nearby points in this semantic space.Agglomerative clustering repeatedly joins the closest pair of clusters according to the dissimilarity criterion.Source: §3.2 · p. 2
• Cluster dissimilarity is defined as the maximum pairwise dissimilarity between points in two clusters.This criterion determines which clusters are merged during agglomerative clustering.Source: §3.2 · p. 3
• Clustering is performed independently within each class until k clusters remain, after which the image closest to each cluster center is retained and the others are discarded.Here k equals the desired subset size, and each resulting cluster is treated as a redundant group.Source: §3.2 · p. 3

4 Experiments

• Experiments use ResNets, compare deterministic semantic-clustering subsets with uniformly sampled subsets, and report accuracy means with standard deviations across repeated trials.CIFAR experiments use 10 trials, while ImageNet validation accuracy is averaged over 5 trials.Source: §4 · p. 3, §4.1 · p. 3, §4.3 · p. 5
• Semantic clustering groups nearby same-class images, retains the image closest to each cluster center, and discards the remaining group members.The semantic representation is extracted from a model pretrained on the full dataset.Source: §4 · p. 3

4.1.1 CIFAR-10

• For CIFAR-10, redundant groups contain slight object variations, whereas most groups are singletons when constructing a 90% subset.The group-size distribution is shown for two classes on a logarithmic y-axis.Source: §4.1.1 · p. 3, Figure 4 · p. 4

4.1.2 CIFAR-100

• CIFAR-100 shows no useful redundancy: both semantic-clustering and random subsets lose test accuracy monotonically as subset size decreases.Its redundant groups contain substantially more semantic variation than CIFAR-10 groups.Source: §4.1.2 · pp. 3–4

4.2 Choice of semantic representation.

• The final average-pooling representation from the ResNet finds the largest redundancy among the three tested network stages.The compared stages are Start, Middle, and End.Source: §4.2 · p. 4, Figure 8 · p. 5

4.1.2 CIFAR-100

• For CIFAR-100, average dissimilarity within redundant groups is about an order of magnitude higher than for CIFAR-10, indicating greater semantic variation.The comparison averages class-wise means for three classes and the full dataset among clusters larger than one.Source: §4.1.2 · p. 4, Table 1 · p. 5

4.3 ImageNet

• ImageNet experiments train a 101-layer ResNet and find that most images remain singleton groups when constructing a 90% subset.Additional examples include semantically similar cars that are not close enough to be grouped as redundant.Source: §4.3 · pp. 5–6

4.1.1 CIFAR-10

• 10% of CIFAR-10 and ImageNet training data can be removed by semantic clustering without reducing test or validation accuracy, unlike random subsets.The ImageNet result uses average validation accuracy because its test set is unavailable.Source: §4.1.1 · p. 3, §4.3 · p. 5, Figure 9 · p. 5

• At least 10% of ImageNet and CIFAR-10 are redundant according to semantic clustering.Source: §5 · p. 6
• The semantic changes treated as redundant differ across groups, rather than reflecting one uniform visual variation.Source: §5 · p. 6
• Redundant groups contain varying attributes, suggesting networks may learn invariance from similar variations elsewhere in the dataset.Source: §5 · p. 6
• Removing 5% or 10% of the data sometimes produces slightly higher accuracy than training on the full dataset.The authors suggest this may indicate that training redundancies hamper optimization.Source: §5 · p. 6
• The scheme fails to find redundancies in CIFAR-100, where redundant-group points are much more dispersed in semantic space than in CIFAR-10.Source: §5 · p. 6
• A better algorithm might still identify CIFAR-100 redundancies, leaving the dataset-specific failure as an open limitation.Source: §5 · p. 6

• Each row represents one redundant image group, with the leftmost image retained in the 90% subset.Source: A Appendix · p. 10