Source-linked AI summary
DeepCore: A Comprehensive Library for Coreset Selection in Deep Learning
Chengcheng Guo, Bo Zhao, Yanbing Bai
TL;DR
Coreset selection seeks informative subsets to reduce the memory and computation required by large-scale deep-learning training, but existing methods are often compared under inconsistent settings. DeepCore re-implements popular methods in a unified library and evaluates them on CIFAR10 and ImageNet; the experiments find that method advantages depend on the setting, while random selection remains a strong baseline.
Problem
Existing coreset methods are often evaluated with different datasets, architectures, coreset sizes, augmentations, and training strategies, making comparisons potentially unfair and conclusions unconvincing.
Method
DeepCore provides a modular PyTorch library re-implementing popular coreset selection methods and comparing them under unified settings on CIFAR10 and ImageNet.
Results
Across CIFAR10 and ImageNet, different methods perform best in particular settings, but random selection remains a strong and stable baseline.
Takeaways & Limitations
DeepCore enables convenient and fair comparison of coreset selection methods across varied deep-learning settings.
Takeaways & Limitations
Geometry-based methods assume that nearby feature-space points have similar properties, so their effectiveness depends on this representation-based assumption.
Abstract
from arXiv · showhide
Coreset selection, which aims to select a subset of the most informative training samples, is a long-standing learning problem that can benefit many downstream tasks such as data-efficient learning, continual learning, neural architecture search, active learning, etc. However, many existing coreset selection methods are not designed for deep learning, which may have high complexity and poor generalization performance. In addition, the recently proposed methods are evaluated on models, datasets, and settings of different complexities. To advance the research of coreset selection in deep learning, we contribute a comprehensive code library, namely DeepCore, and provide an empirical study on popular coreset selection methods on CIFAR10 and ImageNet datasets. Extensive experiments on CIFAR10 and ImageNet datasets verify that, although various methods have advantages in certain experiment settings, random selection is still a strong baseline.
1 Introduction
Coreset selection reduces the cost of training on large datasets by selecting a small informative subset intended to preserve generalization. DeepCore addresses limitations in existing methods and comparisons through a unified empirical framework for deep learning.
- Coreset selection chooses a small subset of informative samples from a large training set while aiming to retain similar generalization performance.
- Traditional coreset methods may be unsuitable for deep learning because of high computational complexity and fixed data representations.
- DeepCore re-implements 12 popular coreset selection methods in a unified PyTorch framework.
- The study compares methods across selection fractions from 0.1% to 90% on CIFAR10 and ImageNet-1K.
2 Review of Coreset Selection Methods
Coreset selection seeks a smaller training subset whose trained model has similar generalization to one trained on the full dataset. Geometry-based methods remove redundant points using feature-space structure, while uncertainty methods prioritize lower-confidence samples.
- 2.1 Problem Statement: Coreset selection finds a subset S ⊂ T with |S| < |T| so models trained on S and T have close generalization performance.
- Geometry Based Methods: Geometry-based methods assume nearby feature-space points have similar properties and remove redundant samples to form a much smaller coreset.
- Geometry Based Methods: Herding greedily adds samples that minimize the distance between coreset and full-dataset centers in feature space.
- Geometry Based Methods: k-Center Greedy selects k samples to minimize the largest distance from any unselected point to its nearest selected point, using a greedy approximation to an NP-hard problem.
- Geometry Based Methods: Uncertainty-based methods select samples in descending order of Least Confidence, Entropy, or Margin scores because lower-confidence samples may affect optimization more.
Uncertainty Based Methods
Error- and loss-based methods prioritize samples by their contribution to training difficulty or model loss, while importance sampling uses worst-case loss contribution as a selection score.
- Error/Loss Based Methods: Samples are considered important when they contribute strongly to training error or loss through their loss, gradient, or influence on predictions.
- Error/Loss Based Methods: Forgetting counts transitions where a previously correct sample becomes misclassified during training, identifying examples that can be removed with minimal performance drop.
- Error/Loss Based Methods: GraNd measures each sample's average contribution to training-loss decline during early epochs across independent runs.
- Error/Loss Based Methods: EL2N approximates GraNd using the norm of the error vector and therefore requires less computational cost when calculated early in training.
- Error/Loss Based Methods: Importance sampling assigns each point a sensitivity score representing its upper-bounded worst-case contribution to the total loss, then constructs the coreset from selection probabilities.
Decision Boundary Based Methods
Decision-boundary methods target difficult-to-separate samples. They approximate boundary proximity through adversarial perturbations or identify samples whose predictive likelihood differs most from neighboring samples.
- Decision Boundary Based Methods: Adversarial DeepFool approximates boundary distance by perturbing samples until their predictive labels change, selecting those requiring the smallest perturbations.
- Decision Boundary Based Methods: Contrastive Active Learning selects samples whose predictive likelihood diverges most from their neighbors to locate decision-boundary examples.
Gradient Matching Based Methods
Gradient matching methods select coresets whose weighted sample gradients approximate the full-dataset gradient, targeting bounded approximation error with efficient optimization.
- Weighted gradients from a selected subset can replace the full-dataset gradient sum while comparing gradients through a distance function.The subset weight vector w is evaluated using its l1 norm and gradient distance D(·, ·).
- Craig converts gradient matching into monotone submodular maximization and optimizes the resulting objective greedily.
- GradMatch achieves the same gradient-matching error ε as Craig with a smaller subset by regularizing sample weights.It uses squared l2 regularization and Orthogonal Matching Pursuit under a preset subset-size constraint.
Bilevel Optimization Based Methods
Bilevel methods optimize subset selection or selection weights outside model training, then optimize model parameters on the selected data inside. The surveyed methods apply this structure to supervised, semi-supervised, active-learning, and related selection settings.
- Bilevel coreset selection places subset or weight optimization in the outer objective and model-parameter optimization on the subset in the inner objective.
- Retrieve: Retrieve formulates semi-supervised selection with labeled data T and unlabeled data P, combining labeled-data and unlabeled-data losses.The regularization coefficient λ weights the unlabeled-data loss.
- Glister: Glister adds a validation set V to the outer optimization and uses log-likelihood ℓℓ in its bilevel formulation.
- Submodularity: Submodular functions assign real values to subsets and are used to measure diversity and information for coreset selection.Graph Cut, Facility Location, and Log Determinant are listed examples, with greedy maximization offering a bounded approximation under cardinality constraints.
- Fass: Fass connects likelihood maximization under cardinality constraints with submodular optimization for Naive Bayes and Nearest Neighbor classifiers.The resulting framework is applied to active learning.
- Prism: Prism selects a labeled subset from a large unlabeled pool while requiring alignment with a targeted set reflecting specific user intent.
- Similar: Similar extends submodularity to settings involving rare classes, redundancy, and out-of-distribution data.
Proxy Based Methods
Proxy-based methods reduce coreset-selection cost by training lighter versions of target models, while coresets support efficient learning and evaluation across several applications.
- Selection via Proxy: Selection via Proxy methods train lighter or shallower proxy models by reducing layers, dimensions, or training epochs.Coresets are then selected more efficiently using these proxy models.
- Data-efficient Learning: Coresets reduce training cost while preserving testing performance and can serve as proxy datasets in neural architecture search.This is useful when thousands to millions of deep models must be trained and evaluated on the same dataset.
- Other applications: Coreset selection is also applied to robust learning, clustering, semi-supervised and unsupervised learning, efficient GAN training, and regression.
3 DeepCore Library
DeepCore addresses inconsistent evaluations of deep-learning coreset methods by providing a shared library that reproduces methods and enables fair comparisons under matched settings.
- Different datasets, architectures, coreset sizes, augmentation, and training strategies make published comparisons potentially unfair and conclusions unconvincing.The variation also makes it difficult for future researchers to identify and improve the state of the art.
- DeepCore reimplements dozens of popular and advanced coreset methods in an extensive, extendable library for fair comparison in the same experimental settings.The library is organized across method categories and includes Random selection as a baseline.
4 Experiment Results
DeepCore evaluates coreset selection methods under unified CIFAR10 and ImageNet-1K settings, including fixed ResNet-18 training and cross-architecture tests. Results show method strengths depend on dataset, selection fraction, and hyperparameters, while random selection remains a strong baseline.
- Experimental setup: DeepCore evaluates methods with ResNet-18 on CIFAR10 and ImageNet-1K across multiple selection fractions.CIFAR10 experiments use fractions from 0.1% to 90%; the supplied experiment passages specify ResNet-18 as the default architecture.
- CIFAR10 results: Graph Cut achieves the best CIFAR10 results when selecting 0.1% to 10% of the training data.At 1% of the training set, Graph Cut outperforms other methods by more than 5% in testing accuracy when 50 samples are selected per class.
- Sensitivity and caveats: Reported CIFAR10 findings depend on hyper-parameter settings, including the feature-extraction model and whether coresets are fixed during training.The authors note that Herding may perform better with a fully trained feature-extraction model.
- ImageNet results: For ImageNet, Forgetting and GraNd generally perform better, but no method outperforms Random when 30% of the data is selected.Forgetting exceeds Random when fewer than 10% of the data are selected; the authors describe Random as a strong and stable baseline.
- Extended experiments: Cross-architecture experiments test whether coresets selected on one architecture generalize to VGG-16, Inception-v3, ResNet-18, and WideResNet-16-8.Forgetting, Glister, GraNd, and Graph Cut are evaluated at 1% and 10% selection fractions.
5 Extended Related Work
Dataset condensation synthesizes informative samples rather than selecting subsets, but applying it to large, high-resolution datasets such as ImageNet-1K remains challenging. The section also includes sensitivity analysis for pre-trained models with varying training durations.
- Dataset condensation: Dataset condensation learns synthetic informative samples instead of selecting subsets from the original training set.
- Pre-trained model sensitivity: Table 4 reports performance for different methods using pre-trained models with varying pre-training epochs.
- Dataset condensation: Applying dataset condensation to large, high-resolution datasets such as ImageNet-1K remains challenging because optimization is expensive and difficult.
6 Conclusion
DeepCore provides a unified library for comparing coreset selection methods in deep learning. Experiments on CIFAR10 and ImageNet show that methods can have setting-specific advantages, while random selection remains a strong baseline.
- Conclusion: DeepCore re-implements dozens of state-of-the-art coreset selection methods across popular datasets and network architectures.The library is intended to support convenient and fair comparisons across learning settings.
- Conclusion: Experiments on CIFAR10 and ImageNet show that different methods have advantages in certain settings, but random selection remains a strong baseline.