PaperlayerView exact revision on arXiv

Source-linked AI summary

Selection via Proxy: Efficient Data Selection for Deep Learning

Cody Coleman, Christopher Yeh, Stephen Mussmann, Baharan Mirzasoleiman, Peter Bailis, Percy Liang, Jure Leskovec, Matei Zaharia

arXiv:1906.11829v4cs.LGstat.ML

TL;DR

Deep-learning data selection is costly because it requires learned representations or trained models. Selection via proxy (SVP) uses smaller or shorter-trained models for selection while retaining the target model for final training. Across five datasets and active-learning and core-set tasks, SVP substantially reduced selection cost while generally preserving downstream performance, including a 1.6× end-to-end speed-up for CIFAR10 core-set selection.

  • Problem

    Data selection is prohibitively expensive in deep learning because active learning and core-set methods depend on representations learned by costly models.

  • Method

    SVP replaces the target model’s selection representation with one from a smaller proxy created by reducing layers, architecture size, or training epochs.

  • Results

    Across five datasets, SVP matched traditional active-learning accuracy with up to 41.9× faster selection runtime and achieved a 1.6× CIFAR10 core-set end-to-end speed-up.

  • Takeaways & Limitations

    Small, less accurate models can provide useful signals for selecting data while the larger target model remains responsible for final prediction performance.

Abstract

from arXiv · show

Data selection methods, such as active learning and core-set selection, are useful tools for machine learning on large datasets. However, they can be prohibitively expensive to apply in deep learning because they depend on feature representations that need to be learned. In this work, we show that we can greatly improve the computational efficiency by using a small proxy model to perform data selection (e.g., selecting data points to label for active learning). By removing hidden layers from the target model, using smaller architectures, and training for fewer epochs, we create proxies that are an order of magnitude faster to train. Although these small proxy models have higher error rates, we find that they empirically provide useful signals for data selection. We evaluate this "selection via proxy" (SVP) approach on several data selection tasks across five datasets: CIFAR10, CIFAR100, ImageNet, Amazon Review Polarity, and Amazon Review Full. For active learning, applying SVP can give an order of magnitude improvement in data selection runtime (i.e., the time it takes to repeatedly train and select points) without significantly increasing the final error (often within 0.1%). For core-set selection on CIFAR10, proxies that are over 10x faster to train than their larger, more accurate targets can remove up to 50% of the data without harming the final accuracy of the target, leading to a 1.6x end-to-end training time improvement.

1 INTRODUCTION

Deep-learning data selection is expensive because active learning and core-set methods require learned representations or trained models. Selection via proxy (SVP) replaces expensive target representations with smaller proxies, preserving useful selection signals across multiple datasets and tasks.

  • Motivation: Data selection methods identify informative examples but require semantic features or trained models to quantify informativeness.Active learning repeatedly trains on labeled data and selects additional examples using uncertainty or related heuristics.
  • Motivation: Deep-learning data selection is prohibitively expensive because models learn complex hidden representations through substantial training.Many methods need representations for diversity or uncertainty estimation before selecting informative points.
  • Selection via proxy: SVP uses a less computationally intensive proxy model’s representation instead of the larger target model’s representation during selection.Proxies can be created by removing layers, using smaller architectures, or training for fewer epochs, despite their lower accuracy.
  • Evaluation: SVP was evaluated for active learning and core-set selection across CIFAR10, CIFAR100, ImageNet, Amazon Review Polarity, and Amazon Review Full.The evaluation covered multiple selection methods, including uncertainty sampling, greedy k-centers, and forgetting events.
  • Results: Across datasets, SVP matched the accuracy of traditional selection while reducing data-selection runtime by up to 41.9×, depending on the proxy.The reported speed-ups reached 7× on CIFAR10 and CIFAR100, 41.9× on Amazon reviews, and 2.9× on ImageNet.

2 METHODS

The methods replace target models with cheaper proxies while retaining the original selection procedures and final target-model evaluation. Proxies are built by reducing architecture size or training duration, and selection quality is assessed through downstream target performance.

  • 2.1 ACTIVE LEARNING: Pool-based active learning begins with unlabeled data and iteratively selects points to label until reaching a labeling budget.The objective is to obtain a model with low error from an initial labeled subset plus selected examples.
  • 2.1 ACTIVE LEARNING: The active-learning baseline retrains a target model on all previously labeled data before selecting the next batch.The target is reinitialized each round, and final performance is measured on a held-out test set.
  • 2.1 ACTIVE LEARNING: Active learning selects examples using lowest model confidence or distances between final-hidden-layer representations with greedy k-centers.Greedy k-centers adds points that maximize distance from the existing selected pool.
  • 2.2 CORE-SET SELECTION: For core-set selection, the goal is to find a subset whose trained-model quality is comparable to that of the full dataset.The selected subset is evaluated by training the target model on it and measuring held-out test error.
  • 2.2 CORE-SET SELECTION: Core-set baselines select points using greedy k-centers, forgetting-event counts, or highest predictive entropy.Forgetting events count transitions from correct to incorrect classification during training; the highest-scoring points are retained.
  • 2.3 APPLYING SELECTION VIA PROXY: SVP substitutes proxies for target models during selection but trains the same final target model after active-learning selection or on the selected core set.This preserves the downstream target-model evaluation while reducing the cost of computing selection metrics.
  • 2.3 APPLYING SELECTION VIA PROXY: Proxies are created by scaling down model dimensions or hidden layers and by training for fewer epochs.The design trades accuracy for training speed while seeking a useful approximation of the target decision boundary.
  • 2.3 APPLYING SELECTION VIA PROXY: A ResNet20 proxy reaches 7.6% CIFAR10 top-1 error in 26 minutes, while ResNet164 takes 4 hours and lowers error by 2.5%.This illustrates the accuracy–complexity trade-off used to construct proxies.

3 RESULTS

Across five datasets and multiple data-selection methods, SVP used smaller proxy models to achieve comparable selection quality with substantially lower runtime. Its effectiveness varied by dataset and task, but ranking correlations and core-set results support proxies as practical substitutes for larger target models.

  • Active learning: SVP achieved similar or higher active-learning accuracy while speeding data selection by up to 41.9×.The runtime measures repeated model training and point selection.
  • Active learning: SVP improved active-learning efficiency across CIFAR, ImageNet, and Amazon Review settings, but gains depended on the selection method and dataset.Least-confidence speed-ups reached 7× on CIFAR10/CIFAR100, 1.6× on ImageNet, and 41.9× on Amazon Review Polarity.
  • Dataset dependence: Amazon Review Full was a boundary case where neither baseline selection nor SVP outperformed random sampling.Greedy k-centers was also too slow on ImageNet and both Amazon datasets because of quadratic selection complexity.
  • Core-set selection: On CIFAR10 and CIFAR100, SVP matched or exceeded an oracle that used ResNet164 for core-set selection.The oracle trained the target model on all data before selecting examples.
  • Core-set selection: 50% data removal on CIFAR10 produced no significant ResNet164 error increase with ResNet20 as the proxy, yielding a 1.6× end-to-end speed-up.The full process took 2 hours and 20 minutes; early stopping with ResNet56 reached 1.8× speed-up at 5.4% versus 5.1% top-1 error.
  • Ranking correlation: Selection rankings remained positively correlated across model depths and architectures, supporting small models as proxies for larger models.MobileNet V2 entropy rankings were highly correlated with ResNet50 despite 3.5M versus 25.6M parameters.

4 RELATED WORK

Prior active-learning methods sometimes use one model to select data for a different, more expensive target, while core-set methods generally assume ready-to-use features. Related work also reports that selected examples or forgetting events can transfer across models.

  • Active learning: Heterogeneous uncertainty sampling used an inexpensive Naïve Bayes classifier to select labels for a more expensive decision-tree target.
  • Core-set selection: Core-set methods generally require ready-to-use features and therefore do not directly apply to deep neural networks without first learning representations.
  • Core-set selection: Forgetting events identified that 30% of CIFAR10 examples were rarely misclassified after learning and could be omitted without affecting generalization.

5 CONCLUSION

The conclusion reports that SVP substantially reduces data-selection and end-to-end training costs while preserving error or accuracy across active learning and core-set selection settings.

  • 41.9× and 3.8× runtime improvements were achieved for least-confidence sampling and greedy k-centers, respectively, with no significant increase in error.
  • Up to 50% of CIFAR10 data could be removed through core-set selection in 10× less time than training the target model.
  • Core-set selection produced a 1.6× end-to-end training speed-up without harming the target’s final accuracy.

A.1 CHOICE OF DATASETS

The evaluation focuses on classification, covering image and text datasets to compare SVP across task types and error regimes. The selected datasets span coarse- and fine-grained image classification and binary and five-class review classification.

  • Classification was chosen because it is widely studied in active learning, while SVP’s simplicity may support broader applications in future work.
  • The experiments covered CIFAR10, CIFAR100, ImageNet, Amazon Review Polarity, and Amazon Review Full.

A.2 IMPLEMENTATION DETAILS

Implementation uses distinct target and proxy architectures across image and text datasets, with ImageNet experiments employing optimized multi-GPU training. A table reports parameter counts for the models.

  • CIFAR10 and CIFAR100: CIFAR10 and CIFAR100 use ResNet164 as the target, while smaller pre-activation ResNet proxies use convolutional residual units and achieve lower accuracy.
  • ImageNet: ImageNet uses ResNet50 as the target and ResNet18 as the proxy, trained with four Nvidia Titan V GPUs, warm-up, scaled learning rates, and mixed precision.
  • Table 2 reports the number of parameters in each model.
  • Amazon Reviews: Amazon Review experiments use VDCNN29 as the target and fastText and VDCNN9 as proxies.

A.3 MOTIVATION FOR CREATING PROXIES

Figures 4 and 5 motivate proxy models by showing that larger models and extended training require substantially more time for relatively small error reductions on CIFAR10 and CIFAR100.

  • CIFAR10 model-size and training curves show substantial time spent achieving small changes in top-1 test error.
  • CIFAR100 model-size and training curves likewise show substantial time spent achieving small changes in top-1 test error.

A.4 ADDITIONAL ACTIVE LEARNING RESULTS

Additional active-learning results show that smaller proxies can select data nearly as effectively as target models while reducing selection cost across datasets and settings.

  • Across five datasets, smaller proxies had higher top-1 error but selected points nearly as effectively as target models without harming final target performance.
  • Table 3 reports average top-1 error and data-selection speed-ups for active learning with varying proxies.
  • Table 4 reports active-learning top-1 error and speed-ups when proxy models are trained for fewer epochs.

A.5 ADDITIONAL CORE-SET SELECTION RESULTS

Additional core-set results compare proxy-based selection across datasets, methods, subset sizes, and runtimes, emphasizing accuracy alongside computational cost.

  • On CIFAR100, SVP performed as well as an oracle baseline while being significantly faster across selection methods and subset sizes.
  • Table 5 reports average top-1 error for core-set selection on ImageNet, Amazon Review Polarity, and Amazon Review Full.
  • Table 6 reports average top-1 error and runtime for CIFAR10 core-set selection with varying proxies, methods, and subset sizes.
  • Table 7 reports average top-1 error and runtime for selection methods calculated from ResNet20 models.

A.6 ADDITIONAL CORRELATION RESULTS

Correlation analyses compare selection rankings across architectures, runs, training epochs, and representations on CIFAR datasets and ImageNet.

  • ImageNet entropy rankings were highly correlated across a wide range of pretrained model architectures.
  • CIFAR100 rankings showed similarly high correlation across model architectures as between runs of the same architecture, supporting small models as selection proxies.
  • Facility-location rankings across ResNet runs and depths were compared using Spearman correlation on CIFAR10 and CIFAR100.
  • Forgetting-event and entropy rankings were compared with ResNet164 rankings across varying ResNet architectures on CIFAR10 and CIFAR100.
  • Forgetting-event and entropy rankings were also compared across training epochs for varying ResNet architectures.
  • Pearson correlations were evaluated across ResNet runs and depths using forgetting events and final-model entropy, with similarly high off-diagonal and on-diagonal correlations.
  • t-SNE plots compare 30% CIFAR10 subsets selected by ResNet164 and ResNet20 using forgetting events, entropy, and greedy k-centers.
  • Forgetting-event rankings were more stable than entropy rankings when ResNet20 was trained for varying numbers of epochs.
Loading 1906.11829v4…