PaperlayerView exact revision on arXiv

Source-linked AI summary

Selection via Proxy: Efficient Data Selection for Deep Learning

Cody Coleman, Christopher Yeh, Stephen Mussmann, Baharan Mirzasoleiman, Peter Bailis, Percy Liang, Jure Leskovec, Matei Zaharia

arXiv:1906.11829v4cs.LGstat.ML

TL;DR

Data selection for deep learning can be prohibitively expensive because many methods rely on learned feature representations. The paper uses a smaller proxy model’s representation to select data in place of the target model’s. Across active learning and core-set selection, it reports faster selection without significant error increases, including up to 41.9× faster active-learning runtime.

  • Problem

    Deep-learning data selection can be prohibitively expensive because many methods need learned feature representations to identify informative points.

    Source: §1 · p. 1

  • Method

    SVP substitutes a cheaper proxy model’s feature representation for the target model’s during data selection.

    Source: §1 · p. 2

  • Results

    Across active learning and core-set selection, SVP improved runtime without significant error increases; active-learning runtime improved up to 41.9×.

    Source: §5 · p. 8

  • Takeaways & Limitations

    In the evaluated settings, proxy representations provided a viable alternative to selecting with the target model’s representation.

    Source: §1 · p. 2

Abstract

from arXiv · show

Data selection methods, such as active learning and core-set selection, are useful tools for machine learning on large datasets. However, they can be prohibitively expensive to apply in deep learning because they depend on feature representations that need to be learned. In this work, we show that we can greatly improve the computational efficiency by using a small proxy model to perform data selection (e.g., selecting data points to label for active learning). By removing hidden layers from the target model, using smaller architectures, and training for fewer epochs, we create proxies that are an order of magnitude faster to train. Although these small proxy models have higher error rates, we find that they empirically provide useful signals for data selection. We evaluate this "selection via proxy" (SVP) approach on several data selection tasks across five datasets: CIFAR10, CIFAR100, ImageNet, Amazon Review Polarity, and Amazon Review Full. For active learning, applying SVP can give an order of magnitude improvement in data selection runtime (i.e., the time it takes to repeatedly train and select points) without significantly increasing the final error (often within 0.1%). For core-set selection on CIFAR10, proxies that are over 10x faster to train than their larger, more accurate targets can remove up to 50% of the data without harming the final accuracy of the target, leading to a 1.6x end-to-end training time improvement.

1 INTRODUCTION

Selection via proxy (SVP) uses a cheaper model’s learned representation to make deep-learning data selection more efficient. Across active-learning and core-set tasks, the paper reports faster selection with comparable accuracy and reduced-data training without accuracy loss.

1 INTRODUCTION

  • Active learning repeatedly trains on labeled data and selects uncertain examples to label, whereas core-set selection seeks a representative subset of a larger dataset (Lewis & Gale, 1994; Rosenberg et al., 2005; Settles, 2011; 2012; Har-Peled & Kushal, 2007; Tsang et al., 2005; Huggins et al., 2016; Campbell & Broderick, 2017; 2018; Sener & Savarese, 2018).The evaluations covered CIFAR10, CIFAR100, ImageNet, Amazon Review Polarity, and Amazon Review Full.Source: §1 · pp. 1–2
  • Deep-learning selection is costly because active learning and core-set methods need learned features, while batch active learning still retrains a full model for each batch (Shen et al., 2017; Sener & Savarese, 2018; Kirsch et al., 2019).Prior core-set applications also used hand-engineered features and simple models pretrained on auxiliary tasks (Wei et al., 2013; 2014; Tschiatschek et al., 2014; Ni et al., 2015).Source: §1 · pp. 1–2
  • By removing layers, shrinking architectures, and training fewer epochs, SVP proxies often train 10× faster yet retain useful ranking signals, with high Spearman’s and Pearson’s correlations for uncertainty, forgetting events, and greedy k-centers.SVP substitutes proxy representations for those of the larger, more accurate target model.Source: §1 · p. 2
  • SVP matched the traditional large-model selection approach’s accuracy, with up to 7× speed-up on CIFAR10 and CIFAR100, 41.9× on Amazon Review Polarity and Full, and 2.9× on ImageNet.These speed-ups measure data-selection runtime: repeatedly training and selecting points.Source: §1 · p. 2
  • On CIFAR10, forgetting-events selection removed 50% of the data without impacting ResNet164 with pre-activation accuracy, yielding about 1.6× end-to-end training-time improvement with a 10× faster proxy.The end-to-end time includes training and using the proxy.Source: §1 · p. 2

2 METHODS

Section 2 presents selection via proxy (SVP) as replacing selection-time target models with cheaper proxies while preserving target-model training for the final active-learning model. It describes the active-learning and core-set procedures and two ways to make proxies cheaper.

2.1 ACTIVE LEARNING

  • Active learning selects budgeted batches by least-confidence sampling (Settles, 2012; Shen et al., 2017) or greedy k-centers (Sener & Savarese, 2018), retraining a reinitialized target each round.The final target model is trained on the selected points and evaluated on a held-out set.Source: §2.1 · pp. 2–3

2.2 CORE-SET SELECTION

  • Core-set selection finds budgeted subsets via greedy k-centers (Sener & Savarese, 2018), forgetting events (Toneva et al., 2019), or entropy (Lewis & Gale, 1994; Settles, 2012).Forgetting events count reversals from correct to incorrect classification; the selected-subset target model is evaluated against a target trained on the full dataset using held-out test error.Source: §2.2 · p. 4

2.3 APPLYING SELECTION VIA PROXY

  • For active learning, SVP replaces the model trained at each selection round with a proxy, then trains the target model on the final labeled budget.For core-set selection, SVP uses a proxy instead of the target to compute selection metrics and choose the subset.Source: §2.3 · p. 4
  • Proxies are made cheaper by scaling down architectures or training for fewer epochs, trading accuracy for faster approximations of the target decision boundary.On CIFAR10, ResNet20 has 7.6% top-1 error in 26 minutes, while ResNet164 takes 4 hours and reduces error by 2.5%; 12 of ResNet20's 26 minutes are spent on a 1.4% test-error improvement.Source: §2.3 · p. 4

3 RESULTS

Across active learning and core-set selection, small proxy models generally preserved target-model performance while making data selection faster. Results varied by dataset and method, with especially strong end-to-end gains on CIFAR10.

3.1 EXPERIMENTAL SETUP

  • Experiments covered three image and two text classification datasets, using smaller proxy architectures than the target models.Targets were ResNet164 for CIFAR10/100, ResNet50 for ImageNet, and VDCNN29 for both Amazon datasets; proxies included smaller ResNets, ResNet18, fastText, and VDCNN9.Source: §3.1 · p. 5

3.2 ACTIVE LEARNING

  • On CIFAR10/100, SVP sped up least-confidence and greedy k-centers selection by up to 7× and 3.8×, respectively, without harming data efficiency.On ImageNet, least-confidence selection was up to 1.6× faster; training for fewer epochs raised this to 2.9× without increasing ResNet50 error, while greedy k-centers was too slow.Source: §3.2 · p. 5, Table 1 · p. 6
  • On Amazon Review Polarity, a fastText proxy yielded up to 41.9× faster selection and up to 14% relative error reduction over random sampling at large budgets.On Amazon Review Full, neither baseline nor SVP outperformed random sampling; greedy k-centers was too slow on both Amazon datasets.Source: Table 1 · p. 6, §3.2 · p. 6

3.3 CORE-SET SELECTION

  • On ImageNet, neither SVP nor the baseline removed a significant fraction without increasing ResNet50 error; on Amazon Review Polarity, fastText removed 20% with only a 0.1% top-1 error increase.FastText trained in under 10 minutes versus 16 hours and 40 minutes for VDCNN29; on Amazon Review Full, both approaches failed to outperform random sampling. Greedy k-centers was infeasible on ImageNet and Amazon, and fastText could not compute forgetting events.Source: §3.3 · pp. 6–7

3 RESULTS

  • Across active learning and core-set selection, SVP generally preserved target performance while reducing selection cost; on CIFAR10, it removed up to 50% of data with a 1.6× end-to-end speed-up.For core-set selection, SVP performed as well as or better than the oracle on CIFAR10 and CIFAR100; the CIFAR10 result used ResNet20 with forgetting events.Source: §3 · p. 5, Table 1 · p. 6, §3.3 · p. 6

3.3 CORE-SET SELECTION

  • On CIFAR10, ResNet20 with forgetting events removed 50% of the data without significantly increasing error, and the full selection-and-training process was 1.6× faster.Early-stopped ResNet56 achieved a 1.8× end-to-end speed-up when removing 50%, with top-1 error of 5.4% versus 5.1% for ResNet164. On CIFAR100, partial training was less effective for large subsets but improved accuracy on small 30% subsets with forgetting events.Source: §3.3 · p. 6, Figure 2 · p. 7

3.4 RANKING CORRELATION BETWEEN MODELS

  • On CIFAR10 and CIFAR100, rankings from models of varying depths had positive correlations comparable to correlations across runs of the same model, supporting smaller models as proxies.This pattern held for forgetting events, entropy, and greedy k-centers; Pearson correlations for forgetting events and entropy were similarly positive.Source: Figure 3 · p. 7, §3.4 · p. 7
  • Entropy rankings from MobileNet V2 were highly correlated with ResNet50 rankings, on par with ResNet18, despite MobileNet V2 having 3.5M versus 25.6M parameters.High correlations also appeared across pretrained ImageNet architectures and between fastText and VDCNN, indicating SVP may apply across different architectures.Source: §3.4 · p. 8

4 RELATED WORK

Prior work used one model to select data for another or relied on ready-to-use features, while this work applies smaller, faster-to-train proxy models to selection for deep learning. The related work connects this approach to active learning, core-set selection, and transferable example-forgetting measures.

Active learning

  • Proxy models generalize cross-model active learning to deep learning, reducing runtime for uncertainty- and representativeness-based selection.Earlier examples include heterogeneous uncertainty sampling and committee-based selection with reusable points; the paper extends the idea using smaller models or fewer training epochs.Source: §4 · p. 8

Core-set selection

  • Unlike core-set and summarization methods that require learned or engineered features, faster proxies provide effective representations for selecting deep-learning core-sets.Prior examples include core-set selection for k-means and SVMs and submodular summarization (Wei et al., 2013; 2014).Source: §4 · p. 8

Example forgetting

  • Proxy models closely match larger models’ rankings on entropy, greedy k-centers, and example forgetting, extending evidence that forgetting events transfer across models.Toneva et al. (2019) reported that 30% of CIFAR10 examples could be omitted as rarely misclassified after learning, and tested transfer of forgetting events from ResNet18 to WideResNet (Zagoruyko & Komodakis, 2016).Source: §4 · p. 8

5 CONCLUSION

Selection via proxy (SVP) uses a cheaper model’s representation during data selection to improve computational efficiency, with reported benefits for active learning and core-set selection.

5 CONCLUSION

  • SVP improved runtime up to 41.9× for least-confidence sampling and 3.8× for Sener & Savarese (2018)’s greedy k-centers, with no significant error increase (often within 0.1%).Source: §5 · p. 8
  • For CIFAR10 core-set selection, SVP removed up to 50% of the data in 10× less time than target-model training, achieving a 1.6× end-to-end training speed-up.Source: §5 · p. 8
  • SVP improves selection efficiency by substituting a cheaper proxy model’s representation for an expensive target model’s during data selection.Source: §5 · p. 8

A APPENDIX · A.1 CHOICE OF DATASETS

Appendix A.1 explains the classification focus and dataset choices used to evaluate SVP, while noting that its core ideas may extend beyond classification in future work.

A.1 CHOICE OF DATASETS

  • The evaluation focuses on classification to compare SVP with broader active-learning research, since many uncertainty methods assume a single categorical probability distribution per example.The authors leave machine translation and named entity recognition to future work.Source: §A.1 · p. 13
  • Although SVP is evaluated on classification rather than machine translation or named entity recognition, its simplicity may support applying the paper’s core ideas more broadly in future work.Source: §A.1 · p. 13
  • Experiments span image classification on CIFAR10, CIFAR100, and ImageNet, and text classification on Amazon Review Polarity and Full.Source: §A.1 · p. 13
  • CIFAR10 and Amazon Review Polarity represent low-error settings, while CIFAR100 and Amazon Review Full represent high-error settings to test SVP across the error and data-efficiency curve.Source: §A.1 · p. 13
  • Amazon Review Polarity and Full provide text classification at larger scale, with 3.6M and 3M examples, respectively.Source: §A.1 · p. 13
  • ImageNet enables study of scale in example count, number of classes, and input dimension; the authors report presenting full ImageNet active-learning results for the first time.Source: §A.1 · p. 13

A.2 IMPLEMENTATION DETAILS

Implementations used dataset-specific target and proxy architectures, with training procedures, GPU allocations, and batch-size choices varying across image and review benchmarks.

A.2 IMPLEMENTATION DETAILS

  • CIFAR10/100 used pre-activation ResNet164 targets and smaller pre-activation ResNet proxies, following He et al. (2016b) except weight decay was 0.0005.The proxies used pairs of 3 × 3 convolutional layers as residual units. Core-set and active-learning experiments used P100 and Titan V GPUs, respectively.Source: §A.2 · p. 13
  • ImageNet used ResNet50 targets and ResNet18 proxies; active learning fixed both at batch size 768, while separate batch sizes would yield speed-ups closer to 2.7×.Training used four Titan V GPUs, a five-epoch warm-up, and mixed precision; the shared batch size underutilized the ResNet18 GPU.Source: §A.2 · p. 13
  • Amazon Review Polarity and Full used VDCNN29 targets with VDCNN9 and fastText proxies, following the cited VDCNN and fastText training procedures.GPU allocation varied by dataset and task: Polarity core-set runs used a P100, while the other listed review experiments used a Titan V.Source: §A.2 · p. 14

A.3 MOTIVATION FOR CREATING PROXIES

On CIFAR10 and CIFAR100, test-error comparisons across model sizes and over training show that substantial time is spent on only small accuracy changes.

A.3 MOTIVATION FOR CREATING PROXIES

  • Across CIFAR10 and CIFAR100, the figures compare top-1 test error for varying model sizes and show that substantial training time yields only small accuracy changes.The model-size comparisons appear on the left; training trajectories for a single model appear on the right.Source: Figure 4 · p. 14, Figure 5 · p. 15

A.4 ADDITIONAL ACTIVE LEARNING RESULTS

SVP makes active-learning data selection faster across five datasets without significantly increasing final target error. Smaller proxies have higher error themselves, but can select points nearly as good as target models; training them for fewer epochs helps over random sampling but trails the full schedule.

A.4 ADDITIONAL ACTIVE LEARNING RESULTS

  • Across five datasets, SVP sped up active-learning selection without significantly increasing target error, although higher-error proxies selected points nearly as good as target models.The comparison covered varying proxies, selection methods, and labeling budgets.Source: Figure 6 · p. 16, Table 3 · p. 17
  • Training proxies for fewer epochs improved active learning over random sampling, but performed less well than training for the full schedule.The comparison covered CIFAR10, CIFAR100, and ImageNet.Source: Table 4 · p. 18

A.5 ADDITIONAL CORE-SET SELECTION RESULTS

On CIFAR100 core-set selection, SVP matched the oracle baseline while running significantly faster; the additional tables cover core-set results across image and review datasets, proxies, methods, subset sizes, and training durations.

A.5 ADDITIONAL CORE-SET SELECTION RESULTS

  • SVP performed as well as an oracle baseline but significantly faster across datasets and selection methods; Figure 7 reports ResNet164 top-1 error over five CIFAR100 runs.The figure compares forgetting events, entropy, and greedy k-centers, with subsets selected using a proxy trained on the entire dataset.Source: Figure 7 · p. 19
  • Table 5 reports average top-1 error for core-set selection methods on ImageNet, Amazon Review Polarity, and Amazon Review Full.The caption does not provide the table's cell values or identify a winning method.Source: Table 5 · p. 19
  • Table 6 covers core-set selection on CIFAR10 and CIFAR100, varying proxies, selection methods, and subset sizes, with top-1 error and runtime.Results are averages over five runs.Source: Table 6 · p. 20
  • Table 7 reports top-1 error and runtime for selection methods using ResNet20 models trained for varying numbers of epochs on CIFAR10 and CIFAR100.The caption specifies averages over five runs but gives no cell values.Source: Table 7 · p. 20

A.6 ADDITIONAL CORRELATION RESULTS

Across ImageNet and CIFAR datasets, rankings from different model architectures are often highly correlated, supporting small models as proxies for data selection. The supplementary results also show that ranking variation can reflect training stochasticity, while forgetting-based rankings are more stable across training epochs than entropy-based rankings.

A.6 ADDITIONAL CORRELATION RESULTS

  • On ImageNet, MobileNet V2 entropy rankings were highly correlated with ResNet50 despite having 3.5M versus 25.6M parameters, and correlations were high across architectures.The authors suggest these cross-architecture correlations, alongside their fastText and VDCNN results, indicate that SVP may be widely applicable.Source: Figure 8 · p. 21
  • On CIFAR100, cross-architecture correlations for forgetting, entropy, and greedy k-centers were similarly high to correlations between runs of the same architecture, suggesting small models can proxy for data selection.The comparison used average Spearman correlations across ResNet models of varying depths.Source: Figure 9 · p. 22
  • With the same initial subset of 1,000 examples, facility-location ranking variation was mostly attributable to training stochasticity rather than the initial subset.The CIFAR10 and CIFAR100 correlations averaged 20 run-pair comparisons for ResNets with varying layer counts.Source: Figure 10 · p. 22
  • Supplementary Spearman plots compare varying ResNet architectures against ResNet164 across five runs, and compare rankings across epochs, for forgetting and entropy on CIFAR10 and CIFAR100.Figures 11–14 report average correlations with ±1 standard deviation.Source: Figure 11 · p. 22, Figure 12 · p. 23, Figure 13 · p. 23
  • Pearson correlations on CIFAR10 and CIFAR100 were generally similarly high across architectures and between runs of the same architecture, further supporting small models as data-selection proxies.The comparisons used forgetting-event counts and final-model entropy across 20 pairs of runs.Source: Figure 15 · p. 24
  • Across varying ResNet20 training durations, forgetting-event rankings were much more stable than entropy rankings because forgetting averages uncertainty throughout training, whereas entropy uses a final-model snapshot.The figure visualizes selected 30% subsets on CIFAR10 using a fully trained ResNet164 representation.Source: Figure 17 · p. 25

Related papers

Loading 1906.11829v4…