PaperlayerView exact revision on arXiv

Source-linked AI summary

One Loop, Two Gains: Can Active Learning win the Lottery for Free?

Benedikt Tscheschner, Eduardo Veas, Marc Masana

arXiv:2609.10311v1cs.LGcs.AIcs.CV

TL;DR

Deep active learning and iterative magnitude pruning both repeatedly retrain models, but their shared computational structure has been underused. The paper introduces Improve & Prune, which inserts pruning into each active-learning cycle. Across the evaluated image-classification settings, the resulting sparse models match dense accuracy while reducing training and acquisition computation.

  • Problem

    Deep active learning and iterative magnitude pruning separately incur repeated retraining costs despite sharing an iterative train-and-prune structure.

  • Method

    Improve & Prune integrates magnitude pruning into each pool-based active-learning retraining cycle, dynamically adapting masks as labeled data changes.

  • Results

    Across four image-classification datasets, six acquisition functions, and CNN and Transformer architectures, I&P sparse models consistently match dense accuracy while reducing training and acquisition FLOPs.

  • Takeaways & Limitations

    I&P yields sparse models at each active-learning iteration that address computational bottlenecks in retraining and acquisition scoring without a separate ticket-search cost.

  • Takeaways & Limitations

    The evaluation is limited to image classification, and whether the observed ticket discovery generalizes to other domains remains open.

Abstract

from arXiv · show

The lottery ticket hypothesis posits the existence of winning tickets: sparse subnetworks that, when trained in isolation from their original initialization, match the accuracy of the full dense network. The predominant method for discovering such tickets, iterative magnitude pruning, alternates pruning with full retraining from scratch until convergence over many cycles. Similarly, deep active learning also retrains a model from scratch after each acquisition round as new labels become available. Despite this shared reliance on iterative retraining with a substantial computational overhead, the two paradigms have been studied separately. We observe that the iterative training loop inherent to pool-based active learning already provides the exact computational structure that iterative magnitude pruning exploits, and propose Improve & Prune (I&P), a method that integrates magnitude pruning into each active learning retraining cycle at practically no additional cost. This raises a key empirical question: can iterative magnitude pruning produce winning tickets under the non-stationary data regime of active learning? We investigate this question across multiple acquisition functions, architecture families, and image classification datasets, including an active fine-tuning scenario. Our results demonstrate that I&P yields sparse, deployable models at each active learning iteration. Those match the accuracy of their dense counterparts at sparsities up to 95%, effectively obtaining winning tickets as a byproduct of the active learning pipeline. These per-iteration sparse models can address two computational bottlenecks - per-round model retraining and acquisition scoring over the unlabeled pool - that currently prevent the practical adoption of DAL on large architectures and large unlabeled pools.

1 INTRODUCTION

Specialized domains face scarce expert annotations and deployment constraints, motivating methods that reduce both labeling and inference costs. The paper proposes Improve & Prune (I&P), which integrates pruning into active-learning retraining to discover sparse winning tickets and improve scalability.

  • Motivation: Deep active learning reduces annotation cost, while iterative magnitude pruning targets model-computation cost; both rely on repeated retraining.The paper studies their shared iterative structure rather than treating the paradigms independently.
  • Improve & Prune: I&P integrates magnitude pruning into each active-learning retraining cycle, using the existing loop to progressively discover winning tickets without additional training cost.The approach avoids warm-start bias by retaining active learning’s from-scratch retraining.
  • Motivation: Acquisition scoring over the entire unlabeled pool makes inference costs scale with pool size, while sparse models reduce per-pass FLOPs and embedding dimensions.This can improve scalability for distance-based acquisition functions on large pools.
  • Improve & Prune: I&P dynamically expands and contracts pruning masks because active learning trains on a non-stationary labeled set.The method is evaluated across six acquisition functions, four image-classification datasets, two backbones, and supervised or self-supervised pretraining.

2 BACKGROUND

The paper reviews pool-based active learning, the lottery ticket hypothesis, iterative magnitude pruning, and alternative sparse-training approaches. These methods differ in when they estimate parameter importance and in how they trade computational cost against sparse-model quality.

  • Deep Active Learning: Pool-based active learning repeatedly scores an available unlabeled pool, selects informative samples, obtains oracle labels, and retrains the model.The initial labeled set may be seeded randomly or through clustering pretrained embeddings, and iterations continue until a budget or target is reached.
  • Deep Active Learning: Acquisition functions include uncertainty-based, coreset, and hybrid uncertainty-diversity strategies, with no single function dominating across all settings.The paper therefore benchmarks six representative functions spanning the three families.
  • Lottery Ticket Hypothesis: The lottery ticket hypothesis states that a randomly initialized dense network can contain a subnetwork that matches the trained network’s test accuracy when trained from its original initialization.Winning tickets are sparse subnetworks whose performance remains comparable to the dense model under the stated training condition.
  • Iterative Magnitude Pruning: Iterative magnitude pruning repeatedly trains to convergence, removes low-magnitude weights, and retrains from the original initialization to identify a sparsity level before performance degrades.Its computational expense comes from repeated full training cycles, while rewinding can stabilize tickets in larger or deeper networks.
  • Alternative Sparse Methods: Initialization-time pruning methods estimate importance before training, but parameter importance at initialization is a poor proxy for importance after training.Dynamic sparse methods instead update topology during one training run, though their dense-gradient regrowth can reduce efficiency.

3 ONE LOOP, TWO GAINS

I&P fuses active learning and iterative pruning into one loop: sparse masks are expanded, trained, pruned, and finetuned before acquisition. This produces progressively sparse models while reducing the computational burden of both retraining and pool scoring.

  • Computational Motivation: Deep active learning incurs repeated from-scratch training and large-pool acquisition costs, so network sparsity targets two dominant scaling barriers.The paper does not address the task-, dataset-, and architecture-dependent cost of optimization strategy and training duration.
  • One Loop, Two Gains: I&P attaches one pruning step to each active-learning training phase, using the shared loop to discover winning tickets without a separate IMP loop.Fusing the loops yields progressively sparser models and label-efficient acquisition without extra winning-ticket search cost.
  • I&P Procedure: At each iteration, I&P reactivates part of the mask, trains to convergence, prunes low-ranked parameters, finetunes, and then scores the unlabeled pool.The mask retains at least the target active-parameter fraction while allowing previously pruned connections to return.
  • Training and Finetuning: Static sparsity during training supports sparse-kernel or hardware-level optimization, unlike dynamic topology changes that require dense-gradient regrowth.Finetuning is capped at a small budget and is intended to realign remaining parameters after pruning.
  • Acquisition: Sparse acquisition models reduce computation over the unlabeled pool, and distance-based acquisition can gain a quadratic speedup from lower latent dimensionality.I&P is agnostic to the acquisition function and uses unstructured magnitude pruning with random layer-wise reactivation in this work.
  • Evaluation: Table 1 evaluates final accuracy for a self-supervised ResNet-18 on CIFAR-100 and Imagewoof across low and high annotation budgets, with AULC non-inferiority tests against Dense.The table reports mean ± standard deviation over 10 seeds.

5 DISCUSSION AND LIMITATIONS

I&P’s efficiency claims span sparsity selection, active-learning synergies, and deployment constraints. The paper’s scope remains limited by image-classification evaluation and unstructured pruning, while hardware realization of FLOP savings is incomplete.

  • Practical realizability of computational savings: Theoretical FLOP reductions may not yield proportional GPU speedups because current dense kernels poorly exploit unstructured sparsity.The paper notes that 2:4 and block-sparse deployment could narrow this gap, while CPU and edge-device sparse libraries may realize the savings more directly.
  • Empirical scope and domain limitations: The evaluation is limited to image classification, so generalization to detection, language, and tabular data remains open.The lottery-ticket effect is empirical and is not guaranteed across architectures, datasets, or sparsity levels.
  • Empirical scope and domain limitations: Structured pruning may produce more hardware-friendly models but could require different reactivation strategies and sparsity schedules.The experiments focus on unstructured magnitude pruning.
  • Efficient selection of target sparsity: I&P enables post hoc sparsity search because candidate masks require only short finetuning after shared main training.The expensive main-training phase is shared across candidate sparsity levels.
  • Synergies between DAL and LTH: Distributional acquisition functions may facilitate ticket discovery because they produce training sets resembling easy-to-learn subsets.These functions are reported to perform strongly in initial active-learning iterations.

6 CONCLUSION

The paper presents Improve & Prune as an integration of active-learning retraining and iterative magnitude pruning. Across varied image-classification settings, it reports sparse models matching dense accuracy while reducing training and acquisition costs, with dynamic masks needed as data evolve.

  • Conclusion: I&P appends one pruning step to each active-learning iteration, uncovering winning tickets at nearly no additional training cost.The method requires no modification to the training procedure or acquisition function.
  • Conclusion: Across four datasets, six acquisition functions, and CNN and Transformer architectures, sparse models consistently match dense-model accuracy while reducing training and acquisition FLOPs.These results provide evidence that winning tickets can emerge under a non-stationary data regime.
  • Conclusion: A fixed IMP mask degrades as the labeled set grows, motivating mask expansion and contraction at each active-learning iteration.Dynamic sparsity keeps the mask aligned with the evolving data distribution.

A.1 METHOD ALGORITHM

Algorithm 1 combines mask reactivation, training on the current labeled pool, pruning with finetuning, and acquisition-based pool updates. It also contains reductions to standard DAL and IMP and an optional late-rewinding mechanism.

  • I&P procedure: I&P begins each iteration by reactivating previously pruned parameters and reloading the initialization under the relaxed mask.The reactivation rate is controlled by the function R.
  • I&P procedure: The model is trained from initialization on the current labeled pool until convergence, producing the iteration’s trained parameters.This preserves the active-learning loop’s retraining structure.
  • I&P procedure: Pruning zeros the lowest-magnitude parameters subject to the target active fraction, followed by short finetuning to compensate for pruning-induced function drift.The pruning rate is pc and the finetuning budget is at most φ epochs.
  • I&P procedure: The sparse finetuned model scores the unlabeled pool, queries the oracle, and updates the labeled and unlabeled partitions.The acquisition function is S.
  • I&P procedure: Optional rewinding replaces the initialization with an early training checkpoint, recovering late rewinding as a special case.This corresponds to the checkpoint θ^(0)_r described in the method.
  • Reductions to baseline protocols: Standard DAL is recovered by removing pruning, reactivation, finetuning, and rewind selection while keeping a dense mask.Standard IMP is recovered by fixing the labeled dataset and removing acquisition updates.

A.2 RESNET-18 BACKBONE ADAPTATIONS

The ResNet-18 adaptation preserves the backbone while modifying only its input stem for smaller resolutions. This avoids premature feature-map collapse and retains the shared 512-dimensional embedding.

  • Input-stem adaptation: For smaller inputs, the ResNet-18 stem uses reduced down-sampling to avoid prematurely collapsing feature maps.All residual blocks and the final 512-dimensional embedding remain unchanged.
  • Input-stem adaptation: Table 5 varies the input stem and initial pooling by resolution while keeping network topology and penultimate feature dimensionality at 512.Convolution padding is set to floor(K/2).

A.3 WINNING-TICKET SIGNIFICANCE TESTING

Winning-ticket status is evaluated by comparing dense and pruned models’ active-learning trajectories with a seed-paired non-inferiority test. The AULC metric captures performance across all labeling rounds, while the verdict depends on a deliberately chosen tolerance.

  • The test uses a one-sided paired Wilcoxon comparison of AULCDense − AULCmethod with non-inferiority margin ϵ = −0.005 AULC.The hypothesis requires the dense-minus-method gap to remain below the specified tolerance.
  • AULC summarizes the entire accuracy–labels learning trajectory across DAL iterations rather than only the final accuracy.Averaging over rounds makes the test sensitive to trajectory-level differences.
  • Rejection at p < 0.05 means significant underperformance relative to Dense cannot be established at seed-level variability, so the method is marked a winning ticket.The test is performed separately for each acquisition function and dataset without power corrections.
  • The non-inferiority verdict is sensitive to the chosen tolerance, with a few CIFAR-100 HB cases recovering non-inferiority when ϵ increases from 0.005 to 0.01.The strict setting is reported to remain conservative.

A.4 HYPERPARAMETER SENSITIVITY

I&P remains stable across the examined pruning-rate, reactivation-rate, and target-sparsity settings. Accuracy begins to collapse only under extreme pruning that removes roughly 99.95% of the original parameters.

  • Per-round pruning rate, target sparsity, and reactivation rate jointly determine how sparsity evolves across DAL iterations.The target sparsity is expressed as the retained active-parameter fraction 1 − s.
  • I&P shows stable performance with limited sensitivity to the per-round pruning rate pc and reactivation rate pr.These hyperparameters were probed on CIFAR-100 with margin sampling using three seeds.
  • Roughly 99.95% of all original parameters must be removed before model accuracy starts to collapse.This collapse occurs only under extreme pruning conditions in the reported sweeps.
  • Table 7 evaluates final CIFAR-100 accuracy across retained active-parameter fractions and coupled reactivation budgets at fixed pc = 0.66.Bold marks I&P runs matching or exceeding Dense within one standard deviation.

A.5 COMPARISON TO PRUNING-AT-INITIALIZATION AND DYNAMIC SPARSE TRAINING

I&P compares favorably with pruning-at-initialization, dynamic sparse training, and one-shot pruning while preserving practical advantages from when and how its mask is obtained. The comparisons show that training-time mask timing affects both accuracy and deployability.

  • RigL surpasses Dense in the high-budget regime but requires dense backward passes for inactive-weight gradients and shifting masks that hinder structured hardware patterns.I&P’s mask stabilizes within a DAL iteration and is comparatively better aligned with N:M patterns.
  • SNIP, SynFlow, and GraSP moderately underperform Dense and I&P at both budgets while remaining above the reported random-sparsity baselines.Their initialization-based importance estimates cannot capture structures emerging during training.
  • One-shot pruning reaches accuracy parity with Dense and I&P at both budgets, but trains densely at every DAL iteration and therefore misses I&P’s sparse-kernel savings from the second round onward.The reported parity at 95% sparsity may not extend to higher sparsities, where iterative pruning has been reported to outperform one-shot pruning.
Loading 2609.10311v1…