Source-linked AI summary
Drawing Early-Bird Tickets: Towards More Efficient Training of Deep Networks
Haoran You, Chaojian Li, Pengfei Xu, Yonggan Fu, Yue Wang, Xiaohan Chen, Richard G. Baraniuk, Zhangyang Wang, Yingyan Celine Lin
TL;DR
The paper addresses the costly identification of winning tickets through full train-prune-retrain procedures. It identifies Early-Bird tickets early using low-cost training and mask distances, then trains only those tickets; across models and datasets, this achieves comparable or better accuracy with substantial savings.
Problem
Winning tickets can match dense-network accuracy, but identifying them requires costly iterative pruning and retraining.
Method
The paper identifies Early-Bird tickets during early, low-cost training using large learning rates, low precision, and a mask distance between consecutive pruned subnetworks.
Results
Across three pruning ratios, two DNN models, and two datasets, FP32 EB Train reduces energy by up to 1.2 ∼4.7× and FLOPs by up to 1.1 ∼4.5× while maintaining comparable or better accuracy.
Takeaways & Limitations
Early-Bird tickets support time- and energy-efficient training by enabling training to continue on only the early-identified sparse subnetworks.
Takeaways & Limitations
The hypothesis concerns a sparse mask m and an early iteration t satisfying t ≪ i, with comparable or better accuracy under the paper’s tested settings.
Abstract
from arXiv · showhide
(Frankle & Carbin, 2019) shows that there exist winning tickets (small but critical subnetworks) for dense, randomly initialized networks, that can be trained alone to achieve comparable accuracies to the latter in a similar number of iterations. However, the identification of these winning tickets still requires the costly train-prune-retrain process, limiting their practical benefits. In this paper, we discover for the first time that the winning tickets can be identified at the very early training stage, which we term as early-bird (EB) tickets, via low-cost training schemes (e.g., early stopping and low-precision training) at large learning rates. Our finding of EB tickets is consistent with recently reported observations that the key connectivity patterns of neural networks emerge early. Furthermore, we propose a mask distance metric that can be used to identify EB tickets with low computational overhead, without needing to know the true winning tickets that emerge after the full training. Finally, we leverage the existence of EB tickets and the proposed mask distance to develop efficient training methods, which are achieved by first identifying EB tickets via low-cost schemes, and then continuing to train merely the EB tickets towards the target accuracy. Experiments based on various deep networks and datasets validate: 1) the existence of EB tickets, and the effectiveness of mask distance in efficiently identifying them; and 2) that the proposed efficient training via EB tickets can achieve up to 4.7x energy savings while maintaining comparable or even better accuracy, demonstrating a promising and easily adopted method for tackling cost-prohibitive deep network training. Code available at https://github.com/RICE-EIC/Early-Bird-Tickets.
2 RELATED WORKS
Prior work established that pruning can support efficient inference and that subnetworks or connectivity patterns may emerge early during training. These findings motivate identifying useful subnetworks before full training.
- Winning-ticket research showed that small subnetworks in dense, randomly initialized networks can match the original network’s test accuracy when trained alone.
- Finding winning tickets remains costly because it depends on iterative pruning and retraining.
- Related work also explored single-shot pruning, supermasks, and model-compression methods including factorization, sharing, quantization, and dynamic inference.
- Early-training studies suggest networks learn important connectivity patterns before later tuning of weights, motivating early subnetwork identification.
3 DRAWING EARLY-BIRD TICKETS: HYPOTHESIS AND EXPERIMENTS
The paper hypothesizes that sparse, high-quality subnetworks can emerge early in training as Early-Bird tickets and investigates their existence, detection, and efficient use across models and datasets. Experiments show that mask distances can identify stabilized tickets, enabling substantial computational and energy savings while preserving accuracy.
- Hypothesis: Early-Bird tickets are sparse subnetworks identified at an early training iteration that can match or exceed the dense network’s test accuracy.The hypothesis requires t ≪ i and a sparse mask, with f′_acc ≈ f_acc or better.
- Experimental setup: Experiments evaluate early-drawn channel-pruned tickets across VGG16 and PreResNet101 on CIFAR-10 and CIFAR-100.The training protocol runs for 160 epochs with batch size 256 and learning-rate drops at epochs 80 and 120.
- EB-ticket observations: Early epoch ranges consistently produce tickets that outperform later tickets, including winning tickets drawn after 160 epochs.Some early tickets also outperform their unpruned, fully trained models, potentially because of sparse regularization.
- Mask-distance detection: Mask distances compare binary ticket masks across epochs, using batch-normalization scaling factors to identify channels for pruning at a target ratio.Channels with the smallest BN scaling factors are pruned, while retained and pruned channels are encoded as one and zero.
- Mask-distance detection: Mask distances change rapidly at the beginning of training and become nearly unchanged after approximately 80 epochs, indicating stabilized connectivity patterns.For VGG16 on CIFAR-100 at p = 0.2, distances change rapidly initially and then remain almost unchanged; similar trends appear elsewhere.
- Efficient training: EB Train uses early ticket detection to reduce training cost, achieving 2.2–2.4× FLOPs reduction over progressive pruning baselines.Across models and datasets, FP32 EB Train improves energy and FLOPs relative to pruning baselines while maintaining comparable or better accuracy; FP8 yields larger reported savings.
A.1 MEASUREMENT OF ENERGY COST
The energy measurement setup connects a laptop, GPU board, and power meter to measure energy consumption and runtime across the full training process.
- Training settings are downloaded from the laptop to the GPU board before measurement.
- The power meter and runtime measurement provide real-measured energy consumption for the whole training process.
- The setup includes a Mac Air laptop, an embedded JETSON TX2 GPU, and a power meter.
A.2 APPROPRIATE LARGE LEARNING RATE FAVORS THE EMERGENCE OF EB TICKETS
The experiments examine whether larger initial learning rates favor early-bird ticket emergence, using retraining accuracy across different drawing epochs with a fixed pruning ratio.
- The study compares the original learning-rate schedule with larger initial rates of 0.5 for VGG16 and 0.2 for PreResNet101.The original schedule starts at 0.1, then decays at the 80th and 120th epochs.
- Table 4 reports retraining accuracy for subnetworks drawn at different training epochs under different initial learning rates.The pruning ratio is fixed at 0.5.