PaperlayerView exact revision on arXiv

Source-linked AI summary

Do We Actually Need Dense Over-Parameterization? In-Time Over-Parameterization in Sparse Training

Shiwei Liu, Lu Yin, Decebal Constantin Mocanu, Mykola Pechenizkiy

arXiv:2102.02887v3cs.LGcs.AIcs.CV

TL;DR

Dense over-parameterization makes modern neural networks costly, while sparse training has limited expressibility. The paper introduces ITOP, which explores sparse connectivities across training time, and reports improved sparse performance, including dense-level performance at extreme sparsity.

  • Problem

    Dense over-parameterization makes state-of-the-art models expensive, while sparse training must close an expressibility gap without relying on dense pre-training.

  • Method

    ITOP starts from a sparse network and continuously explores sparse connectivities across training time, treating the resulting exploration as over-parameterization in the space-time manifold.

  • Results

    ITOP improves state-of-the-art sparse-training performance on ImageNet, and sufficiently reliable exploration enables DST to match or outperform dense networks at extreme sparsity.

  • Takeaways & Limitations

    Reliable parameter exploration is central to sparse training and can let dynamically sparse models outperform dense-to-sparse alternatives without dense over-parameterization throughout training.

  • Takeaways & Limitations

    The study focuses on parameter exploration and defines reliable exploration mainly through newly activated weights updated for at least ΔT ≥ T0.

Abstract

from arXiv · show

In this paper, we introduce a new perspective on training deep neural networks capable of state-of-the-art performance without the need for the expensive over-parameterization by proposing the concept of In-Time Over-Parameterization (ITOP) in sparse training. By starting from a random sparse network and continuously exploring sparse connectivities during training, we can perform an Over-Parameterization in the space-time manifold, closing the gap in the expressibility between sparse training and dense training. We further use ITOP to understand the underlying mechanism of Dynamic Sparse Training (DST) and indicate that the benefits of DST come from its ability to consider across time all possible parameters when searching for the optimal sparse connectivity. As long as there are sufficient parameters that have been reliably explored during training, DST can outperform the dense neural network by a large margin. We present a series of experiments to support our conjecture and achieve the state-of-the-art sparse training performance with ResNet-50 on ImageNet. More impressively, our method achieves dominant performance over the overparameterization-based sparse methods at extreme sparsity levels. When trained on CIFAR-100, our method can match the performance of the dense model even at an extreme sparsity (98%). Code can be found https://github.com/Shiweiliuiiiiiii/In-Time-Over-Parameterization.

1. Introduction

The paper proposes In-Time Over-Parameterization (ITOP), which explores sparse connectivities across training time instead of relying on dense pre-training. This approach aims to improve sparse expressibility while reducing the cost of dense over-parameterization.

  • 1. Introduction: Dense-to-sparse methods require full or partial dense pre-training, creating substantial training and deployment costs.These costs motivate sparse training approaches that avoid expensive dense over-parameterization.
  • 1. Introduction: ITOP continuously explores sparse connectivities across training time, performing over-parameterization in the space-time manifold while keeping the active parameter count fixed.It differs from methods that inherit weights from a dense, pre-trained model.
  • 1. Introduction: ITOP improves sparse-training expressibility, reduces training and inference costs, and supports analysis of dynamic sparse training and generalization.The paper presents these as key uses of the proposed concept.
  • 1. Introduction: The method improves state-of-the-art sparse-training performance with ResNet-50 on ImageNet and can match dense networks with fewer training FLOPs.The comparison includes over-parameterization-based methods such as LTH, GMP, and PI.

2. Related Work

Related work distinguishes sparse methods by how much they depend on dense over-parameterization. Dynamic Sparse Training instead evolves a fixed-size sparse network during training, while prior analyses study its changing topology and initialization behavior.

  • 2. Related Work: Dense-to-sparse methods include full, partial, and one-shot over-parameterization, differing in when and how extensively dense training precedes sparsification.Examples include iterative pruning after convergence, gradual magnitude pruning, and pruning at initialization.
  • 2. Related Work: Dynamic Sparse Training starts from a random sparse network, maintains a fixed parameter count, and evolves connectivity during training.SET established this sparse-to-sparse training paradigm and improved performance over static sparse networks.
  • 2. Related Work: Prior work finds that DST can move from its initial topology toward a substantially different sparse topology, while dense initialization can harm sparse-network performance.These findings motivate further investigation of DST's underlying mechanism.

3. In-Time Over-Parameterization

In-Time Over-Parameterization treats continuous sparse-connectivity exploration as over-parameterization across a space-time manifold. The hypothesis links Dynamic Sparse Training performance to reliably explored parameters, and experiments show that exploration can improve sparse models beyond dense baselines under suitable training conditions.

  • 3. In-Time Over-Parameterization: Continuous parameter exploration across training time provides an alternative form of over-parameterization that increases sparse-training expressibility.The paper distinguishes this space-time exploration from dense over-parameterization, which expands the spatial parameter space.
  • 3.1. In-Time Over-Parameterization Hypothesis: Dynamic Sparse Training is hypothesized to improve mainly because it explores parameters across time while searching for sparse connectivity.This hypothesis identifies parameter exploration as the central mechanism differentiating dynamic from static sparse training.
  • 3.1. In-Time Over-Parameterization Hypothesis: Reliably explored parameters are newly grown weights updated long enough to exceed the pruning threshold, and their total number is tied to performance.The paper defines the In-Time Over-Parameterization rate R_s as the ratio of reliably explored parameters to total parameters.
  • 3.1. In-Time Over-Parameterization Hypothesis: With sufficient reliable exploration, Dynamic Sparse Training can match or substantially outperform dense models, including at extremely high sparsity levels.The paper formalizes this as a hypothesis involving thresholds for update intervals and exploration rates.
  • 3.2.1. TYPICAL TRAINING TIME: In typical-time experiments, reducing the connectivity-update interval initially improves accuracy but eventually harms performance when exploration becomes unreliable, especially at high sparsity.Extremely small intervals can reach complete exploration yet perform worse than static sparse training because newly activated weights receive insufficient updates.
  • 3.2.2. EXTENDED TRAINING TIME: With extended training and increased R_s, sparse models consistently improve, can surpass the dense baseline, and lower sparsities reach their accuracy plateau sooner.Extended dynamic sparse training continued improving after matching the dense baseline, whereas extended dense training was inferior for MLPs and VGG-16 and equal for ResNet-34.

4. Effect of Hyperparameter Choices

Hyperparameter choices that increase reliable parameter exploration improve sparse-training performance, but their effects depend on the weight-growth method and training budget.

  • 4.1. Effect of Weight Growth methods on ITOP: RigL benefits more than SET from reliable exploration but suffers more from unreliable exploration as the update interval changes.RigL’s gradient-based growth can accelerate loss decrease when exploration is faithful, but requires a larger ΔT.
  • 4.1. Effect of Weight Growth methods on ITOP: With sufficient training, SET surpasses RigL because random growth explores a broader connectivity space, whereas gradient growth drives connectivity toward similar structures.RigL can perform better with limited training, while SET becomes stronger with sufficient training.
  • 4.2. Effect of Batch Size on ITOP: Smaller batch sizes increase the exploration rate by producing more updates, improving SET accuracy relative to large-batch training.Large batches leave parameter exploration insufficient and test accuracy substantially below the dense model.
  • 4.3. Effect of Pruning Rate on ITOP: Larger initial pruning rates broaden exploration and improve accuracy until excessive pruning reduces model capacity.This pattern is observed across update intervals for sparse ResNet-18 on CIFAR-10.
  • 4.4. Boosting the Performance of DST: 2× training time lets RigL-ITOP match dense ResNet-50 at 80% sparsity, while requiring less training than RigL’s 5× baseline.The method uses ΔT=4000, batch size 64, and initial pruning rate 0.5.

5. The Versatility of ITOP

ITOP extends beyond dynamic sparse training by improving initialization-based sparse methods and achieving strong performance at extreme sparsity.

  • 5. The Versatility of ITOP: At 98% sparsity, SET-ITOP has dominant performance over LTH and SNIP while matching corresponding dense models with at most 5% of their parameters.The comparison trains models for 200 epochs and reports best test accuracy.
  • 5. The Versatility of ITOP: SNIP-SET-ITOP and LTH-SET-ITOP match or exceed SET-ITOP with MLP and ResNet-34, showing that ITOP benefits multiple sparse-training methods.These variants combine the original methods with ITOP-style weight exploration.

6. Generalization Improvement of ITOP

ITOP improves generalization in addition to sparse-model performance: ITOP models have lower generalization error than corresponding dense models.

  • 6. Generalization Improvement of ITOP: ITOP models generalize better than corresponding dense MLPs on CIFAR-10, while dense-model generalization error increases with model density.The comparison includes SET-ITOP, RigL-ITOP, and dense models.

7. Conclusion and Future Work

The paper presents ITOP as a space-time alternative to dense over-parameterization and emphasizes parameter exploration as the central resource-allocation principle for sparse training.

  • 7. Conclusion and Future Work: ITOP improves sparse expressibility, accelerates training and inference, helps explain dynamic sparse training, and improves generalization.The conclusion lists these as demonstrated abilities of ITOP.
  • 7. Conclusion and Future Work: A remaining challenge is achieving high ITOP rates with large batch sizes within typical training times, because small batches slow modern-architecture training.The paper identifies this as an open direction and also suggests possible applications to interpretability, robustness, uncertainty, and spurious-correlation detection.

Appendices

The appendix evaluates ITOP using MLP, VGG-16, and ResNet-34 models, with SET as the dynamic sparse training method. SET repeatedly updates a fixed-size sparse topology by pruning weights and randomly growing replacements.

  • Models and sparse training: SET evaluates ITOP without dense-gradient exploration, starting from an Erdős-Rényi sparse topology and optimizing connectivity toward a scale-free topology.The appendix uses SET to avoid the dense over-parameterization bias associated with gradient-based methods such as RigL and SNFS.
  • SET procedure: SET initializes a sparse neural network at sparsity S, trains it for ΔT iterations, then prunes by magnitude and randomly grows new weights.The initialization, training, pruning, and growth steps define each sparse-connectivity update.
  • SET procedure: Repeating training and topology updates preserves a fixed parameter count throughout training.Steps 2 and 3 continue iteratively until training ends.

A.3. Training

The training appendix specifies hyperparameters for typical and extended sparse-training experiments. Extended training scales the learning and pruning schedules while using update intervals selected from an empirical trade-off.

  • Hyperparameter specification: Table 3 organizes the hypothesis-evaluation settings by learning rate, batch size, epochs, schedule, initialization, update interval, and pruning rate.The table covers both typical and extended-training-related hyperparameters.
  • Training schedules: Typical training explores sparse connectivities throughout training, using random weight growth, magnitude-based pruning, and an initial pruning rate of 0.5 decayed to 0.The pruning-rate decay follows cosine annealing, and 10% of the training data is used for validation.
  • Model-specific settings: The model-specific settings train MLPs for 200 epochs, VGG-16 for 250 epochs, and ResNet-34 for 200 epochs.The schedules also specify model-specific learning rates, decay points, batch sizes, and weight decay values.
  • Training schedules: Extended training scales the training duration, learning-rate schedule, and pruning-rate schedule by the same factor while using a large update interval ΔT.Accuracy is averaged over three seeds with mean and standard deviation for each training time.

B. Implementation Details of RigL-ITOP in Section 4.2

The appendix replicates RigL and configures RigL-ITOP to explore sparse connectivities throughout training. It also documents one-shot LTH and SNIP baselines for comparison.

  • RigL-ITOP: RigL-ITOP explores sparse connectivities throughout 100-epoch ImageNet training instead of tuning separate mask-update start and end points.The setup uses ResNet-50, batch size 64, update interval 4000, and two NVIDIA Tesla V100 GPUs.
  • RigL-ITOP: RigL grows weights expected to receive high-magnitude gradients in the next iteration and uses ERK to improve sparse performance over ER.The appendix describes RigL as a state-of-the-art dynamic sparse training method.
  • Baselines: LTH is replicated with one-shot magnitude pruning after dense pretraining, followed by retraining the pruned model with its original initialization.This setup avoids the larger training cost of iterative LTH pruning.
  • Baselines: SNIP prunes at initialization using connection-sensitivity scores and then trains the resulting sparse model without sparse exploration.The score is based on the absolute product of a loss derivative and its corresponding weight.

D. Extended Training Performance of RigL with ∆T = 1500

The appendix compares extended RigL training with update intervals of 1500 and 4000. The larger interval performs better, indicating that small intervals may not give newly grown weights enough time to catch up.

  • Extended RigL training: RigL models with ΔT = 1500 fall short of models with ΔT = 4000 during extended training.The comparison is shown in Figure 9, with the larger interval identified as the stronger configuration.
  • Extended RigL training: The appendix interprets the poorer ΔT = 1500 performance as evidence that small update intervals are insufficient for newly grown weights to catch up.This comparison follows the earlier selection of ΔT = 4000 as a good RigL update interval.
  • Batch-size comparison: RigL performance increases as batch size decreases from 256 to 32, then drops at extremely small batch sizes because of noisy input.The batch-size experiment uses an MLP with ΔT = 4000.

F. Regrowing from the Non-Activated Weights First

Regrowing non-activated weights first increases parameter exploration and improves SET performance when exploration is insufficient. However, the resulting increase in exploration remains limited, motivating methods that achieve higher exploration rates within typical training times.

  • F. Regrowing from the Non-Activated Weights First: Regrowing non-activated weights first is proposed to increase the In-Time Over-Parameterization rate during training.The idea is evaluated with SET+ on sparse ResNet-18 trained on CIFAR-10.
  • F. Regrowing from the Non-Activated Weights First: SET+ consistently achieves higher accuracy and higher Rs than SET when parameter exploration is insufficient.The modification is effective under small Rs, according to the reported comparison.
  • F. Regrowing from the Non-Activated Weights First: The Rs increase from this modification is relatively limited, leaving higher exploration within typical training time as future work.
  • F. Regrowing from the Non-Activated Weights First: Table 5 reports sparse ResNet-18 performance on CIFAR-10 across pruning rates using three runs and mean ± std with Rs.
Loading 2102.02887v3…