PaperlayerView exact revision on arXiv

Source-linked AI summary

Parameter Efficient Training of Deep Convolutional Neural Networks by Dynamic Sparse Reparameterization

Hesham Mostafa, Xin Wang

arXiv:1902.05967v3cs.LGstat.ML

TL;DR

The paper asks how to train deep residual CNNs effectively under a small, fixed training-time parameter budget without relying on a large dense model. It introduces dynamic sparse reparameterization that reallocates parameters during training and reports performance comparable to iterative pruning while surpassing static and dynamic alternatives. The findings indicate that continual structural exploration, not only the final sparse structure or initialization, is central to effective learning.

  • Problem

    Existing compression methods often require a pre-trained or fully overparameterized model, leaving the problem of training accurately under a small fixed parameter budget.

  • Method

    The paper directly trains sparse residual CNNs and adaptively reallocates non-zero parameters within and across layers while maintaining a constant small model size.

  • Results

    The proposed method outperformed previous static and dynamic reparameterization methods, achieved accuracy comparable to iterative pruning of pre-trained dense models, and exceeded same-size dense training.

  • Takeaways & Limitations

    The results suggest that exploring structural degrees of freedom during training can be more effective than allocating the entire memory budget to conventional weights.

  • Takeaways & Limitations

    The method’s unstructured sparsity is difficult for mainstream CPUs and GPUs to handle efficiently, while structured sparsity produced significantly different results in additional experiments.

Abstract

from arXiv · show

Modern deep neural networks are typically highly overparameterized. Pruning techniques are able to remove a significant fraction of network parameters with little loss in accuracy. Recently, techniques based on dynamic reallocation of non-zero parameters have emerged, allowing direct training of sparse networks without having to pre-train a large dense model. Here we present a novel dynamic sparse reparameterization method that addresses the limitations of previous techniques such as high computational cost and the need for manual configuration of the number of free parameters allocated to each layer. We evaluate the performance of dynamic reallocation methods in training deep convolutional networks and show that our method outperforms previous static and dynamic reparameterization methods, yielding the best accuracy for a fixed parameter budget, on par with accuracies obtained by iteratively pruning a pre-trained dense model. We further investigated the mechanisms underlying the superior generalization performance of the resultant sparse networks. We found that neither the structure, nor the initialization of the non-zero parameters were sufficient to explain the superior performance. Rather, effective learning crucially depended on the continuous exploration of the sparse network structure space during training. Our work suggests that exploring structural degrees of freedom during training is more effective than adding extra parameters to the network.

1. Introduction

The paper targets parameter-efficient training of deep CNNs without retaining a large dense model, using differentiable reparameterization with dynamically changing sparse connectivity. Its method reallocates parameters during training and indicates that structural exploration can be more valuable than assigning all memory to conventional weights.

  • Motivation: Compression methods reduce model size but commonly require a pre-trained or fully overparameterized model during training, leaving training memory-inefficient.The paper frames direct training under a small, fixed parameter budget as the central challenge.
  • Reparameterization: Reparameterization represents an original model θ through φ and ψ, where the differentiable function g maps these variables into the model parameters.The reparameterized network treats ψ as metaparameters and can use fewer total degrees of freedom while preserving generalization.
  • Sparse reparameterization: Sparse reparameterization uses non-zero entries as weights and their indices as connectivity, with dynamic variants adapting connectivity during training.Static variants fix connectivity, whereas dynamic variants adjust it adaptively.
  • Contribution: The proposed dynamic reparameterization method achieved the highest parameter efficiency among evaluated methods for training sparse deep residual networks.It outperformed existing static and dynamic reparameterization methods.
  • Implication: Under limited storage and computation, the method motivates allocating part of the budget to describing and evolving network structure instead of only dense-network parameters.This conclusion connects the method’s performance to the value of structural degrees of freedom during training.
  • Key finding: Training-time structural exploration, rather than final sparse structure or initial non-zero weights alone, was necessary for best generalization.The paper characterizes structural exploration and gradient-based parameter optimization as synergistic.

2. Related work

Prior work compresses trained networks, reparameterizes dense layers, or trains sparse structures with fixed or heuristic connectivity changes. The paper situates its approach among dynamic sparse methods that alter non-zero parameter locations during training.

  • Dense reparameterization: Dense reparameterization methods reduce parameterization size or impose algebraic properties, but the cited dense methods are static.Examples include low-rank, transform-based, hashing, and structured matrix parameterizations.
  • Sparse reparameterization: Conventional sparse training usually starts from a large pre-trained model, gradually prunes weights, and fine-tunes the remaining sparse network.Training a small static sparse model from scratch has been reported to perform worse at the same sparsity.
  • Sparse reparameterization: Structured pruning removes channels or layers and therefore produces essentially small dense models rather than unstructured sparse models.The paper benchmarks non-structured sparsification and distinguishes it from coarse-grained structured pruning.
  • Sparse subnetworks: The lottery-ticket line of work links successful sparse subnetworks to both their structure and initialization, while hypothesizing that overparameterization supplies many candidate subnetworks.This provides a contrasting explanation to the paper’s emphasis on training-time exploration.
  • Dynamic sparse methods: Dynamic sparse methods adaptively move non-zero parameters during training using heuristics such as magnitude pruning, random growth, gradient-guided growth, or rewiring.SET, NeST, and DeepR exemplify distinct dynamic strategies discussed by the paper.

3. Methods

The method directly trains sparse residual CNNs while maintaining a constant non-zero-parameter budget, periodically reallocating connectivity within and across tensors. It uses adaptive magnitude pruning and heuristic random growth, with comparisons made at matched descriptive size.

  • Dynamic reallocation: Sparse tensors are randomly initialized, and free parameters are moved within and across weight tensors every few hundred iterations while total non-zero parameters remain constant.The procedure applies to most CNN layers and preserves the global parameter budget throughout training.
  • Pruning: Magnitude pruning uses an adaptive global threshold H to remove small-magnitude weights and maintain approximately Np pruned or grown parameters per reallocation step.A negative-feedback adjustment controls H within tolerance δ.
  • Growth and allocation: After pruning, zero-initialized parameters are redistributed across tensors, with layers receiving larger non-zero fractions assigned proportionally more free parameters.The heuristic is motivated by allocating parameters toward layers whose weights receive stronger or more consistent loss gradients.
  • Algorithmic differences: Compared with SET, the algorithm uses adaptive rather than fixed-fraction pruning and automatically reallocates parameters across layers without manually fixed layerwise sparsity.The paper attributes the first change to lower sorting overhead and the second to better-performing, extremely sparse networks.
  • Baselines: The evaluation compares dynamic sparse training with full dense, thin dense, static sparse, compressed sparse, and other sparse baselines at specified global sparsity.Compressed sparse uniquely trains a large dense model first and then prunes and fine-tunes it across additional epochs.
  • Evaluation protocol: Models are compared at equal descriptive length because sparse tensors require storage for both non-zero values and connectivity locations.The comparison assumes one bit per tensor position for the connectivity mask.

4. Experimental results

Experiments show that dynamic sparse reparameterization trains accurate sparse CNNs across CIFAR10 and ImageNet while reallocating parameters with negligible overhead. The results indicate that evolving connectivity during training, rather than final structure or initialization alone, drives the observed generalization advantage.

  • WRN-28-2 on CIFAR10: Dynamic sparse training performed on par with or slightly better than compressed sparse models for WRN-28-2 on CIFAR10.Static sparse and thin dense methods underperformed compressed sparse models, while DeepR underperformed all other methods.
  • Layer-wise sparsity: Dynamic parameter reallocation produced consistent layer-wise patterns: larger tensors and deeper layers became sparser across sparsity levels.Figure 1b breaks down final sparsity across residual blocks.
  • Resnet-50 on ImageNet: Dynamic sparse reparameterization outperformed static and dynamic baselines on Resnet-50 trained on ImageNet, slightly surpassing compressed sparse models.The same reliable layer-wise sparsity trends observed on WRN-28-2 also emerged during ImageNet training.
  • Computational cost: Dynamic reallocation added negligible computational overhead compared with intermittent sparse reallocation methods, whereas DeepR incurred significantly larger overhead.The method's reduced operation count did not substantially improve GPU time over SET, but it produced better sparse models and automatically reallocated free parameters.
  • Mechanism: The superior generalization of dynamically trained sparse networks could not be explained by their final structure, initialization, or the combination of both.Networks retaining the final structure but using random or original initialization failed to match dynamic training accuracy.
  • Training dynamics: Dynamic parameter reallocation need only remain active during some initial epochs, suggesting that network structure converges faster than parameter values.Stopping reallocation later in training may reduce computational cost.

5. Discussion

With a fixed parameter budget, dynamic parameter reallocation trains deep residual CNNs more effectively than static reparameterization and can match iterative pruning of dense models. The results indicate that continuously exploring structural degrees of freedom, rather than merely increasing parameter counts, is central to effective learning, although unstructured sparsity limits hardware efficiency.

  • 5. Discussion: Dynamic parameter reallocation achieved significantly better accuracies than static reparameterization at the same model size.The method also outperformed the static and dynamic reparameterization methods benchmarked in the study.
  • 5. Discussion: Automatic reallocation produced layer-wise sparsity patterns in which larger tensors and deeper layers tended to be sparser.These patterns were observed consistently at different sparsity levels.
  • 5. Discussion: The final sparse connectivity and its original initialization were each insufficient to explain the method’s high generalization performance.Reinitializing the discovered structure, including with the original initial values, still fell short of dynamic training.
  • 5. Discussion: Imposing structured sparsity by pruning 3 × 3 kernel slices produced significantly worse generalization than unstructured dynamic sparse training.The resulting networks performed on par with statically parameterized thin dense networks trained for twice as many epochs.
  • 5. Discussion: Dynamic exploration of structural degrees of freedom during training was crucial to effective learning and may be more effective than simply increasing model parameter counts.Structural degrees of freedom are explored through non-differentiable heuristics rather than directly optimized by SGD.

A. A full description of the dynamic parameter reallocation algorithm

The method maintains a fixed global sparsity while periodically reallocating non-zero parameters through global pruning and tensor-wise growth during training.

  • The algorithm represents each sparse tensor with non-zero values and their indices, while dynamic reparameterization adaptively changes those indices during training.
  • Global sparsity remains constant throughout training, with sparse positions sampled initially and associated parameters randomly initialized.
  • Dynamic reallocation repeats model training and parameter reallocation, preserving the fixed parameter budget while changing connectivity.
  • Each reallocation performs global magnitude-based pruning followed by tensor-wise growth, enabling parameter redistribution across layers.Pruning uses a global threshold rather than layer-specific sorting; growth samples zero weights uniformly and allocates parameters tensor-specifically.
  • An adaptive threshold targets roughly Np pruned parameters per iteration, using setpoint control to avoid sorting all network weights.
  • Growth allocates more free parameters to tensors with more surviving entries while balancing the numbers of pruned and grown parameters.

B. Details of implementation

The experiments use dense tensor representations with binary masks, standard training and augmentation settings, and baselines implemented through specified pruning or reallocation procedures.

  • All models were implemented in PyTorch, and sparse tensors were represented as dense tensors filtered by binary masks.
  • CIFAR10 and ImageNet experiments used standard augmentation, while WRN-28-2’s final linear layer remained dense because it contained few parameters.
  • The sparse compression baseline trained a dense model first and gradually pruned it to a target sparsity over T steps.
  • An extra redistribution step handled tensors exceeding their dense size after reallocation, assuring exact global sparsity despite rounding errors.
  • The implementation adjusted pruning thresholds according to whether the number of pruned parameters fell below, exceeded, or matched the target tolerance.
  • The dynamic sparse method used hyperparameter settings listed in the third block of Table 4, while DeepR settings were selected by parameter sweep.
  • SET used the dynamic method’s hyperparameters and reparameterization schedule for larger-scale comparisons where original settings were unavailable.

C. Comparison to dense reparameterization method HashedNet

Against HashedNet, the dynamic sparse method was evaluated on MNIST and CIFAR10 and significantly outperformed the hashed dense reparameterization baseline.

  • The comparison measures test accuracy for LeNet-300-100-10 on MNIST and WRN-28-2 on CIFAR10.
  • HashedNet creates dense parameter tensors with shared components through a cheap random mapping from original parameter indices.
  • HashedNet tensors were configured so each contained a fraction 1 −s of unique parameters at the corresponding global sparsity s.
  • Dynamic sparse significantly outperformed HashedNet across the reported MNIST and CIFAR10 comparisons.

D. A taxonomy of training methods that yield “sparse” deep CNNs

The taxonomy distinguishes sparse-CNN training methods by parameter-budget policy, sparsity granularity, and whether layer-wise sparsity is predefined or discovered automatically.

  • Methods are compared by three features: fixed parameter footprint, sparsity granularity, and automatic versus predefined layer-wise sparsity.
  • Strict parameter budget throughout training and inference: Direct training methods such as DeepR, SET, and this method maintain a fixed small parameter budget throughout training, unlike most compression methods.
  • Strict parameter budget throughout training and inference: A fixed budget is practically memory-efficient and theoretically informs whether gross overparameterization during training is necessary for good generalization.
  • Granularity of sparsity: Non-structured sparsity permits each individual weight to be independently zero or non-zero, whereas structured pruning removes coarser units such as feature maps or layers.
  • Granularity of sparsity: Structured methods can accelerate computation on GPUs by producing fewer or smaller dense parameter tensors, but they were not used as the main baseline here.
  • Granularity of sparsity: Constrained structured-sparsity experiments obtained significantly worse results than the method’s non-structured setting.
  • Predefined versus automatically discovered sparsity levels across layers: The method automatically discovers layer-wise sparsity levels during training while maintaining a fixed parameter footprint.

E. Structured versus non-structured sparsity

At kernel granularity, dynamic sparse reparameterization remains less accurate than unstructured sparsity, while channel-level sparsity fails to produce performant models.

  • Group-wise reallocation: The method adapts pruning and growth to groups of weights by comparing group L1-norms with an adaptive threshold.The threshold tracks the difference between target and actual group pruning or growth, while reallocation occurs within and across parameter tensors.
  • Kernel-level sparsity: Kernel-level sparsity leads to significantly worse accuracy than unstructured sparsity on WRN-28-2 and ResNet-50.For WRN-28-2, kernel-level reallocation still outperforms the thin dense baseline at higher sparsity, but this advantage disappears as sparsity decreases.
  • Training at extreme sparsity: At the highest sparsity level, static sparse, SET, and DeepR fail to train in the LeNet-300-100-10 MNIST comparison.Figure 7 reports medians over five runs with standard-deviation error bars, counting all trainable parameters.
  • Channel-level sparsity: At channel-level granularity, grouping Cin × 3 × 3 slices failed to produce performant models.This is a stricter coarsening than kernel-level sparsity.

F. Multi-layer perceptrons and training at extreme sparsity levels

Experiments on LeNet-300-100 trained on MNIST test whether dynamic parameter reallocation can distribute parameters effectively at extreme sparsity. The method automatically moves parameters between layers and discovers a lower sparsity for the top layer than for the hidden layers.

  • Multi-layer perceptrons: Dynamic parameter reallocation effectively trains LeNet-300-100 on MNIST at the high-sparsity setting.The method is the only evaluated approach besides pruning from a large dense model reported to succeed there.
  • Automatic layer allocation: The method automatically moves parameters between layers to realize layer sparsities that can be effectively trained.All layers were initialized at the same sparsity level, equal to the global sparsity.
  • Automatic layer allocation: The discovered sparsity pattern assigns much lower sparsity to the top layer than to the two hidden layers.Similar patterns were found through hand-tuning to improve DeepR performance.
Loading 1902.05967v3…