PaperlayerView exact revision on arXiv

Source-linked AI summary

A Signal Propagation Perspective for Pruning Neural Networks at Initialization

Namhoon Lee, Thalaiyasingam Ajanthan, Stephen Gould, Philip H. S. Torr

arXiv:1906.06307v2cs.LGcs.CVstat.ML

TL;DR

Pruning an untrained, randomly initialized network can be effective, but the conditions making connection sensitivity reliable are not fully understood. The paper characterizes initialization through signal propagation and dynamical isometry, studies trainability after pruning, and introduces a data-free recovery method plus unsupervised pruning analyses. These modifications improve results across tested architectures, while the two-stage orthogonalization process and changing weights during training remain limitations.

  • Problem

    The paper addresses why pruning an untrained, randomly initialized neural network can produce effective results and how initialization affects connection-sensitivity reliability.

  • Method

    The paper uses a signal-propagation and dynamical-isometry analysis to characterize reliable initialization, then studies sparse-network trainability and introduces a data-free method to recover approximate dynamical isometry.

  • Results

    The proposed modifications improve pruning results across tested convolutional and residual architectures, while unsupervised pruning reaches extreme sparsity without compromising accuracy and often compares competitively with supervised pruning.

  • Takeaways & Limitations

    Signal propagation provides a practical perspective for improving pruning at initialization and for applying unsupervised pruning to non-standard architectures.

  • Takeaways & Limitations

    Two-stage orthogonalization can be suboptimal at high sparsity, and training-induced weight changes may prevent dynamical isometry from continuing to hold.

Abstract

from arXiv · show

Network pruning is a promising avenue for compressing deep neural networks. A typical approach to pruning starts by training a model and then removing redundant parameters while minimizing the impact on what is learned. Alternatively, a recent approach shows that pruning can be done at initialization prior to training, based on a saliency criterion called connection sensitivity. However, it remains unclear exactly why pruning an untrained, randomly initialized neural network is effective. In this work, by noting connection sensitivity as a form of gradient, we formally characterize initialization conditions to ensure reliable connection sensitivity measurements, which in turn yields effective pruning results. Moreover, we analyze the signal propagation properties of the resulting pruned networks and introduce a simple, data-free method to improve their trainability. Our modifications to the existing pruning at initialization method lead to improved results on all tested network models for image classification tasks. Furthermore, we empirically study the effect of supervision for pruning and demonstrate that our signal propagation perspective, combined with unsupervised pruning, can be useful in various scenarios where pruning is applied to non-standard arbitrarily-designed architectures.

1 INTRODUCTION

Pruning at initialization can be effective, but its reliability depends on how random weights preserve signal and gradient propagation. The paper develops a signal-propagation account, improves sparse-network trainability, and reports robust gains across modern architectures and unsupervised settings.

  • Motivation: Pruning at initialization can remove parameters before training while retaining good accuracy when random weights are appropriately scaled.The approach uses connection sensitivity and can reach extreme sparsity levels.
  • Problem: Improper initialization can make connection sensitivity unreliable because propagated signals produce saturated gradients and potentially catastrophic pruning failures.Initial weights affect sensitivity through the interaction between gradients and weights.
  • Signal propagation perspective: Layerwise dynamical isometry provides a sufficient initialization condition for faithful connection sensitivity measurements and effective pruning.It concentrates all singular values of layerwise Jacobians around 1.
  • Sparse-network trainability: Pruning can break dynamical isometry and degrade trainability, but a simple data-free method recovers layerwise orthogonality and significantly improves compressed-network training.Trainability also depends on the pruning method and sparsity level.
  • Empirical results: Experiments across convolutional and residual architectures show that dynamical-isometry-based sensitivity robustly outperforms sensitivity based on other initialization schemes.The method remains useful even when ReLU or batch normalization prevents perfect dynamical isometry.
  • Unsupervised pruning: Unsupervised pruning reaches extreme sparsity without compromising accuracy and can compare competitively with supervised pruning, while neural architecture sculpting compares favorably with hand-designed architectures.The paper applies this perspective to arbitrarily designed architectures and architecture discovery.

2 PRELIMINARIES

The preliminaries define pruning at initialization through connection-sensitivity scores and motivate dynamical isometry as a way to preserve signal norms during propagation. Scores are computed from gradients with respect to connectivity variables and used to retain the most salient parameters.

  • Pruning at initialization: SNIP prunes an untrained network at initialization, avoiding the conventional requirement to train before pruning.The method is designed as a single-shot pruning procedure before training.
  • Connection sensitivity: Connection sensitivity measures the loss change associated with removing a parameter and is computed using derivatives with respect to connectivity indicators.The indicators encode whether parameters remain connected.
  • Dynamical isometry: Faithful signals preserve norms with minimal amplification or attenuation, which is the operational meaning of dynamical isometry used in the paper.The term is borrowed from prior work introducing dynamical isometry.
  • Pruning rule: The pruning procedure keeps the top-κ parameters according to connection-sensitivity scores, where κ denotes the desired sparsity level.A global threshold is applied to select the retained parameters.
  • Dynamical isometry: Dynamical isometry is stronger than matching the mean squared singular value because it controls worst-case amplification or attenuation of error vectors.Mean squared singular values close to 1 preserve norms only on average.

3 SIGNAL PROPAGATION PERSPECTIVE TO PRUNING RANDOM NETWORKS

The paper links pruning-at-initialization reliability to signal propagation: initialization controls connection sensitivities through gradients, while layerwise dynamical isometry provides a sufficient condition for faithful measurements. Poorly conditioned propagation can create nonuniform sparsity and disconnected networks.

  • 3.1 EFFECT OF INITIALIZATION ON PRUNING: Variance-scaling initialization changes connection sensitivities and can determine whether globally thresholded pruning leaves later layers critically underconnected.In tanh networks, later-layer sensitivities decrease, so high sparsity can retain very few later-layer parameters.
  • 3.2 GRADIENT SIGNAL IN CONNECTION SENSITIVITY: Poor forward signal propagation can saturate backpropagated gradients, miscalibrate sensitivities across layers, and potentially disconnect entire signal paths during pruning.The paper identifies this mechanism as the source of unreliable connection sensitivity in poorly initialized networks.
  • 3.2 GRADIENT SIGNAL IN CONNECTION SENSITIVITY: Connection sensitivity equals the gradient with respect to a connectivity variable multiplied elementwise by its corresponding weight.Thus, faithful sensitivity scores require both gradients and weights to remain comparably scaled across layers.
  • 3.3.2 ENSURING FAITHFUL GRADIENTS: Layerwise dynamical isometry requires each layer’s Jacobian singular values to be concentrated near 1, which is sufficient for faithful signal propagation and gradients.For linear networks, orthogonal weight matrices provide exact layerwise dynamical isometry.
  • 3.3.2 ENSURING FAITHFUL GRADIENTS: Connection-sensitivity pruning fails catastrophically when the input-output Jacobian condition number exceeds 1e+11, producing completely disconnected networks.Moderate changes in Jacobian singular values are tolerated, but very poor conditioning corresponds to unreliable sensitivity measurements.

4 SIGNAL PROPAGATION IN SPARSE NEURAL NETWORKS

Pruning can disrupt dynamical isometry, weakening signal propagation and slowing training in sparse networks. A data-free approximate-isometry procedure repairs this degradation and improves trainability across tested architectures.

  • Effect of pruning on signal propagation and trainability: Pruning progressively reduces Jacobian singular values, with random pruning degrading signal propagation faster than connection-sensitivity pruning.Layerwise dynamical isometry preserves signal propagation somewhat better than variance scaling, especially at lower sparsity.
  • Effect of pruning on signal propagation and trainability: At 90% sparsity, better signal propagation correlates with faster convergence, while random pruning takes over an order of magnitude longer than connection-sensitivity pruning to approach near-zero loss.The comparison is between the training behavior of differently pruned sparse networks and the dense LDI network.
  • Enforcing approximate dynamical isometry: The proposed data-free method optimizes sparse weights toward layerwise orthogonality using the pruning mask and initial weights.It minimizes the Frobenius distance between each masked weight Gram matrix and the identity matrix.
  • Enforcing approximate dynamical isometry: After approximate-isometry enforcement, decreased singular values rise close to their pre-pruning level, indicating restored orthogonality in the sparse topology.The method is evaluated by comparing LDI-CS and LDI-Rand with their approximate-isometry variants.
  • Results: Table 2 reports that LDI-AI achieves the best results across all tested architectures at 90% sparsity.The evaluation covers VGG16 and ResNets on CIFAR-10 and WRN16 on Tiny-ImageNet, using orthogonality scores and generalization errors.
  • Results: The signal-propagation perspective improves sparse-network trainability while complementing reduced-capacity explanations and extends to convolutional networks.The approach is described as simple and data free, and the paper also reports improved training speed for randomly pruned networks.

5 VALIDATION AND EXTENSIONS

The validation experiments test signal-propagation-based pruning across architectures, datasets, supervision settings, and arbitrarily designed networks. Approximate dynamical isometry and unsupervised pruning consistently support effective sparse models, including architecture sculpting.

  • 5.1 EVALUATION ON VARIOUS NEURAL NETWORKS AND DATASETS: Layerwise dynamical isometry with enforced approximate isometry achieves the best pruning results across all tested architectures.Orthogonality scores and generalization performance are highly correlated, with lower scores associated with better trained-network errors.
  • 5.1 EVALUATION ON VARIOUS NEURAL NETWORKS AND DATASETS: Orthogonal initialization consistently outperforms variance scaling methods across tested nonlinear activation functions.The comparison includes tanh, leaky-ReLU, and SELU activations.
  • 5.2 PRUNING WITHOUT SUPERVISION: Unsupervised pruning remains competitive with supervised pruning, including at extreme sparsity levels and across several architectures.Uniform surrogate loss produces errors of 8.25%, 11.69%, 11.01%, and 8.82% for VGG16, ResNet32, ResNet56, and ResNet110, respectively.
  • 5.2 PRUNING WITHOUT SUPERVISION: Transferred unsupervised pruning is less affected by task changes than other transferred or unsupervised pruning settings.Reported changes are 0.52 to 0.14 on MNIST and 1.11 to −0.78 on Fashion-MNIST.
  • 5.3 NEURAL ARCHITECTURE SCULPTING: Pruning wider networks can produce better sparse equivalents than pruning deeper networks, even when all models have the same parameter count.Sparse equivalents outperform the dense base model without pretraining or supervision.

6 DISCUSSION AND FUTURE WORK

The discussion consolidates signal propagation as the basis for understanding and improving pruning at initialization, while identifying unresolved issues in two-stage orthogonalization and training-time dynamics.

  • 6 DISCUSSION AND FUTURE WORK: Signal propagation characteristics of sparse networks correlate highly with trainability, and pruning can break dynamical isometry established at initialization.The resulting degradation in trainability motivates recovering orthogonality after pruning.
  • 6 DISCUSSION AND FUTURE WORK: A data-free method to recover orthogonality improves the trainability of compressed networks.The method is introduced to address signal-propagation degradation caused by pruning.
  • 6 DISCUSSION AND FUTURE WORK: Two-stage pruning followed by orthogonalization can be suboptimal at high sparsity, and training may invalidate dynamical isometry over time.The authors propose jointly addressing optimization and signal propagation as future work.

A GRADIENTS IN TERMS OF JACOBIANS

This appendix expresses gradients with respect to weights through error signals, layerwise Jacobians, activations, and Kronecker products, then specializes the relation to linear networks.

  • A GRADIENTS IN TERMS OF JACOBIANS: The gradient with respect to a layer’s weights combines the output error signal, the Jacobian from that layer to the output, and the preceding-layer activation.The expression uses a Kronecker product to represent the vectorized matrix gradient.
  • A GRADIENTS IN TERMS OF JACOBIANS: The derivation obtains these expressions by applying the chain rule and representing matrix gradients in vectorized form.The Jacobians are defined between the input, intermediate layers, and output.
  • A GRADIENTS IN TERMS OF JACOBIANS: For linear networks, the activation function becomes the identity and intermediate states reduce to linear transformations of the input plus a constant.The preceding-layer state is written using products of weight matrices and bias terms.

B EXPERIMENT SETTINGS

The experiments prune at initialization, train sparse networks on image-classification benchmarks, evaluate signal propagation, and include architecture-sculpting configurations with controlled parameter counts.

  • B EXPERIMENT SETTINGS: Pruning at initialization uses connection sensitivity scores, typically removes 90% of parameters, and compares variance-scaling, orthogonal, random, and pretrained magnitude pruning.Connection sensitivity is computed using all training examples to avoid mini-batch stochasticity.
  • B EXPERIMENT SETTINGS: Sparse networks are evaluated on MNIST, CIFAR-10, and Tiny-ImageNet using SGD with momentum for up to 80k or 100k iterations.Learning rates are initialized at 0.1 and decayed according to the dataset-specific schedule.
  • B EXPERIMENT SETTINGS: Approximate dynamical isometry is enforced by optimizing an orthogonality objective for pruned networks using a data-free procedure.The procedure uses gradient descent for up to 10k update steps and usually converges earlier.
  • B EXPERIMENT SETTINGS: Architecture-sculpting models begin wider or deeper than ResNet20 and are pruned to its 269k-parameter size.The widening factor controls filter count, while all models use three residual block layers.

C SIGNAL PROPAGATION IN SPARSE NETWORKS: ADDITIONAL RESULTS

Additional experiments compare signal propagation and training behavior across pruning methods, architectures, and activation functions. They show that connection-sensitivity pruning supports faster training, while approximate isometry further improves trainability.

  • Comparative results: Signal propagation and training behavior are evaluated for multiple pruning methods on 7-layer linear and tanh MLP networks.The comparisons include singular-value statistics and training logs, with separate method sets for linear and tanh cases.
  • Training behavior: Random and magnitude-pruned sparse networks take much longer to train than networks pruned using connection sensitivity.The methods use layerwise orthogonal initialization and the same training procedure.
  • Signal propagation: Magnitude pruning performed on a pretrained model causes an abrupt change in the singular-value distribution after pruning.This differs from pruning at initialization using random or connection-sensitivity criteria.
  • Activation functions: ReLU and Leaky-ReLU experiments resemble the tanh results, supporting the same conclusion about signal propagation and training behavior.The comparison is presented for both activation functions.
  • Approximate isometry: Approximate isometry improves VGG16 training speed dramatically compared with connection-sensitivity pruning without isometry.The experiment uses CIFAR-10 without batch-normalization layers to isolate initialization effects on trainability.
Loading 1906.06307v2…