Source-linked AI summary
What's Hidden in a Randomly Weighted Neural Network?
Vivek Ramanujan, Mitchell Wortsman, Aniruddha Kembhavi, Ali Farhadi, Mohammad Rastegari
TL;DR
The paper asks whether accurate subnetworks can exist inside overparameterized neural networks with random, fixed weights. It proposes edge-popup to find them by optimizing scores rather than weights, and reports strong CIFAR-10 and ImageNet results, while noting stochastic sampling can destabilize training.
Problem
The paper asks whether overparameterized neural networks with random weights contain subnetworks that perform well without modifying the weights.
Method
The edge-popup algorithm assigns scores to fixed weights, selects top-scoring edges, and optimizes scores while leaving all weight values unchanged.
Results
Untrained subnetworks approach learned-weight networks as width and depth increase; on ImageNet, a smaller Wide ResNet-50 subnetwork matches ResNet-34 accuracy.
Takeaways & Limitations
Randomly weighted neural networks contain compelling subnetworks, offering an avenue for studying neural-network optimization and initialization.
Takeaways & Limitations
Stochastic forward-pass sampling may destabilize gradient estimates because the same subnetwork is unlikely to be observed twice.
Abstract
from arXiv · showhide
Training a neural network is synonymous with learning the values of the weights. By contrast, we demonstrate that randomly weighted neural networks contain subnetworks which achieve impressive performance without ever training the weight values. Hidden in a randomly weighted Wide ResNet-50 we show that there is a subnetwork (with random weights) that is smaller than, but matches the performance of a ResNet-34 trained on ImageNet. Not only do these "untrained subnetworks" exist, but we provide an algorithm to effectively find them. We empirically show that as randomly weighted neural networks with fixed weights grow wider and deeper, an "untrained subnetwork" approaches a network with learned weights in accuracy. Our code and pretrained models are available at https://github.com/allenai/hidden-networks.
1. Introduction
The paper asks whether overparameterized networks with random weights contain accurate subnetworks and introduces a way to find them without changing weight values.
- 1. Introduction: Overparameterized randomly weighted networks can contain subnetworks that perform well without modifying their weight values.The weights, including batch-normalization parameters and the first and last layers, remain fixed.
- 1. Introduction: The number of possible subnetworks grows combinatorially with network size, motivating an algorithm for finding effective subnetworks.Modern neural networks contain millions or even billions of parameters.
- 1. Introduction: The edge-popup algorithm assigns scores to fixed weights, selects the top-k% edges, and optimizes scores with stochastic-gradient updates.The algorithm searches within a fixed, randomly weighted network rather than learning weight values.
- 1. Introduction: On CIFAR-10, untrained subnetworks perform as well as dense networks with learned weights as the networks become wider and deeper.The comparison concerns randomly weighted networks with fixed weights and dense networks whose weights are learned.
- 1. Introduction: On ImageNet, a randomly weighted Wide ResNet-50 contains a smaller subnetwork matching trained ResNet-34 accuracy, while a ResNet-101 subnetwork surpasses VGG-16 performance.These results support the reported effectiveness of randomly weighted neural networks for image recognition.
2. Related Work
Related work includes trainable sparse subnetworks, stochastic supermask discovery, architecture search, randomly wired networks, and weight-agnostic networks; this paper searches standard architectures with fixed random weights.
- 2. Related Work: The Lottery Ticket Hypothesis proposes sparse subnetworks that can be trained from scratch after being reset to their initialization.Winning tickets are found by iteratively pruning low-magnitude weights after training runs.
- 2. Related Work: Zhou et al. introduced supermasks: subnetworks of randomly initialized networks that achieve high accuracy without training, including 65.4% accuracy on CIFAR-10.Their method learns inclusion probabilities for weights and samples the mask during the forward pass.
- 2. Related Work: This work replaces stochastic supermask sampling with an algorithm that does not sample supermasks on the forward pass and reports a significant performance boost.The authors state that this approach matches the performance of a dense network with a supermask.
- 2. Related Work: Neural Architecture Search learns architectures jointly with weights, whereas this work searches for subnetworks within fixed randomly weighted networks.The paper also relates its analysis and gradient estimator to Discovering Neural Wirings.
- 2. Related Work: Weight Agnostic Neural Networks use a shared weight value across connections, while this work uses random values and searches for supermasks within standard architectures.The supermasks found here are specific to a given random initialization rather than agnostic to weight values.
- 2. Related Work: Unlike methods using linear classifiers or pruning followed by weight modification, this work seeks high-performing subnetworks without changing any weight values.This distinction concerns the motivation and optimization setting of the proposed approach.
3. Method
The edge-popup algorithm searches for effective subnetworks by optimizing scores over fixed random weights, selecting the top-k% edges while leaving every weight unchanged. Its analysis motivates the search through overparameterization and shows that score-driven edge swaps can reduce mini-batch loss under smoothness assumptions.
- The Existence of Good Subnetworks: The infinite-width argument says that a sufficiently wide randomly initialized network contains a subnetwork close enough to a good reference network to match its accuracy, because the number of subnetworks grows rapidly.For a candidate subnetwork probability q, the probability that none of S subnetworks is suitable is described as (1 − q)^S.
- The Existence of Good Subnetworks: Unlike stochastic subnetwork sampling, deterministic top-k selection avoids repeatedly observing different subnetworks and thereby targets a more stable score-optimization procedure.The paper identifies unstable gradient estimates as a likely difficulty of the probability-based approach.
- For every fixed weight, edge-popup learns a positive score and selects the top-k% highest-scoring weights in each layer to form the subnetwork.The same k is used across layers for simplicity.
- In the fully connected formulation, node inputs are weighted sums over preceding-layer neurons, with random weights independently sampled from layer-specific distributions before optimization.The framework applies to layered networks with nonlinear activations such as ReLU.
- The method keeps all network weights at random initialization and optimizes a subnetwork instead, including no tuning of batch-normalization, first-layer, or last-layer parameters.
- The algorithm updates scores with stochastic gradient descent using a straight-through gradient estimator because the discrete top-k selection has zero gradient almost everywhere.The score update uses learning rate α and minimizes loss L.
- 3.2. The edge-popup Algorithm and Analysis: When one edge replaces another, the theorem states that mini-batch loss decreases provided the loss is sufficiently smooth and the remaining subnetwork stays fixed.The proof uses the score update and a first-order Taylor argument when the swapped input remains close to the original.
4. Experiments
Experiments evaluate edge-popup across CIFAR-10 and ImageNet, varying weight distributions, sparsity, width, depth, and comparisons with prior methods. Wider and deeper randomly weighted networks yield stronger subnetworks, including ImageNet subnetworks matching trained baselines under specific settings.
- Experimental Setup: Experiments cover CIFAR-10 VGG-like models and ImageNet ResNet-50, ResNet-101, and wide variants.All experiments optimize for 100 epochs and report validation accuracy, with specified optimizer schedules.
- Experimental Setup: The subnetwork retains the top-k% weights by learned popup scores while the random weight values remain fixed.The experiments vary k, random-weight distributions, architecture width, and depth.
- Varying the % of Weights: Best accuracy occurs for k ∈[30, 70], whereas performance worsens as k approaches 0 or 100.Very small subnetworks contain few weights, while retaining nearly all weights leaves random outputs.
- Varying the Width: As width increases, the accuracy gap between edge-popup subnetworks and trained dense models shrinks, even when parameter count is fixed.The fixed-parameter experiments attribute the improvement to a larger search space rather than only to more parameters.
- Varying the Width: On CIFAR-10, a sufficiently wide Conv6 allows a 50%-weight randomly weighted subnetwork to match the trained dense model.The experiments vary Conv4 and Conv6 widths and use width multipliers to scale layer widths.
- Comparing with Zhou et al.: Edge-popup substantially improves over Zhou et al.’s method on CIFAR-10, including when their baselines receive hyperparameter tuning.The comparison uses alternative Xavier-based and Kaiming-based distributions.
- Weight Distributions: Weight distribution strongly affects performance: xavier normal and kaiming uniform substantially reduce accuracy, while ImageNet is more sensitive than CIFAR-10.The authors also report that scaled distributions may offer better choices for ImageNet.
- ImageNet Experiments: On ImageNet, performance increases with network width and depth, but randomly weighted subnetworks do not generally match fully trained dense models.A randomly weighted Wide ResNet-50 nevertheless contains a smaller subnetwork matching trained ResNet-34 accuracy.
5. Conclusion
The paper finds compelling-accuracy subnetworks hidden within randomly weighted neural networks and presents this as a step toward understanding neural-network optimization and initialization.
- Randomly weighted neural networks contain subnetworks with compelling accuracy.
- The authors identify their findings as an avenue for further exploration.
- The work is intended to contribute to understanding neural-network optimization and initialization.
A. Table of ImageNet Results
This section presents ImageNet image-classification results corresponding exactly to Figure 8, while the preceding section describes a generalization and convolutional extension of the method.
- ImageNet image-classification results are tabulated in Table 2 and correspond exactly to Figure 8.
- The section first proves a more general case of Theorem 1.
- The section then extends edge-popup to convolutional layers and provides PyTorch code in Algorithm 1.
B.1. A More General Case of Theorem 1
The proof generalizes the loss-decrease result to swapping a nonzero number of edges in one layer while keeping the rest of the network fixed, assuming sufficient smoothness.
- When nonzero edges are swapped in one layer, the loss decreases for the mini-batch if the loss is sufficiently smooth.
- The proof compares the network inputs before and after the gradient update using I_v and Ĩ_v.
- A Taylor expansion can ignore second-order terms when the loss is smooth and Ĩ_jk is close to I_jk.
- The argument rewrites the relevant sum over E_new, edges entering the subnetwork, and E_old, edges leaving it.
- Because edge counts are conserved, |E_new| = |E_old|, and the assumed swap guarantees both sets are nonempty.
- The edge-wise inequality for every entering and leaving edge pair yields the required equation after matching the equal-sized edge sets.
B.2. Extension to Convolutional Neural Networks
The convolutional extension represents spatially indexed connections with kernel-sized edge sets, applies top-k% score selection, and implements the procedure in PyTorch.
- B.2. Extension to Convolutional Neural Networks: Convolutions are written in a form resembling Equation 2 using an odd kernel size κ over spatial locations w and h.
- B.2. Extension to Convolutional Neural Networks: In the convolutional case, inputs and outputs are two-dimensional, with batch normalization often followed by ReLU and implicit zero padding.
- B.2. Extension to Convolutional Neural Networks: The convolutional edge-popup algorithm assigns scores indexed by kernel coordinates and retains the top k% during the forward pass.
- B.2. Extension to Convolutional Neural Networks: When κ = W = H = 1, the convolutional formulation is equivalent to the original edge-popup formulation.
- B.2. Extension to Convolutional Neural Networks: The convolutional score update sums over all spatial locations w and h.
- B.2. Extension to Convolutional Neural Networks: Algorithm 1 implements the convolutional procedure in PyTorch using GetSubnet to sort scores and select the retained weights.
- B.2. Extension to Convolutional Neural Networks: The backward pass sends gradients straight through the subnet-selection function, while gradients to the fixed weights and bias are disabled.
- B.2. Extension to Convolutional Neural Networks: Figure 11 repeats the Figure 3 experiments with ResNet18 on CIFAR.
C.1. Resnet18 on CIFAR-10
The section experiments with a more advanced network architecture on CIFAR-10.
- The experiment evaluates a more advanced network architecture on CIFAR-10.
C.2. Are these subnetworks lottery tickets?
Training the discovered subnetworks on ImageNet does not substantially improve their accuracy over random subnetworks or reach dense-network accuracy. These results suggest that their strong initialization-time performance does not explain the lottery phenomena described in [5].
- The discovered subnetworks do not train to the same accuracy as a dense network on ImageNet.
- These results suggest that the subnetworks' good initialization-time performance does not explain the lottery phenomena described in [5].
- Their trained accuracy is not substantially better than that of a random subnetwork.