Source-linked AI summary
Predicting Parameters in Deep Learning
Misha Denil, Babak Shakibi, Laurent Dinh, Marc'Aurelio Ranzato, Nando de Freitas
TL;DR
The paper addresses redundancy in deep-network parameterizations and the inefficiency of communicating many dynamic parameters during distributed training. It factors weight matrices and predicts unlearned values from a smaller learned subset, showing that more than 95% of parameters can be predicted without an accuracy drop in the best cases. The approach reduces dynamic parameters while remaining complementary to other neural-network advances.
Problem
Distributed training is inefficient because coordination overhead grows with the number of machines, motivating methods that reduce learned and communicated parameters.
Method
The method exploits structured weights by representing W as a low-rank product W = UV, fixing one factor and learning the other so many parameters can be predicted.
Results
More than 95% of a network’s parameters can be predicted without any drop in predictive accuracy in the best cases.
Takeaways & Limitations
The approach achieves significant reductions in dynamic parameters and is orthogonal but complementary to dropout, rectified units, and maxout.
Takeaways & Limitations
The study uses uniformly random parameter-index selection and does not exhaustively investigate kernel choices.
Abstract
from arXiv · showhide
We demonstrate that there is significant redundancy in the parameterization of several deep learning models. Given only a few weight values for each feature it is possible to accurately predict the remaining values. Moreover, we show that not only can the parameter values be predicted, but many of them need not be learned at all. We train several different architectures by learning only a small number of weights and predicting the rest. In the best case we are able to predict more than 95% of the weights of a network without any drop in accuracy.
1 Introduction
The paper targets inefficient distributed training by reducing the number of dynamic neural-network parameters through prediction based on structure in learned weights. Its technique learns only a small subset of parameters and predicts the rest, achieving over 95% parameter prediction without an accuracy drop in the best cases.
- Motivation: Distributed training becomes less efficient as coordination overhead grows with the number of machines.Using 81 machines reduced training time per mini-batch by 12×, while 128 machines achieved roughly 14× speedup.
- Motivation: Reducing the parameters learned and communicated can reduce the machines required and the associated coordination overhead.
- Contribution: The method exploits structure in learned weights to reduce the number of free neural-network parameters and can complement dropout, rectified units, and maxout.
- Motivation: Smooth, locally edge-like first-layer features make separately storing every input weight redundant because neighboring values can often be predicted from one another.
- Result: More than 95% of a network’s parameters can be predicted without any drop in predictive accuracy in the best cases.
2 Low rank weight matrices
The paper reduces neural-network parameterization by factoring each weight matrix into two smaller matrices. Fixing one factor removes factorization redundancy, leaving the other factor to be learned while controlling parameter count through the intermediate rank.
- Low rank weight matrices: A weight matrix W is represented as W = UV, where U has size nv × nα and V has size nα × nh.Choosing nα much smaller than nv and nh substantially reduces the number of parameters.
- Low rank weight matrices: Naively learning both factored matrices can perform worse than directly learning a full-rank weight matrix.
- Low rank weight matrices: Because W = UV = (UQ)(Q−1V) for any invertible Q, the factorization is redundant; fixing U and learning only V removes this redundancy.
3 Feature prediction
Feature prediction represents neural-network weights through a smaller dictionary-based parameterization, using selected weights to predict complete features. The approach supports multiple dictionaries or columns and can also be interpreted as fixed linear pooling before an ordinary layer.
- Feature prediction: A feature’s weights are modeled as a function over weight space and represented in a lower-dimensional basis-function dictionary.The columns of U form the dictionary, while V provides the coefficients for linear combinations of its basis features.
- Choice of dictionary: Dictionaries can be constructed from unsupervised features, prior-informed Fourier or wavelet bases, or kernels encoding expected structure.Unsupervised dictionaries require an additional training phase, whereas Fourier and wavelet bases directly encode smoothness assumptions for image features.
- Choice of dictionary: Kernel ridge regression predicts the full parameter vector from weights observed at a restricted subset of locations.The kernel models covariance between observed locations and produces smooth predictions over the entire domain, with λ controlling ridge regularization.
- A concrete example: For image patches, selected pixel locations provide explicit weights, while a squared exponential kernel controls smoothness when predicting the remaining weights.The selected index set α corresponds to pixel locations and dictionary basis functions; its selection may be random or carefully designed.
- Alternative interpretation: The predicted layer can be viewed as fixed linear pooling followed by an ordinary fully connected layer with |α| visible units.This follows by reordering the computation from vUαWα to vαWα, where vα is the data after applying the fixed pooling operator.
- Columnar architecture: Multiple index sets create independent columns whose outputs are concatenated, and the same columnar idea extends from fully connected to convolutional networks.Adding columns increases static dictionary parameters but leaves dynamic parameters fixed for a fixed-size hidden layer; convolutional filter banks must be reshaped after vectorized prediction.
4 Experiments
Experiments across MLPs, convnets, and RICA show that parameter prediction can reduce learned parameters while retaining performance. The approach uses architecture-appropriate dictionaries and predicts parameters in selected layers or features.
- Multilayer perceptron: MLP experiments on MNIST varied dictionary construction, dictionary width, and dynamic-parameter reduction across 784–500–500–10 sigmoid networks.Parameter prediction was applied only to the first two layers; the final softmax layer was retained and contained approximately 1% of total parameters.
- Multilayer perceptron: SE–Emp and SE-Emp2 dictionaries substantially outperformed the alternatives, especially with few dynamic parameters.The compared alternatives included LowRank, random-connection and random-projection dictionaries, kernel regressions, and an autoencoder dictionary.
- Convolutional network: 25% of the parameters learned in a CIFAR-10 convnet had a negligible effect on predictive accuracy.The convolutional layers used squared exponential kernel dictionaries, and the fully connected layer was predicted with squared exponential-kernel ridge regression.
- Reconstruction ICA: RICA predicted more than half of its dynamic parameters on CIFAR-10 and STL-10 without a substantial drop in accuracy.The experiments used a squared exponential kernel dictionary with length scale 1.0.
- Reconstruction ICA: With the same number of dynamic parameters, RICA with 50% parameter prediction used twice as many features as ordinary RICA.The comparison evaluated ordinary RICA against a model predicting 50% of the parameters in each feature.
5 Related work and future directions
The paper positions parameter prediction as complementary to existing parameter-reduction methods and identifies kernel choice, index selection, topology, and deeper columns as directions for further work.
- Related work: Unlike post-training pruning such as Optimal Brain Damage, this technique limits the number of parameters before training.The paper also distinguishes its deep-model focus from prior shallow dictionary and filter-bank approximations.
- Related work: Parameter prediction is orthogonal to weight tying in tiled or convolutional networks: tying reduces feature maps, whereas this method reduces parameters within each feature.The paper states that the two approaches can be incorporated together.
- Related work: The technique subsumes random connections and remains applicable alongside other neural-network parameterizations and training advances.The paper describes it as broadly applicable, including to tiled or convolutional networks and factored RBMs.
- Future directions: Deeper independent columns could potentially increase parallelism, but the paper does not study deep columns.The authors note that columns operate completely independently and identify deeper columns as future work.
- Future directions: Uniformly random selection of the index set α is used throughout, while alternative selection strategies remain possible.The paper specifically mentions incorporating methods that learn receptive fields directly.
- Future directions: The study of kernel functions is not exhaustive, motivating kernels that encode different prior knowledge or are learned during optimization.The authors describe these as possibilities for future exploration.
- Future directions: When weight-space topology is unavailable, the dictionary topology is inferred from empirical statistics; directly inducing a desired topology is another open direction.The paper connects this possibility to work on topology in representations and learned pooling structures.
6 Conclusion
The paper shows that deep models can achieve significant reductions in dynamic parameters through parameter prediction. It presents the technique as complementary to advances such as dropout, rectified units, and maxout, while questioning established neural-network parameterizations.
- Conclusion: The paper achieves significant reductions in the number of dynamic parameters in deep models.This is presented as the central conclusion of the work.
- Conclusion: Parameter prediction is orthogonal but complementary to dropout, rectified units, and maxout.The paper identifies improved large-scale industrial implementations as one avenue for future work.
- Conclusion: The results raise the question of whether current deep-learning parameterizations are appropriate.The conclusion frames this as a broader implication of reducing dynamic parameters.