Source-linked AI summary
Deep Fried Convnets
Zichao Yang, Marcin Moczulski, Misha Denil, Nando de Freitas, Alex Smola, Le Song, Ziyu Wang
TL;DR
Deep convolutional networks need parameter reductions because fully connected layers dominate storage, especially for memory-constrained deployment. This paper replaces those layers with an end-to-end trainable Adaptive Fastfood transform, producing deep fried convnets that preserve predictive performance while substantially reducing parameters. The approach attains approximately half the parameters of standard convolutional networks on ImageNet, although convolution-dominated computation means it is not necessarily faster in practice.
Problem
Fully connected layers dominate the parameter storage of convolutional networks, motivating reductions that preserve predictive performance for memory-constrained deployment.
Method
The paper replaces fully connected layers with an end-to-end trainable Adaptive Fastfood transform integrated with convolutional layers.
Results
Deep fried convnets achieve the same predictive performance as standard convolutional networks on ImageNet using approximately half the parameters.
Takeaways & Limitations
The approach substantially reduces parameters on MNIST and ImageNet without sacrificing predictive performance and compares favorably with simple test-time low-rank factorization.
Takeaways & Limitations
Because convolution dominates computation in convolutional neural networks, deep fried convnets are not necessarily faster in practice.
Abstract
from arXiv · showhide
The fully connected layers of a deep convolutional neural network typically contain over 90% of the network parameters, and consume the majority of the memory required to store the network parameters. Reducing the number of parameters while preserving essentially the same predictive performance is critically important for operating deep neural networks in memory constrained environments such as GPUs or embedded devices. In this paper we show how kernel methods, in particular a single Fastfood layer, can be used to replace all fully connected layers in a deep convolutional neural network. This novel Fastfood layer is also end-to-end trainable in conjunction with convolutional layers, allowing us to combine them into a new architecture, named deep fried convolutional networks, which substantially reduces the memory footprint of convolutional networks trained on MNIST and ImageNet with no drop in predictive performance.
1. Introduction
Deep fried convnets replace fully connected layers with an end-to-end trainable Adaptive Fastfood transform, reducing parameters while preserving predictive performance. The approach addresses memory and training-efficiency concerns that limit deployment and distinguishes itself from pooling, pruning, hashing, and fixed-kernel alternatives.
- Millions- and billions-parameter convolutional networks motivate reductions for more efficient distributed training and embedded deployment.
- Fully connected layers contain most network parameters, whereas convolutional layers account for most computation.
- Adaptive Fastfood replaces fully connected layers as a generalization of Fastfood kernel approximation and supports end-to-end training with convolutional layers.
- Deep fried convnets achieve the same ImageNet predictive performance as standard convolutional networks using approximately half the parameters.
- Global average pooling can reduce model size but makes feature transfer more difficult and may increase evaluation time through additional convolutions.
- Unlike pruning and sparsity methods, the approach does not require training the original full model, while fixed-kernel methods cannot jointly learn convolutional filters.
2. The Adaptive Fastfood Transform
The Adaptive Fastfood transform reparameterizes dense matrix multiplication using structured modules, replacing O(nd) storage and computation with substantially lower costs. Its components support efficient forward and backward passes, and stacked transforms handle wider output layers.
- Dense layer propagation uses a matrix with d input activations and n output activations, incurring O(nd) storage and computation.
- The proposed reparameterization replaces W with an Adaptive Fastfood transform requiring O(n) storage and O(n log d) computation.
- Adaptive Fastfood combines diagonal parameter matrices S, G, and B with a permutation matrix Π and a Walsh-Hadamard matrix H.
- For square transforms, the overall costs are O(d) storage and O(d log d) computation instead of O(d2) for ordinary fully connected layers.
- When n exceeds d, multiple transforms can be stacked to obtain O(n) storage and O(n log d) computation, with output size refinable by pruning.
- The learned parameters S, G, and B use standard backpropagation, while Hadamard, permutation, and diagonal operations keep backward computation efficient.
3. Intuitions behind Adaptive Fastfood
Adaptive Fastfood connects structured random projections and kernel feature approximations to create trainable neural-network modules with lower storage and computation costs.
- 3. Intuitions behind Adaptive Fastfood: Adaptive Fastfood is a trainable structured random projection and an approximation to the feature space of a learned kernel.Its diagonal matrices are learned by backpropagation, adapting feature relevance, kernel bandwidth and spectral incoherence, and kernel type.
- 3. Intuitions behind Adaptive Fastfood: Random projections preserve metric information, but storing their dense matrix costs O(nd), while sparse matrices can have high-estimate variance for sparse inputs.Hashing reduces storage and computation but has irregular memory access patterns that may hinder GPU performance.
- 3. Intuitions behind Adaptive Fastfood: Fastfood replaces dense random matrix multiplication with structured transforms, reducing random-projection computation to O(n log d) and storage to O(n).The transform uses diagonal matrices, a Hadamard matrix, and a permutation; its original diagonal parameters are random and stored once.
- 3. Intuitions behind Adaptive Fastfood: Kernel-feature duality allows kernel methods to be mapped into neural-network modules, while Fastfood substitutes SHGΠHBx for Wx to reduce costs.The paper also relates ReLU features to the rotationally invariant arc-cosine kernel.
- 3. Intuitions behind Adaptive Fastfood: Random features approximate kernels by sampling vectors from a distribution and applying feature nonlinearities whose inner products represent the implicit kernel.For shift-invariant kernels, Bochner’s Lemma links the kernel to a spectral density; squared exponential kernels use Gaussian feature weights.
4. Deep Fried Convolutional Networks
Deep fried convolutional networks replace fully connected layers with an Adaptive Fastfood transform followed by a nonlinearity, while retaining the standard convolutional and pooling layers.
- 4. Deep Fried Convolutional Networks: Deep fried convolutional networks replace fully connected layers with an Adaptive Fastfood transform followed by a nonlinearity.The architecture is illustrated in Figure 1.
- 4. Deep Fried Convolutional Networks: The method focuses on internal fully connected layers rather than the softmax classifier, whose memory reduction is already addressed by low-rank factorization.Prior results cited by the paper found low-rank factorization performed poorly for internal layers.
5. MNIST Experiment
The MNIST experiment evaluates jointly trained deep fried networks against a LeNet reference, showing large parameter reductions while maintaining high accuracy; adaptation provides no benefit in this setting.
- 5. MNIST Experiment: MNIST serves as an easy proof of concept and a scale contrast for evaluating Adaptive Fastfood behavior.All layers, including convolutional layers, are jointly trained from scratch with adaptive and non-adaptive transforms using 1024 or 2048 features.
- 5. MNIST Experiment: 0.87% error is achieved by the reference Caffe LeNet convolutional network on MNIST.This is the experiment’s reference model.
- 5. MNIST Experiment: 11 times reduction in parameters is achieved by the best deep fried MNIST model while retaining high accuracy.Because the deep fried network is wider than the reference, dropout was also tested and increased performance in the deep fried case.
- 5. MNIST Experiment: Adaptive transforms perform equivalently or worse than non-adaptive transforms on MNIST.The paper contrasts this result with its later ImageNet experiments.
6. Imagenet Experiments
ImageNet experiments replace fully connected layers with Fastfood transforms in both fixed-feature and jointly trained settings. Deep fried networks substantially reduce parameters while matching or exceeding reference performance.
- ImageNet uses 1.2M training examples, 50K validation examples, and 1000 classes.
- The reference Caffe model has 58.7M parameters, with 99.9% in fully connected layers.
- Fixed feature extractor: In fixed-feature experiments, deep fried networks outperform the retrained MLP and prior results while using fewer parameters.
- Fixed feature extractor: Adaptive Fastfood improves top-1 performance by 4.5-6.5% over the non-adaptive version.
- Jointly trained model: With 16,384 Fastfood features, parameters fall from 58.7M to 16.4M, a 3.6x reduction, with less than 0.3% top-1 validation loss.
- Jointly trained model: With 32,768 features, the model performs 0.6% better than the reference while using approximately half as many parameters.
- Jointly trained model: More than 99% of deep fried network parameters remain in the dense final softmax layer because ImageNet has many classes.
7. Comparison with Post Processing
The paper compares training compressed networks from scratch with post-processing methods such as SVD, pruning, and sparsity regularization. Deep fried convnets achieve better accuracy-parameter trade-offs than the evaluated alternatives.
- Post-processing methods compress a fully trained model, whereas the proposed method trains the compressed network from scratch.
- SVD factorizes fully connected weight matrices into lower-rank factors, reducing storage when the retained rank k is sufficiently small.
- Directly training a factorized representation can perform poorly, while SVD followed by fine-tuning preserves much of the original performance.
- SVD-half and SVD-quarter cause 1% and 3.5% accuracy drops, respectively.
- Fine-tuning reduces the drops to 0.1% for SVD-half-F and 1.3% for SVD-quarter-F, but deep fried convnets still perform better in accuracy and parameter count.
- Rank-600 SVD on the final softmax removes 12.5M additional parameters at the expense of approximately 0.7% top-1 error.
- Sparse representations require index storage and are advantageous over dense matrices only when the number of nonzero entries is small.
8. Conclusion
The paper emphasizes train-time parameter reduction through end-to-end differentiable Adaptive Fastfood transforms, while noting that convolutional computation can limit practical speedups. Experiments also identify adaptation benefits on ImageNet and the output softmax as a remaining compression target.
- End-to-end differentiability enables parameter reductions during training, unlike approaches focused only on test-time network compression.
- Deep fried convnets substantially reduce parameters without sacrificing predictive performance on MNIST and ImageNet, and compare favorably with simple test-time low-rank factorization.
- On ImageNet, adaptive weights provide substantially better performance than structured random transformations, revealing a benefit not visible on small datasets alone.
- Theoretical computation falls from O(nd) to O(n log d), but convolutional layers dominate computation, so deep fried convnets are not necessarily faster in practice.
- ImageNet results indicate that remaining parameters are mostly in the output softmax layer, motivating compression with low-rank decomposition, Adaptive Fastfood, or pruning.