Source-linked AI summary
Understanding Deep Architectures using a Recursive Convolutional Network
David Eigen, Jason Rolfe, Rob Fergus, Yann LeCun
TL;DR
Convolutional networks are difficult to size because architectural choices such as layer count, feature-map dimensions, and parameter count are interrelated. The paper uses a recursive convolutional model with tied higher-layer weights to compare these factors, finding that layers and parameters matter while feature-map count has little effect once parameter count is considered.
Problem
Convolutional architectures have interrelated sizing choices, and the independent effects of layer count, feature-map count, and parameter count are difficult to assess.
Source: §1 · p. 1Method
Results
Layer and parameter counts affect performance, while feature-map count has little effect once parameter count is taken into account.
Source: §4 · p. 8Takeaways & Limitations
In these convolutional experiments, adding layers can improve performance without adding parameters, and parameter allocation across multiple layers can help even with fewer feature maps.
Source: §3.2.1 · p. 5, §4 · p. 8The study uses a special tied architecture and simplified models with one pooling layer; behavior at L > 8 or M < 16 and applicability to more complex architectures remain uncertain.
Source: §4 · p. 8
Abstract
from arXiv · showhide
A key challenge in designing convolutional network models is sizing them appropriately. Many factors are involved in these decisions, including number of layers, feature maps, kernel sizes, etc. Complicating this further is the fact that each of these influence not only the numbers and dimensions of the activation units, but also the total number of parameters. In this paper we focus on assessing the independent contributions of three of these linked variables: The numbers of layers, feature maps, and parameters. To accomplish this, we employ a recursive convolutional network whose weights are tied between layers; this allows us to vary each of the three factors in a controlled setting. We find that while increasing the numbers of layers and parameters each have clear benefit, the number of feature maps (and hence dimensionality of the representation) appears ancillary, and finds most of its benefit through the introduction of more weights. Our results (i) empirically confirm the notion that adding layers alone increases computational power, within the context of convolutional layers, and (ii) suggest that precise sizing of convolutional feature map dimensions is itself of little concern; more attention should be paid to the number of parameters in these layers instead.
1 Introduction
The paper asks how to size convolutional networks when architectural factors are interrelated, and studies layers, feature maps, and parameters separately. It situates its recursive model alongside recurrent, auto-encoder, and other recursive approaches to image modeling.
1 Introduction
- The study isolates the effects of layer count, feature-map count, and parameter count because these interrelated choices make convolutional architectures difficult to size.The authors note that designing such models relies on intuition and extensive validation-set trial and error.Source: §1 · p. 1
- The recursive model enables controlled comparisons by matching layer and parameter counts or feature-map and parameter counts across architectures.The comparisons help separate the effects of additional nonlinearities from model capacity.Source: §1 · p. 1
- Stacks of unpooled convolutional layers are important to characterize because prior work found substantial performance changes when convolutional layers were altered.In Zeiler and Fergus, removing two middle convolutional layers reduced performance by 4.9%, while expanding them improved it by 3.0%.Source: §1 · p. 2
1.1 Related Work
- The proposed network is related to recurrent image models and is described as a purely discriminative, convolutional version of LISTA or DrSAE.Unlike the cited image-segment models, this network has the same input and output dimension; prior work also used unrolled inference in convolutional Deep Belief Networks and Multi-Prediction Deep Boltzmann Machines.Source: §1.1 · p. 2
2 Approach
The approach ties weights across higher convolutional layers, making repeated application of one nonlinear transformation equivalent to recurrence. The CIFAR-10 and SVHN instantiations use a pooled first layer followed by unpooled hidden layers and a classifier.
2 Approach
- The model uses equal-sized hidden layers with ReLU activations, and denotes feature maps per layer by M and higher-layer count by L.The first layer uses kernels V, later layers use W_l, and a final classification matrix maps the last hidden layer to softmax inputs.Source: §2 · p. 2
- Tying higher-layer parameters makes the deep convolutional network recurrent: each layer repeatedly applies the same nonlinear transformation.The architecture also ties parameters across spatial positions through convolution, and is reminiscent of LISTA without direct input projections to each hidden layer.Source: §2 · p. 2, Figure 1 · p. 3
2.1 Instantiation on CIFAR-10 and SVHN
- For CIFAR-10 and SVHN, 32 × 32 × 3 images pass through an 8 × 8 first-layer convolution and 4 × 4 max-pooling, yielding an 8 × 8 × M hidden layer.The first convolution uses stride one and same padding; subsequent layers preserve the hidden-layer size with 3 × 3 × M kernels, same convolution, bias, and ReLU.Source: §2.1 · p. 3
- In the tied model, higher-layer kernels and biases are shared, and the final hidden layer is pixel-wise L2-normalized before logistic classification.Higher-layer kernels are initialized to the identity transformation; training minimizes logistic loss with minibatches of 128, learning rate 10^-3, and momentum 0.9.Source: §2.1 · p. 3
3 Experiments
The experiments compare convolutional networks while varying layers, parameters, and feature maps independently. More layers and parameters generally improve performance, while changing feature-map counts at fixed depth and parameter count has little effect.
3.1 Performance Evaluation
- The tied and untied models achieve competitive performance on CIFAR-10 and SVHN against approaches without data transformations or stochastic regularization.The overview also reports that increasing layers, feature maps, or parameters tends to improve performance, particularly on the training set.Source: §3.1 · p. 4, Table 1 · p. 4
3.2 Effects of the Numbers of Feature maps, Parameters and Layers
- Tying weights across higher layers makes it possible to vary layers, parameters, and feature maps independently in three controlled comparisons.The comparisons hold fixed M and P while varying L, fixed M and L while varying P, or fixed P and L while varying M.Source: §3.2 · p. 5
3.2.1 Case 1: Number of Layers
- Adding layers generally improves performance even at fixed parameter count, though CIFAR-10 test error rises at eight layers in the tied model and with eight-layer untied models.The tied-model results show an upward tick in CIFAR-10 test error at eight layers despite declining training error; the untied-model exception is attributed to overfitting.Source: §3.2.1 · p. 5, Figure 4 · p. 6, Figure 5 · p. 6
3.2.2 Case 2: Number of Parameters
- With feature maps and layers held constant, untied models generally outperform tied models, but their test performance converges as both models become sufficiently flexible.The comparison attributes the untied models’ advantage to their greater number of parameters and notes overfitting at the largest layer and feature-map counts.Source: Figure 6 · p. 7, §3.2.2 · p. 7
3.2.3 Case 3: Number of Feature Maps
- At fixed parameter count and depth, tied and untied models perform about the same despite having different numbers of feature maps.The model pairs were sampled with parameter counts within 1.0% of each other.Source: §3.2.3 · pp. 7–8, Figure 7 · p. 8
4 Discussion
The discussion concludes that layer count and parameter count matter more to convolutional-layer performance than feature-map dimension, and that parameters may be best allocated across multilayer stacks. These conclusions are qualified by the tied architecture and simplified pooling setup.
4 Discussion
- Feature-map dimension has little effect once parameter count is accounted for, while layer and parameter counts each affect performance.The authors report that higher-dimensional representations did not increase performance in their experiments.Source: §4 · p. 8
- With a fixed parameter count, allocating parameters across multiple layers tends to improve performance over using fewer layers, despite reducing feature-map dimension.This pattern is consistent with feature-map count having little effect relative to layer count.Source: §4 · p. 8
- The findings come from a special tied architecture and a simplified setup with one pooling layer; their application to more complex cases remains uncertain.Models with L > 8 or M < 16 were mostly unexamined.Source: §4 · p. 8
- Increasing layers alone can improve convolutional-layer performance, and filter parameters may be best allocated across multilayer stacks, even with fewer feature maps.Feature-map dimension itself has little effect, with most sizing effects coming from layer and parameter counts.Source: §4 · p. 8