How far can we go without convolution: Improving fully-connected networks

1 INTRODUCTION

• Fully connected networks provide a test bed for architectural choices beyond convolution, including tasks without local translation invariance such as recurrent-network hidden layers.The motivation includes that convolutional weight sharing requires long-range communication during learning and does not map neatly to simple, densely parallel hardware.Source: §1 · p. 1
• The authors identify linear bottleneck layers and pre-training with autoencoders whose hidden units have no biases as two choices with a strong impact on performance.Together, they yield state-of-the-art performance on permutation-invariant CIFAR-10 and performance close to CNNs when training data include deformations.Source: §1 · p. 1, §1.1 · p. 2

1.1 SPARSITY IN NEURAL NETWORKS

• In deep networks, sparse activations can impede optimization because zero or near-zero activations in common functions coincide with zero derivatives.PReLU and Maxout reduce zero-derivative regimes but give up sparsity, motivating alternatives that preserve sparsity while allowing gradient flow.Source: §1.1 · p. 2
• The paper frames bias-free autoencoder pre-training as a way to orthogonalize subsets of network weights and improve performance.The motivation is that sparse ReLU or sigmoid units typically learn strongly negative biases that may be suboptimal for hidden representations.Source: §1.1 · p. 2

2 LINEAR BOTTLENECK LAYERS

• Sparsity in the hidden representation increases with network depth and rises further with dropout in the tested CIFAR-10 MLPs.In a fully connected layer, a unit that is off 80% of the time effectively sees only 20% of the training data, without weight-sharing compensation.Source: §2 · p. 2, Figure 1 · p. 3

2.1 LINEAR LAYERS AS DISTRIBUTION RESHAPING

• As input dimension grows, a linear unit’s output tends toward a Gaussian distribution regardless of its weights; ReLU then sets negative values to zero.Source: §2.1 · p. 3
• For zero-bias ReLU, half of the probability mass concentrates at zero, while the bias controls the zero-spike intensity for ordinary ReLU.The figure compares ReLU output (left) with zero-bias ReLU output (right).Source: §2.1 · p. 4, Figure 2 · p. 4

2.2 DERIVATIVES IN THE PRESENCE OF LINEAR LAYERS

• Although at least 50% of ReLU representations are typically zero, inserting a linear layer makes the equivalent ReLU-layer update dense.The linear representation is dense, and the product and cross-terms in the equivalent update yield a dense matrix.Source: §2.2 · pp. 4–5

2.3 REDUCING PARAMETERS

• A linear layer with L units between two nonlinear layers with N units uses 2NL+L+N parameters, versus N^2+N for direct connections.Source: §2.3 · p. 5

3 PRE-TRAINING AND ZERO-BIAS ACTIVATIONS

• Earlier work proposed that unsupervised pre-training could produce more orthogonal weights and reduce vanishing gradients, but sparsity-inducing negative biases can make reconstructions affine rather than linear.The authors describe this as a possible reason those pre-training schemes had limited practical success relative to supervised backpropagation.Source: §3 · p. 5
• Orthogonality should be sought among units active on the same inputs, since sparsity can make an example’s effective weight matrix non-square.This motivates seeking “orthogonal active paths” rather than simply orthogonal weight matrices.Source: §3 · p. 5
• Zero-bias autoencoders threshold activations during reconstruction pre-training, then remove thresholds when using the weights for classification or network initialization.Under linear encoding, squared-error reconstruction is expected to encourage orthogonality among weights of units that tend to activate together; earlier single-hidden-layer MLP tests showed only weak improvement over standard autoencoders.Source: §3 · p. 5
• Zero-bias autoencoder pre-training yields a separate, significant performance improvement in fully connected networks, including when combined with interleaved bottlenecks.The authors investigate its use in networks with many hidden layers.Source: §3 · p. 5

4.1 CIFAR-10

• On CIFAR-10, ReLU-Lin reached 56.84% and Z-Lin reached 65.76%, outperforming the other models in their respective comparisons.The ReLU-Lin network was also more stable than pure ReLU networks during layer-wise pre-training.Source: §4.1 · p. 6
• On the same network architecture, bottleneck layers performed better than PReLU and Maxout, while their improvements were about the same as batch normalization.Source: §4.1 · pp. 6–7

4.2 THE HIGGS DATASET

• On HIGGS, bottleneck layers improved performance less than on CIFAR-10, while zero-bias units yielded no improvement.The compared models had the same architecture and were trained without pre-training.Source: §4.2 · p. 7

4.3 REDUCING PARAMETERS

• A linear layer of size 100 retained reasonable accuracy with 1/20 of the parameters, while the smallest model had around 2.44 million parameters.The passage compares this with around 112 million parameters for a seven-layer, 4000-unit fully connected network and around 3.5 million for a typical convolutional network achieving over 80% on CIFAR-10.Source: §4.3 · p. 7

4.4 ACHIEVING STATE-OF-THE-ART ON PERMUTATION INVARIANT CIFAR-10

• 65.7% accuracy on permutation-invariant CIFAR-10 exceeded the next best published results, 63.1% and 63.9% (Konda et al., 2014).This result followed pre-training and fine-tuning.Source: §4.4 · p. 8
• 69.62% accuracy on permutation-invariant CIFAR-10 was achieved by a deep Z-LIN network trained with dropout, exceeding the reported state of the art by a large margin.The network used three Z-LIN pairs, a ZAE layer, and a logistic regression classifier.Source: §4.4 · p. 8
• 78.62% test accuracy on CIFAR-10 was achieved by a fully connected Z-LIN network trained with flipping, rotation, and shifting, about 10% behind a well-trained convolutional network.This setting gives up permutation invariance.Source: §4.4 · p. 8, Figure 6 · p. 8

• Earlier linear bottlenecks were used in autoencoders, network factorization, and high-dimensional output layers, but not to alleviate vanishing gradients and sparsity throughout a network.Those methods used a single bottleneck layer.Source: §5 · p. 8
• Ba & Caruana (2014) used linear bottlenecks to factorize a single-layer network and speed learning, whereas this paper targets gradient flow and sparsity.Source: §5 · p. 8
• Linear bottleneck layers and batch normalization both adjust inputs to a subsequent layer, but bottlenecks reshape the whole distribution rather than only its mean and variance.Source: §5 · p. 8

6 DISCUSSION

• A single hidden layer can model any non-linear function under mild conditions by dividing space into half-spaces and linearly combining the resulting regions.This perspective suggests defining nonlinear functions with a non-linear layer followed by a linear layer, interleaved through the network.Source: §6 · p. 9
• Approximating any given function with a single hidden layer requires an exponentially large number of hidden units, limiting the practical usefulness of the result.The discussion presents multilayer networks as a tractable alternative, computing a sequence of more restricted nonlinear functions.Source: §6 · p. 9
• With enough training data and computational resources, wider hidden layers would still be preferable, but in practice they entail more sparsity that prevents derivative flow.As sparse activations propagate upward, they tend to proliferate and aggravate the problem in higher layers.Source: §6 · p. 9