Source-linked AI summary
ReNet: A Recurrent Neural Network Based Alternative to Convolutional Networks
Francesco Visin, Kyle Kastner, Kyunghyun Cho, Matteo Matteucci, Aaron Courville, Yoshua Bengio
TL;DR
Object recognition is dominated by CNNs, while RNNs are primarily used for sequential data. ReNet replaces CNN convolution+pooling layers with four directional RNN sweeps that provide whole-image context, and performs comparably to CNNs on MNIST, CIFAR-10, and SVHN. The paper presents ReNet as a viable alternative while identifying areas requiring further investigation.
Problem
CNNs are the established approach for object recognition, motivating investigation of whether recurrent architectures can provide a viable alternative for image-related tasks.
Method
ReNet replaces each CNN convolution+pooling layer with four coupled uni-dimensional RNNs sweeping vertically and horizontally in both directions across image features.
Results
ReNet performs comparably to CNNs on the MNIST, CIFAR-10, and SVHN object-recognition benchmarks.
Takeaways & Limitations
ReNet is a viable alternative to CNNs for image-related tasks, including tasks where CNNs have historically dominated.
Takeaways & Limitations
The study did not conduct extensive hyperparameter search and focused on a small hyperparameter set to demonstrate the model's potential.
Abstract
from arXiv · showhide
In this paper, we propose a deep neural network architecture for object recognition based on recurrent neural networks. The proposed network, called ReNet, replaces the ubiquitous convolution+pooling layer of the deep convolutional neural network with four recurrent neural networks that sweep horizontally and vertically in both directions across the image. We evaluate the proposed ReNet on three widely-used benchmark datasets; MNIST, CIFAR-10 and SVHN. The result suggests that ReNet is a viable alternative to the deep convolutional neural network, and that further investigation is needed.
1 Introduction
ReNet replaces CNN convolution+pooling layers with coupled one-dimensional RNN sweeps, providing whole-image context while reducing architectural complexity relative to multidimensional RNNs. On MNIST, CIFAR-10, and SVHN, it performs comparably to CNNs.
- Architecture: ReNet replaces each convolution+pooling layer with four RNNs sweeping vertically and horizontally in both directions.The sweeps operate bottom-to-top, top-to-bottom, left-to-right, and right-to-left over lower-layer features.
- Architecture: Each ReNet activation incorporates the whole image rather than only a local convolutional context window.The lowest layer sweeps over the input image, while later layers process hierarchical representations from preceding layers.
- Relation to prior work: ReNet uses coupled uni-dimensional RNNs instead of a multidimensional RNN.This design lets ordinary sequence RNNs produce activations computed with respect to the whole input image.
- Relation to prior work: The number of RNNs scales linearly as 2d with input dimensionality d, unlike the exponential requirement of multidimensional RNNs.The architecture is also described as more amenable to distributed computing because its RNNs depend along horizontal or vertical sequences.
- Evaluation: ReNet performs comparably to CNNs on MNIST, CIFAR-10, and SVHN, suggesting RNNs can be competitive alternatives for image-related tasks.These are widely used object-recognition benchmarks.
2 Model Description
ReNet transforms image patches through bidirectional recurrent sweeps, first vertically and then horizontally, producing features that incorporate increasingly broad spatial context. Multiple recurrent transformations can be stacked before classification, with parameters learned by backpropagation-based stochastic gradient descent.
- Input patching: The input image or feature map is divided into non-overlapping patches, which recurrent networks process sequentially.The passage defines patches over the spatial dimensions of the input and assigns each patch a grid location.
- Vertical sweep: Two RNNs sweep vertically in opposite directions, processing one flattened patch at a time along each input-image column.One RNN runs bottom-up and the other top-down, updating hidden states for each column.
- Vertical sweep: Concatenated vertical hidden states form a feature map whose vectors summarize each column's patches using 2d recurrent features.Each resulting vector acts as a feature detector at a patch location with respect to all patches in that column.
- Horizontal sweep: Two further RNNs sweep horizontally across the vertical feature map, yielding vectors that represent each patch in the context of the whole image.The horizontal sweep proceeds along each row and produces the output feature map H.
- Deep architecture and training: The recurrent transformation can be stacked to increase depth, after which the final activations are flattened and passed to a differentiable classifier.The network parameters, including recurrent and fully connected layers, are estimated with stochastic gradient descent using backpropagation.
3 Differences between LeNet and ReNet
ReNet differs from LeNet through image-wide lateral connections, no required pooling, and end-to-end differentiability, while recurrent sequencing limits model parallelism. Its design may reduce parameters and computation, but this remains to be explored.
- Lateral connections: ReNet propagates information across the whole image through lateral connections, whereas LeNet uses only local information.These connections are intended to resolve redundant features and small spatial displacements.
- Pooling: ReNet replaces LeNet’s max-pooling with learned lateral connections, though pooling could reduce feature-map dimensionality and computational cost.The authors state that lateral connections can emulate local competition induced by max-pooling.
- Differentiability: ReNet is end-to-end smooth and differentiable, making it more suitable than max-pooling-based LeNet as a decoder in convolutional autoencoders.Max-pooling may be problematic in this setting because the max operator is not invertible.
- Receptive field: Unlike a pooling-free strided-convolution variant, ReNet gives each feature activation access to the whole input image rather than only a subset.The comparison concerns the receptive-field scope of feature activations.
- Parallelization: ReNet is not easily parallelizable because recurrent computation is sequential, whereas LeNet can compute layer activations independently.This limitation applies to model parallelism; data parallelism remains available to both architectures.
- Efficiency: Sequential lateral connections may provide more efficient parameterization with fewer computations, but the authors state that this requires further exploration.The claim is presented as a possibility rather than an established result.
4 Experiments
The experiments evaluate ReNet on MNIST, CIFAR-10, and SVHN using recurrent-unit variants and selected architectures, training procedures, and data augmentations. The reported test errors include 0.45% on one benchmark and 2.38% on another.
- Datasets: ReNet was evaluated on MNIST, CIFAR-10, and SVHN, three widely used object-recognition benchmarks.
- Datasets: MNIST contains 70,000 centered 28 × 28 grayscale digit images, split into 50,000 training, 10,000 validation, and 10,000 test samples.
- Datasets: CIFAR-10 contains 60,000 32 × 32 RGB images across ten categories, split into 40,000 training, 10,000 validation, and 10,000 test samples.
- Datasets: SVHN uses 32 × 32 RGB cropped house-number images, with 543,949 training, 60,439 validation, and 26,032 test samples.
- Model and training: The experiments used GRU units on MNIST and CIFAR-10, LSTM units on SVHN, and selected augmentation procedures per dataset to maximize validation performance.
- Model and training: The study used a limited hyperparameter set as a proof of concept rather than conducting extensive search for optimal validation performance.
- Reported test errors: MNIST test error was 0.45% for ReNet, while another reported model achieved 0.28%.
- Reported test errors: CIFAR-10 test error was 2.38% for ReNet, while another reported model achieved 1.92%.
5 Results and Analysis
Table 2 compares single-model generalization errors for ReNet and previous work across MNIST, CIFAR-10, and cropped-digit SVHN. ReNet performs comparably to deep CNNs but does not outperform state-of-the-art CNNs on any benchmark.
- Table 2 reports generalization errors for ReNet and previous single-model results on MNIST, CIFAR-10, and cropped-digit SVHN.
- ReNet performs comparably to deep convolutional neural networks across all three benchmark datasets.
- ReNet does not outperform state-of-the-art convolutional neural networks on any of the three benchmark datasets.
6 Discussion
The ReNet architecture is independent of the recurrent unit, with preliminary evidence favoring gated units. The evaluation remains quantitative, and its sequential computation is only partly parallelizable.
- GRU and LSTM units outperform usual sigmoidal units in preliminary experiments.The authors interpret this as indirect evidence that ReNet uses long-term dependencies across an input image.
- Test-set accuracies do not reveal which image structures ReNet captures for object recognition.The authors call for further investigation of ReNet’s internal behavior and exploration of RNN–CNN ensembles.
- ReNet is less parallelizable because recurrent computations are sequential, but forward and backward RNNs can run independently.This construction permits some parallel computation despite the sequential nature of RNNs.