Source-linked AI summary
Bridging the Gaps Between Residual Learning, Recurrent Neural Networks and Visual Cortex
Qianli Liao, Tomaso Poggio
TL;DR
The paper addresses how ultra-deep ResNets relate to recurrent computation and the much shallower primate visual cortex. It establishes an equivalence between shared-weight ResNets and shallow RNNs, generalizes the idea to multi-stage recurrent models, and reports effectiveness on CIFAR-10 and ImageNet.
Problem
The paper investigates whether the depth and effectiveness of ultra-deep residual networks can be related to recurrent computations in the substantially shallower primate visual cortex.
Method
It formulates ResNets and RNNs as dynamical systems, develops shared-weight and moderately deep multi-stage recurrent architectures, and uses time-specific batch normalization for recurrent transitions.
Results
Weight-shared RNNs retain most of the corresponding deep ResNet’s performance with orders of magnitude fewer parameters, while the generalized architectures are effective on CIFAR-10 and ImageNet.
Takeaways & Limitations
The results support viewing some ultra-deep feedforward networks as efficient models of recurrent computations and motivate recurrent models of multi-stage visual-cortex processing.
Takeaways & Limitations
The visual-system model separates the retina as a pre-net because it receives no cortical feedback, although a three-layer convolutional pre-net is also tested.
Abstract
from arXiv · showhide
We discuss relations between Residual Networks (ResNet), Recurrent Neural Networks (RNNs) and the primate visual cortex. We begin with the observation that a special type of shallow RNN is exactly equivalent to a very deep ResNet with weight sharing among the layers. A direct implementation of such a RNN, although having orders of magnitude fewer parameters, leads to a performance similar to the corresponding ResNet. We propose 1) a generalization of both RNN and ResNet architectures and 2) the conjecture that a class of moderately deep RNNs is a biologically-plausible model of the ventral stream in visual cortex. We demonstrate the effectiveness of the architectures by testing them on the CIFAR-10 and ImageNet dataset.
1 Introduction
The paper connects ultra-deep residual networks to recurrent computation, asking whether recurrence can reconcile their depth with the shallower primate visual cortex. It proposes shared-weight RNNs and biologically plausible multi-stage recurrent models, evaluating them on CIFAR-10 and ImageNet.
- Motivation: Ultra-deep residual networks achieve strong vision-benchmark performance but are far deeper than the approximately half-dozen ventral-stream cortical areas.The paper contrasts ResNet depths of up to 1202 layers with roughly six ventral-stream areas.
- Motivation: Recurrent, lateral, and feedback connections within cortical areas could account for this depth discrepancy when their computations are unrolled over time.Unrolling recurrent cortical computations yields an equivalent ultra-deep feedforward network.
- Contributions: A deep residual network is formally equivalent to a specific shallow RNN, while weight sharing reduces parameters by orders of magnitude and retains most corresponding-ResNet performance.This is the paper’s central evidence for interpreting deep residual computation recurrently.
- Contributions: The authors generalize the recurrent architecture to moderately deep models intended to represent multi-stage recurrent processing in visual cortex and test them on CIFAR-10 and ImageNet.The proposed models are presented as more biologically plausible than a single shallow recurrent stage.
- Contributions: Time-specific batch normalization is proposed for hidden-to-hidden RNN transitions, enabling training of ReLU RNNs, including multi-state fully recurrent models, without careful initialization.The paper identifies this as a minor contribution and reports additional language-modeling experiments in supplementary materials.
2 Equivalence of ResNet and a specific RNN
The paper formulates residual and recurrent networks as discrete-time dynamical systems. Under weight sharing, unrolling a shallow recurrent system produces a deep ResNet whose layers share weights and retain identity shortcuts.
- Intuition: The recurrent update adds a nonlinear transformation of the previous state to the current input, matching the residual-update structure.The operator K denotes the nonlinear transformation used in the recurrent formulation.
- Equivalence: Unrolling the feedback system over discrete time yields a deep residual network with shared weights, with unrolling depth equal to the number of recurrent iterations.The identity shortcut mapping appears directly in the unrolled network.
- Dynamical-systems formulation: A neural network can be represented as a discrete-time dynamical system whose state is the activity of its neurons.The formulation assumes a single layer with n neurons, state h_t ∈ R^n, and a continuous bounded transition function f parameterized by weights.
- Terminology: Here, RNN is used broadly for neural networks with recurrent activations rather than specifically for the vanilla or plain RNN baseline.This terminology covers the recurrent architectures studied in the paper.
- Dynamical-systems formulation: Weight-shared residual networks correspond to homogeneous, time-invariant systems with input supplied only at t = 0.The associated transition is f(z) = (K + I) ◦ z.
- Dynamical-systems formulation: The paper distinguishes these systems from normal residual networks, which correspond to homogeneous, time-variant systems.The time-variant case allows weights to differ across recurrent steps.
3 A Generalized RNN for Multi-stage Fully Recurrent Processing
The paper generalizes recurrent ResNet-like processing into moderately deep, multi-state fully recurrent models designed to reflect multi-stage visual-cortex processing. These systems use recurrent state graphs, preprocessing and postprocessing networks, and time-specific normalization, with experiments on CIFAR-10 and ImageNet.
- Multi-state Graph: The generalized model represents multi-stage processing as a directed cyclic graph whose vertices are processing states and whose edges are transition functions.For the ventral stream example, states include LGN, V1, V2, V4, and IT, with connections such as V1-V2, V1-V4, and V2-IT.
- Pre-net and Post-net: The recurrent system is trained end-to-end with a pre-net that supplies inputs and a post-net that provides supervisory signals.The pre-net, recurrent system, and post-net are trained jointly with backpropagation; typical implementations use a 3x3 convolutional pre-net and a BN-ReLU-global-average-pooling classifier post-net.
- Transition Matrix: Transition matrices connect states across time, allowing time-varying transitions and sequentially connected locally recurrent subsystems.When multiple transitions feed a state, their outputs are summed in ResNet models or averaged in fully recurrent neural networks; averaging improves FRNN performance by about 1%.
- Architecture: The models use biologically motivated state resolutions and recurrent transition modules, with subsampling or upsampling between nearby states.The illustrated states progress from 32x32 through 16x16 and 8x8 to 4x4, while BRC and BRD pipelines use convolutional or deconvolutional transitions and identity shortcuts.
- Experiments: The experiments compare 3-state and 4-state time-invariant models at readout time t=5, while larger 4-state models are limited by computational cost.The figure specifies the state dimensions for both model families and notes that weights are shared across time.
- Unrolling Depth vs. Readout Time: The paper defines readout time biologically as the simulated time after visual-stimulus onset, with each transition function taking constant time 1.At t=0, only the first state contains pre-net data; transitions begin once their input state is populated, and wall-clock biological time is estimated as 20t to 50t ms.
- Sequential vs. Static Inputs/Outputs: As an RNN, the model supports sequential data processing in addition to the static-input and output setting used for the main vision experiments.The paper points to supplementary character-level language-modeling experiments as an example of sequential processing.
- Batch Normalizations for RNNs: Time-specific normalization computes batch-normalization statistics independently at each time step because sharing statistics across time generally hurts performance.The paper reports that this procedure restores good performance and enables training recurrent networks with ReLUs; learnable BN parameters differ between CIFAR-10 and ImageNet experiments.
4 Related Work
The paper situates its deep recurrent architecture among stacked, recursive, convolutional recurrent, stochastic-depth, feedback-loop, and highway-network approaches. Its distinctions include recurrent feedback and self-transitions, identity shortcuts, and deep convolutional transition functions.
- Deep Recurrent Neural Networks: The proposed deep recurrent model resembles stacked RNNs but adds feedback between hidden layers, self-transitions, identity shortcuts, and deep convolutional transition functions.The paper treats depth as potentially occurring in input-to-hidden, hidden-to-hidden, and hidden-to-output connections, and describes its model as deep in all three senses.
- Recursive Neural Networks and Convolutional Recurrent Neural Networks: Unfolding an RNN ties weights across feedforward layers, connecting this work to recursive neural networks and earlier convolutional recurrent neural networks.Recursive neural networks apply the same operations repeatedly on a structure, while the convolutional version was studied subsequently.
- Related Connections: The recurrent interpretation offers an explanation for why stochastic depth can tolerate randomly replacing some ResNet layers with identity mappings.If layers implement similar refinements repeatedly, the paper argues that dropping a few should not be catastrophic.
- Highway Networks and Cortical Feedback: Unlike ResNet's hardwired identity shortcuts, Highway Networks use gated shortcut mappings and are inspired by Long Short-Term Memory.The paper also notes related work introducing loops into neural networks to imitate cortical feedback.
5 Experiments
Experiments evaluate shared-weight recurrent formulations of ResNet across CIFAR-10 and ImageNet, varying recurrence, readout time, and state count. The results examine performance, parameter sharing, and computational scaling.
- ResNet with Shared Weights Across Time: Shared weights across time retain most of ResNet’s performance while using orders of magnitude fewer parameters.The paper evaluates this formulation on CIFAR-10 and ImageNet.
- Fully Recurrent Neural Networks with Shared and Non-shared Weights: Fully recurrent networks can use shared or independently parameterized transitions across time.The paper reports results for both variants of 2-state fully recurrent neural networks.
- The Effect of Readout Time: Readout time is varied because recurrent systems may gain representational power as more processing time is allowed.Training and testing experiments use different unrolling depths, with matched readout times in one setting.
- Larger Models With More States: 3-state recurrent models generally outperform 2-state models, while 4-state models can become computationally heavy without exceeding 3-state performance.Adjacent-state transitions scale linearly with the number of states and perform well for 3-state networks.
6 Discussion
The discussion links weight sharing and increasing readout time to recurrent processing, while extending the proposed interpretation toward biological visual computation. It also outlines broader conjectures about deep networks and cortex.
- When is Weight Sharing Good/Bad?: Weight sharing improves performance in the first ResNet stage but can slightly reduce performance in later stages.The authors conjecture that sharing helps when the transition function resembles recurrent computations in the corresponding visual area.
- Psychophysics Support: Intermediate network layers matching intermediate human prediction times is consistent with the paper’s prediction about processing over time.Higher layers perform better on recognition, while intermediate layers agree best with rapid human predictions.
- Future Directions: The paper conjectures that deep feedforward networks derive much of their effectiveness from approximating recurrent computations with larger effective readout times.This conjecture is presented as a perspective on why deeper networks can outperform shallower ones.
- Future Directions: The proposed cortical interpretation predicts temporal dynamics within each cortical area, including improving performance with increasing time for briefly flashed images.The proposal also predicts disruption of recurrent computations by masking briefly flashed inputs.
- Readout Time and TSBN: Figure 5 examines generalization across readout times and the temporal behavior of time-specific batch-normalization statistics.A shared-weight model trained at t=13 is tested from t=7 to 23, while TSBN statistics appear to implement a simple decay.
- ImageNet Comparison: Figure 6 compares training and validation errors for shared and non-shared 4-state ResNets on ImageNet under single-crop evaluation.The models are trained for 12 epochs with logarithmically decaying learning rates.