Source-linked AI summary
Task-Driven Convolutional Recurrent Models of the Visual System
Aran Nayebi, Daniel Bear, Jonas Kubilius, Kohitij Kar, Surya Ganguli, David Sussillo, James J. DiCarlo, Daniel L. K. Yamins
TL;DR
The paper asks whether recurrent and feedback mechanisms absent from typical CNNs improve ImageNet recognition and explain primate visual dynamics. It augments CNNs with local recurrent cells and long-range feedback, then searches architectural variants. Carefully designed recurrence improves task performance, and ImageNet-optimized ConvRNNs better match macaque V4 and IT neural dynamics than feedforward models.
Problem
Typical CNNs model time-averaged primate responses but omit local recurrence and long-range feedback needed to study fine-timescale visual dynamics.
Method
The paper augments CNNs with local recurrent cells and long-range feedback, evaluates controls, and searches thousands of recurrent architectures for ImageNet recognition and neural-dynamics modeling.
Results
Task-optimized ConvRNNs improved ImageNet performance over standard recurrent designs and strongly predicted macaque V4 and IT neural dynamics.
Takeaways & Limitations
Appropriate recurrent architectures can support object recognition while providing normative models of temporal encoding dynamics in primate visual cortex.
Takeaways & Limitations
ConvRNN predictions exceeded feedforward predictions mainly when feedforward CNNs had no visual response, while some dynamics may arise from neuronal biophysics rather than recurrent connections.
Abstract
from arXiv · showhide
Feed-forward convolutional neural networks (CNNs) are currently state-of-the-art for object classification tasks such as ImageNet. Further, they are quantitatively accurate models of temporally-averaged responses of neurons in the primate brain's visual system. However, biological visual systems have two ubiquitous architectural features not shared with typical CNNs: local recurrence within cortical areas, and long-range feedback from downstream areas to upstream areas. Here we explored the role of recurrence in improving classification performance. We found that standard forms of recurrence (vanilla RNNs and LSTMs) do not perform well within deep CNNs on the ImageNet task. In contrast, novel cells that incorporated two structural features, bypassing and gating, were able to boost task accuracy substantially. We extended these design principles in an automated search over thousands of model architectures, which identified novel local recurrent cells and long-range feedback connections useful for object recognition. Moreover, these task-optimized ConvRNNs matched the dynamics of neural activity in the primate visual system better than feedforward networks, suggesting a role for the brain's recurrent connections in performing difficult visual behaviors.
1 Introduction
The paper extends task-driven CNN modeling toward the recurrent and feedback dynamics found in biological vision. It tests whether these mechanisms improve ImageNet recognition and better explain primate neural trajectories.
- Biological vision must recognize objects despite large variation in position, pose, contrast, background, and foreground.
- Feedforward CNNs model time-averaged primate visual responses well, but do not capture the visual system’s complex temporal dynamics.
- These dynamics motivate adding local recurrence within cortical areas and long-range feedback between visual regions to task-optimized CNNs.
- The paper integrates recurrent structures within CNNs and searches architectures to identify mechanisms that improve recognition and predict neural dynamics.
- Prior recurrent CNNs had not demonstrated generalization to ImageNet-scale object recognition or matched neural responses as well as ImageNet-optimized CNNs.
2 Methods
The methods augment a CNN with local recurrent cells and resized long-range feedback, producing discrete-time hidden-state trajectories. The study also uses simpler recurrence and parameter-matched feedforward controls to separate recurrent computation from parameter increases.
- ConvRNNs augment standard CNNs with local recurrence and long-range feedback, modeling approximately 10 ms conduction between cortical areas.
- Within each layer, resized feedback and feedforward inputs are convolved, combined in a recurrent cell, and passed through post-memory nonlinearities such as pooling.
- The network state is a discrete-time trajectory whose learned parameters include incoming feedforward and feedback connections and local-cell parameters.
- The Time Decay model provides a simple control with a learned layer-specific time constant intended to model synaptic facilitation and depression.
- Controls use wider or deeper feedforward models and minimally unrolled ConvRNN replicas to approximately match recurrent-model parameter counts.
3 Results
The study found that standard recurrent cells add little to deep CNNs, whereas cells combining bypassing and gating substantially improve ImageNet recognition. Automated architecture search further identified effective local motifs and feedback connections, yielding recurrent models that matched or exceeded deeper feedforward networks and better captured primate neural dynamics.
- Local recurrent cells: Standard vanilla RNNs and LSTMs added little beyond parameter-matched minimal unrolling, while widening or deepening the feedforward model was more effective.These results suggested that standard recurrent cells are poorly suited to deep CNNs.
- Local recurrent cells: Gating controls whether bottom-up input is passed, retained, or discarded, while bypassing lets zero-initialized hidden states pass feedforward input unchanged.The study proposed that combining both properties could improve optimization and recurrent computation in CNNs.
- Local recurrent cells: The Reciprocal Gated Cell substantially improved accuracy over both the feedforward baseline and a minimally unrolled parameter-matched control, using fewer parameters than standard cells.It combines LSTM-like gating with bypassing of the zero-initialized hidden state.
- Architecture search: More than 80% of initially sampled architectures failed to improve above chance or had exploding gradients, but search performance rose over approximately 7000 samples and the best model exceeded the initial top model by more than 15%.The search jointly varied architectural and learning hyperparameters using TPE, with sample models trained for five epochs on 128 px images.
- Architecture search: Top-performing architectures combined bypassing and gating, depth-separable hidden-state updates, conventional input-to-output convolutions, and selected long-range feedback connections.The search identified these recurring structural patterns among the best models.
- Task performance: The median ConvRNN reached 72.9% Validation Top1 accuracy with 15.5 M parameters, nearly matching ResNet-34 at 73.1% with 21.8 M parameters.The recurrent model therefore approached the accuracy of a deeper feedforward model with fewer parameters.
- Neural dynamics: ConvRNN dynamics predicted held-out neural response trajectories as well as or better than feedforward features across almost every 10 ms bin, while Time Decay fit some phases less well.Persistent recurrent activations supported predictions after image offset and improved single-image temporal-response prediction.
4 Discussion
Task-optimized ConvRNNs improved object-recognition performance and predicted macaque neural dynamics, extending the modeling framework beyond feedforward CNNs. The discussion highlights both the promise of learned recurrence and boundaries requiring further testing.
- Architectural implications: Specific long-range feedback patterns increased performance when paired with appropriate local recurrence.The authors suggest this may indicate a role for long-range visual-cortex connections in object recognition.
- Neural dynamics: ConvRNNs optimized for ImageNet strongly predicted neural dynamics in macaque V4 and IT.Their features fit temporal trajectories of neuronal firing rates with linear regression.
- Neural dynamics: ConvRNNs outperformed feedforward CNNs in predicting neural dynamics mainly when feedforward models had no visual response.Across stimuli, early V4 and IT dynamics were therefore dominated by feedforward processing, while single-image recurrent trajectories better captured non-square-wave responses.
- Role of recurrence: The learned recurrent structures improved the same categorization task as feedforward networks without imposing a predefined semantic role for recurrence.The discussion relates this result to challenge images, where recurrent circuits may be engaged despite the absence of clear image-level defining properties.
- Future directions: Future work could optimize recurrent augmentations for more sophisticated CNN baselines and test alternative tasks for matching neural responses.The authors specifically mention NAS-Net and tasks that do not require difficult-to-obtain category labels.
6 Supplemental Methods
The supplemental methods specify the CNN architectures, recurrent-cell equations, training procedure, hyperparameter search, and neural-response fitting protocol used to evaluate ConvRNNs.
- Model architectures: The study compares feedforward CNN variants, including wider and deeper six-layer models and an 18-layer ResNet-style CNN.An auxiliary classifier was placed 70% of the way up the network.
- Recurrent cell: The Reciprocal Gated Cell updates its state using convolutional inputs and elementwise gating of memory and cell outputs.The equations define the input at layer ℓ and distinguish Hadamard products from convolutions.
- Architecture search: The recurrent-cell search varies input sets, connection possibilities, affine operators, and nonlinearities within a formalized update-equation space.Candidate affine operations include addition, 1 × 1 convolution, K × K convolution, and depth-separable convolution, with K from 3 to 7.
- Optimization: ConvRNNs were trained on 128 px ImageNet with stochastic gradient descent, a 0.01 initial learning rate, Nesterov momentum, and gradient clipping.The loss was computed from the final time step, and models were trained synchronously during the search.
- Optimization: The search used Tree-structured Parzen Estimator Bayesian optimization to model configuration performance and optimize expected improvement.Approximately 6000 models were sampled, with top candidates later fully trained on 224 px ImageNet using matched batch sizes for comparison.
- Neural-data evaluation: Neural-data fits used 256 sites from V4, posterior IT, and central or anterior IT, with images and gray-screen periods matched to monkey trials.Models received 10 image time steps followed by 15 mean-gray steps; features were linearly fit to neural responses using L2 loss.