Source-linked AI summary
Brain-Like Object Recognition with High-Performing Shallow Recurrent ANNs
Jonas Kubilius, Martin Schrimpf, Kohitij Kar, Ha Hong, Najib J. Majaj, Rishi Rajalingham, Elias B. Issa, Pouya Bashivan, Jonathan Prescott-Roy, Kailyn Schmidt, Aran Nayebi, Daniel Bear, Daniel L. K. Yamins, James J. DiCarlo
TL;DR
The paper addresses whether increasingly deep, high-performing visual ANNs remain sufficiently brain-like and anatomically interpretable. It develops CORnet-S, a shallow recurrent ANN with four anatomically mapped areas, and evaluates it with Brain-Score and ImageNet. CORnet-S achieves leading brain-predictivity and strong object-recognition performance, while recurrence is identified as a main predictive factor and its IT dynamics partly resemble monkey IT dynamics.
Problem
Deep ANNs perform well on object recognition but are difficult to map onto brain anatomy and often omit biologically important recurrence.
Method
The paper develops CORnet-S, a shallow recurrent ANN with four anatomically mapped visual areas, and evaluates it using Brain-Score and ImageNet.
Results
CORnet-S is the top model on Brain-Score, achieves 73.1% ImageNet top-1 accuracy, and partly captures primate IT neural response trajectories.
Takeaways & Limitations
Neuroanatomical compactness and recurrence can support a model that predicts neural firing rates, behavior, and neural dynamics while maintaining high ImageNet performance.
Takeaways & Limitations
CORnet-S remains incomplete as a ventral-stream model because it lacks biologically plausible learning mechanisms and several anatomical and circuitry-level details.
Abstract
from arXiv · showhide
Deep convolutional artificial neural networks (ANNs) are the leading class of candidate models of the mechanisms of visual processing in the primate ventral stream. While initially inspired by brain anatomy, over the past years, these ANNs have evolved from a simple eight-layer architecture in AlexNet to extremely deep and branching architectures, demonstrating increasingly better object categorization performance, yet bringing into question how brain-like they still are. In particular, typical deep models from the machine learning community are often hard to map onto the brain's anatomy due to their vast number of layers and missing biologically-important connections, such as recurrence. Here we demonstrate that better anatomical alignment to the brain and high performance on machine learning as well as neuroscience measures do not have to be in contradiction. We developed CORnet-S, a shallow ANN with four anatomically mapped areas and recurrent connectivity, guided by Brain-Score, a new large-scale composite of neural and behavioral benchmarks for quantifying the functional fidelity of models of the primate ventral visual stream. Despite being significantly shallower than most models, CORnet-S is the top model on Brain-Score and outperforms similarly compact models on ImageNet. Moreover, our extensive analyses of CORnet-S circuitry variants reveal that recurrence is the main predictive factor of both Brain-Score and ImageNet top-1 performance. Finally, we report that the temporal evolution of the CORnet-S "IT" neural population resembles the actual monkey IT population dynamics. Taken together, these results establish CORnet-S, a compact, recurrent ANN, as the current best model of the primate ventral visual stream.
1 Introduction
The paper asks whether visual models can become more brain-like without sacrificing object-recognition performance. It introduces CORnet-S, a shallow recurrent model that aligns more closely with ventral-stream anatomy and achieves strong ImageNet and Brain-Score performance.
- Deeper ANNs improve ImageNet performance but have not been shown to improve brain-likeness, while their many layers are difficult to map onto ventral-stream areas.
- CORnet-S uses a shallow recurrent anatomical structure to improve correspondence with ventral-stream measurements while maintaining high ImageNet performance.
- 73.1% ImageNet top-1 accuracy accompanies CORnet-S’s state-of-the-art prediction of neural firing rates and image-by-image human behavior on Brain-Score.
- Recurrence primarily drives the reported results, and CORnet-S partly captures time-varying neural response trajectories in primate IT.
2 CORnet-S: Brain-driven model architecture
CORnet-S is designed around predictivity, compactness, and recurrence, transforming deep feedforward ideas into a shallow recurrent model mapped onto four ventral-stream-like areas. Its current definition uses simple canonical circuitry, omits several biological pathways, and is trained on ImageNet.
- CORnet-S is developed using predictivity, compactness, and recurrence as criteria for a brain-oriented ANN.Predictivity targets brain-matching internals and outputs; compactness uses depth as a proxy for biological constraint; recurrence enables responses over time.
- The model transforms very deep feedforward architectures into a shallow recurrent structure inspired by ResNets and weight sharing.
- CORnet-S maps four computational areas to V1, V2, V4, and IT, followed by a linear category decoder.
- Each visual area uses canonical computations including convolution, addition, nonlinearity, response normalization, or receptive-field pooling, with area-specific neuron counts.
- The current architecture lacks across-area bypass and feedback connections and does not explicitly model retinal or LGN processing.
- Training uses ImageNet 2012 with stochastic gradient descent and momentum, using specified image preprocessing, batch size, hardware, and learning-rate scheduling.
3 Brain-Score: Comparing models to brain
Brain-Score compares ANN and biological visual processing through neural, behavioral, and temporal benchmarks. The paper evaluates model-to-brain predictivity using held-out neural responses, behavioral response patterns, and image-specific object-solution timing.
- Brain-Score combines prediction of primate V4 and IT neural responses, human behavioral choices, and object-category resolution in monkey IT.
- Neural predictivity maps model activations to recorded neuron responses using linear prediction and evaluates held-out image responses through cross-validation.
- Neural evaluations include recordings from 88 V4 sites and 168 IT sites for 2,560 images, plus independent datasets testing new monkeys, new images, and behavioral generalization.
- Behavioral predictivity measures similarity to human or monkey response patterns rather than only overall accuracy, so perfect accuracy need not produce perfect behavioral similarity.
- Behavioral evaluation uses 1,472 humans viewing 2,400 briefly presented images, with model probabilities estimated for 240 held-out images and transformed into d′ scores.
- Object solution times are obtained by decoding category from model activations across 70–250 ms and correlating each image’s threshold-crossing time with monkey IT timing.
4 Results
CORnet-S achieved the strongest reported Brain-Score while retaining competitive ImageNet performance at shallow depth. Its results and circuitry analyses highlight recurrence as central to performance and show robust transfer and temporal-response evaluation.
- Brain-Score .471 made CORnet-S outperform other compared alternatives by a large margin.
- 82.9% ImageNet top-1 accuracy did not translate to high Brain-Score: PNASNet ranked 13th overall.Below 70% ImageNet performance, Brain-Score correlated strongly at .90; above 70%, correlation was not significant.
- Four generalization tests evaluated Brain-Score across new subjects, new stimuli, new object categories, and CIFAR-100.
- Depth 15 and 73.1% ImageNet top-1 gave CORnet-S a stronger shallow-model trade-off than AlexNet at 57.7% and IamNN at 69.6%.CORnet-S also achieved the best transfer performance among similarly shallow models.
- At least a few recurrent steps in each block were the most important circuitry factor, while a wide bottleneck and skip connection also mattered.The analysis varied circuitry hyperparameters and measured changes relative to a baseline in ImageNet top-1 and Brain-Score.
- CORnet-S estimated image-specific object solution times and compared model IT trajectories with time-varying monkey IT recordings.The model was not trained to predict monkey object solution times; a linear classifier decoded object category at successive 10 ms windows.
5 Discussion
CORnet-S combines shallow recurrent neuroanatomical structure with strong performance across brain and machine-learning benchmarks. The discussion emphasizes recurrence, neural dynamics, and remaining biological incompleteness.
- 5 Discussion: CORnet-S remains competitive on ImageNet and CIFAR-100 while ranking among the top models on Brain-Score.Its shallow recurrent architecture follows neuroanatomy more closely than standard machine-learning ANNs.
- 5 Discussion: The model remains biologically incomplete because it lacks biologically plausible learning mechanisms and detailed structures such as the retina and lateral geniculate nucleus.Its skip connection was motivated by deep-network optimization rather than cortical circuitry.
- 5 Discussion: Recurrence is associated with improved brain alignment, including partly matching time-varying primate IT response trajectories.CORnet-S was the first model reported to do so on this neural benchmark.
- 5 Discussion: Brain-Score combines neural and behavioral benchmarks and can reveal model differences not captured by ImageNet performance alone.The discussion contrasts strong ImageNet performance with weaker brain measures in some models.
- 5 Discussion: The paper argues that neuroanatomical alignment can support compact models that predict neural firing rates, behavior, and neural dynamics while maintaining high ImageNet performance.This is presented as an opportunity for collaboration between neuroscience and machine learning.
A Numerical Brain-Scores
Table 1 presents Brain-Scores alongside individual performances for state-of-the-art models.
- A Numerical Brain-Scores: Table 1 reports Brain-Scores and individual performances for state-of-the-art models.The supplied passage provides the table title but no numerical entries.
B.1 Brain-Score benchmark
The Brain-Score benchmark aggregates neural and behavioral measures to evaluate models overall, using compactness and controlled testing choices to support interpretation.
- B.1 Brain-Score benchmark: Global Brain-Score is the mean of V4 predictivity, IT predictivity, IT object solution times, and behavioral I2n predictivity.The mean preserves score distances but does not normalize differences in scale or variance.
- B.1 Brain-Score benchmark: The neural benchmarks use responses to 2,560 naturalistic stimuli recorded from 88 V4 neurons and 168 IT neurons.The stimuli are grayscale images spanning eight object categories.
- B.1 Brain-Score benchmark: Two macaque monkeys passively viewed 100 ms images separated by 100 ms gaps, with each image retested about 50 times.Recordings were obtained from V4 and IT arrays.
- B.1 Brain-Score benchmark: Behavioral benchmarks use human responses to images from 24 object categories collected on Amazon Mechanical Turk.Human and non-human primate behavioral patterns were reported as very similar.
- B.1 Brain-Score benchmark: Brain benchmarks test frozen model parameters on independent data, training only the linear decoder introduced for evaluation.The independent data are sampled from distributions different from the training data.
D Predictors of neural scores
Behavioral and neural scores are related but not interchangeable, while feature count shows limited and benchmark-dependent association with neural predictivity.
- D Predictors of neural scores: .65 for V4 and .87 for IT are the correlations between neural scores and behavior.The correlations connect neural measurements to behavior but do not make behavior sufficient to explain the full neural population.
- D Predictors of neural scores: r = .46, p < .05 describes the V4 association with having more than 1000 features, whereas IT shows no significant correlation at any feature count.Neural scores are therefore not consistently correlated with feature number across benchmarks.
- D Predictors of neural scores: Figure 6 indicates that the number of neurons does not appear to predict better brain-likeness.The comparison plots feature counts in models’ highest-scoring layers against V4 and IT neural scores.
E Early and late neural predictions
The analysis separates neural predictions into early and late temporal bins, revealing that strong ImageNet models predict late IT responses well but not early responses, while CORnet-S performs well on both.
- 90-110 ms and 190-210 ms bins distinguish early and late neural prediction patterns more clearly than mean temporal prediction.The early-late division was used to highlight functional differences between models.
- Strong ImageNet models predict IT responses well at 190-210 ms but not at 90-110 ms.Prediction correlations were evaluated separately for early and late spike-rate bins.
- CORnet-S predicts both early and late IT responses well.
F CORnet-S search
The CORnet search examined 106 circuitry configurations across training, finding a robust relationship between behavioral Brain-Score and ImageNet top-1 performance alongside substantial variance.
- 106 CORnet configurations were evaluated across different circuitry designs and training points.Each plotted dot represents one CORnet configuration at one point during training.
- The correlation between Brain-Score behavioral performance and ImageNet top-1 performance is robust but highly variable.