PaperlayerView exact revision on arXiv

Source-linked AI summary

Convolutional Neural Networks as a Model of the Visual System: Past, Present, and Future

Grace W. Lindsay

arXiv:2001.07092v2q-bio.NCcs.CVcs.NE

TL;DR

The review asks what makes CNNs useful models of biological vision and how they can provide insight beyond task performance. It synthesizes their biological origins, validation against neural and behavioral data, and uses of controlled model experimentation. CNNs predict neural activity beyond previous methods and succeed as working recognition systems, while mismatches and limited task and species coverage define important boundaries.

  • Problem

    The review addresses how CNNs should be evaluated as models of biological vision and what they can reveal beyond standard object recognition.

  • Method

    The review traces CNN origins, examines neural and behavioral validation, and considers how varying datasets, architectures, and training procedures can probe biological vision.

  • Results

    CNN unit activity predicts real-neuron activity beyond previous methods, and CNNs provide successful working recognition systems while exposing behavioral mismatches under image degradation.

  • Takeaways & Limitations

    Experimenting with CNNs can provide insight into why features of biological visual processing exist and can guide future laboratory experiments and model development.

  • Takeaways & Limitations

    Evidence is concentrated on primate vision and standard object or scene classification, while CNNs generally degrade more than humans under noise, occlusion, or blur.

Abstract

from arXiv · show

Convolutional neural networks (CNNs) were inspired by early findings in the study of biological vision. They have since become successful tools in computer vision and state-of-the-art models of both neural activity and behavior on visual tasks. This review highlights what, in the context of CNNs, it means to be a good model in computational neuroscience and the various ways models can provide insight. Specifically, it covers the origins of CNNs and the methods by which we validate them as models of biological vision. It then goes on to elaborate on what we can learn about biological vision by understanding and experimenting on CNNs and discusses emerging opportunities for the use of CNNS in vision research beyond basic object recognition.

Introduction

CNNs developed from biologically inspired models of visual processing and now serve as mechanistic models that connect neural computations with computer-vision performance. This review examines their origins, validation, experimental use, and broader applications in understanding biological vision.

  • Introduction: Computational models let neuroscientists test mechanistic intuitions, explore controlled hypotheses, and use mathematical analysis to understand visual systems.CNNs have served all three purposes as models of the visual system.
  • Introduction: The review treats CNNs as mechanistic models, requiring correspondence between CNN subparts and components of the visual system in addition to matching outputs.This distinguishes the review’s focus from models that only reproduce system-level input-output behavior.
  • Origins of the model: Hubel and Wiesel’s simple and complex cells inspired hierarchical architectures in which spatially selective responses are combined into more spatially invariant representations.Fukushima’s Neocognitron implemented these ideas with S-cells and C-cells, and repeated layers produced a hierarchy resembling the ventral pathway.
  • Origins of the model: Hierarchical models such as HMAX enabled direct comparisons between model behavior and human rapid visual categorization using the same images.HMAX used a max operation over simple-cell activity and was robust to image variations.
  • CNN architecture: Modern CNNs apply filters across images to form feature maps, use rectification and max-pooling to create complex-cell-like responses, and then classify categories with later fully connected layers.These operations preserve the core analogy between convolutional networks and hierarchical visual processing while supporting object-recognition tasks.
  • CNNs in computer vision: 2012 AlexNet’s success on ImageNet showed that biologically inspired visual features could support strong vision when combined with appropriate learning algorithms and data.The network used eight layers and classified over a million images into one thousand object categories, far exceeding prior state-of-the-art performance.

2. Validating CNNs as a model of the visual system

CNNs are validated as visual-system models by testing whether their engineered architecture yields biological correspondences in neural representations, unit properties, and behavior. These comparisons show strong matches in several respects while also revealing important mismatches, especially under image degradation, texture reliance, and human-level behavioral tests.

  • Neural-level validation: CNNs mirror visual-system architecture through successive convolution, nonlinear, and pooling stages that approximate processing along the ventral stream.The architecture includes input normalization and color channels, followed by stacked processing stages associated with areas such as V1, V2, V4, and IT.
  • Neural-level validation: Artificial-unit activity predicts real-neuron activity from shared image inputs, with accuracy beyond previous methods and strong correspondence in ventral-stream areas.Studies report especially notable matches for V4 and IT, while early-to-middle task-trained layers can also predict V1 activity beyond traditional V1 models.
  • Validation methods: Representational similarity analysis compares population-level response structure by correlating image-pair dissimilarity matrices across brains, models, and behavior.RSA incorporates full population responses, whereas regression can selectively weight model features for single-neuron or voxel prediction.
  • Neural-level validation: CNNs match some neural response properties, including response sparseness and size tuning, but differ in object selectivity and orientation tuning.Additional comparisons examine tuning to shape and categories, as well as sensitivity to pose, location, and size.
  • Behavioral-level validation: CNN behavior aligns with humans on some similarity, typicality, Gestalt, animacy, and classification tests, but often diverges when images are degraded or when shape matters more than texture.Degradation commonly harms CNN classification more than human performance, and CNNs can sometimes exceed humans on tasks, which remains a mismatch for neuroscience models.
  • Other forms of validation: Feature visualization and optimized-neuron stimuli provide additional validation because CNN layers develop neuroscience-aligned patterns and can generate stimuli that effectively drive real neurons.Early filters resemble Gabors, later units respond to partial and fuller object features, and optimized stimuli can increase neuronal firing beyond normal rates despite being unnatural.

3. What we learn from varying the model

Varying datasets, architectures, and training procedures lets researchers test how CNN design choices affect their fit to biological visual data. These comparisons extend models beyond standard object recognition toward recurrent, unsupervised, scene-processing, and action-oriented systems.

  • Alternative datasets: Changing datasets and training procedures can make CNN representations better or worse fits to biological visual data.ImageNet favors object-recognition pathways, whereas scene-trained networks have modeled representations associated with scene processing.
  • Alternative datasets: Scene-trained networks predict occipital place area responses and capture scene-size representations measured with MEG.One study also related learned scene features to navigational affordances.
  • Alternative training procedures: Reinforcement-learning CNNs transform visual state information into actions, while autoencoders reconstruct images through compact representations.LSTM units augment reinforcement-learning architectures with memory.
  • Alternative architectures: Biologically constrained early connections produce on- and off-center responses before orientation tuning, unlike CNNs that often learn Gabor filters immediately.This suggests anatomical constraints may contribute to the progression of selectivity seen in primate vision.
  • Alternative architectures: Local recurrent connections improve performance on challenging tasks and better match neural data, especially for challenging images and later response times.Feedback recurrence is used in models of cued detection and is associated with goal-directed selective attention.
  • Alternative training procedures: Unsupervised methods capture input statistics and may identify low-dimensional features, but currently match neural representations less well than supervised methods.A predictive-coding model nevertheless predicted object movement and replicated motion illusions, while a generative model performed worse on human image categorization.

4. How to understand CNNs

CNNs can be understood by probing their units, representations, connectivity, gradients, and mathematical structure. These analyses reveal both useful correspondences to biological vision and limits of interpreting complex networks through simple tuning descriptions.

  • Empirical methods: Researchers can vary CNN structure and training, then probe trained networks with lesions, recordings, anatomical analyses, and stimulation-like interventions.These tools allow controlled tests of how network components contribute to behavior and representations.
  • Empirical methods: Ablating individual units can change classification accuracy, but ablation impact is not strongly related to the unit’s selectivity.This separates a unit’s tuning properties from its functional contribution.
  • Mathematical methods: Manifold geometry and mathematical analyses provide compact descriptions of representation untangling and CNN computations.Information theory, wavelet scattering, and deep linear approximations are among the approaches used.
  • Empirical methods: Gradient-based visualization changes input pixels to increase a selected unit’s activity, revealing preferred features of units, feature maps, layers, or classes.The same gradients can assess a unit’s role in classification, not just its response preference.
  • Empirical methods: A unit’s contribution to classifying an image is not tightly correlated with how strongly it responds to images of that class.This provides a second demonstration that tuning-based analyses may not fully reveal function.
  • Are they understandable?: CNN interpretability faces a complexity–interpretability tradeoff, motivating searches for compact descriptions based on essential weights, architectures, and datasets.Lottery-ticket findings suggest high-performing networks may be described with far fewer weights, while natural-image datasets remain a major non-compact component.

5. Beyond the Basics

CNNs can model visual phenomena beyond basic object recognition, including memorability, semantic processing, attention, and perceptual learning. Future work aims to connect these abstract models with broader brain systems and more biological implementation details.

  • Exploring cognitive tasks: Later-layer CNN response magnitude correlates with experimentally measured image memorability.Adding a semantic-feature attractor network enabled the model to account for perirhinal cortical activity during a semantic task.
  • Exploring cognitive tasks: Applying attention-like neuromodulatory effects increases CNN performance on challenging visual tasks, with larger gains at later layers.Task-performing models have also informed theories of how attention may work.
  • Exploring cognitive tasks: CNNs recapitulate several behavioral and neural effects of fine-grain perceptual learning.
  • Exploring cognitive tasks: CNN modeling has focused mainly on the ventral visual stream, although visual information supports many other brain functions and goals.Future models should integrate CNNs into this broader neural context.
  • Biological detail: Biologically inspired additions such as recurrence, foveation, saccading, spiking units, and circuit-based architectures seek either improved task performance or more detailed brain implementation.A V1 circuit model has been embedded in a CNN to replicate visual-attention effects, and fly connectome structure has defined another model architecture.

5. Limitations and future directions

CNNs remain useful models of vision, but their biological limitations motivate more plausible architectures, learning rules, species comparisons, and behaviorally richer tasks.

  • Architectural limitations: Weight sharing simplifies CNN training but does not directly match how visual features are distributed across biological retinotopic maps.The same filter is applied at every location, whereas biological uniformity may arise through genetic programming or exposure-based learning.
  • Architectural limitations: Most CNNs violate Dale’s law because individual neurons can receive both inhibitory and excitatory feedforward weights.Negative weights could be interpreted through local inhibitory neurons, but this remains an incomplete biological correspondence.
  • Learning limitations: Backpropagation is often viewed as biologically implausible, although it may be treated as a parameter-fitting technique rather than a learning model.Researchers are investigating brain mechanisms that could perform something like backpropagation, enabling comparisons with biologically plausible training methods.
  • Species scope: CNN comparisons are strongest for primates; results for rodents are substantially weaker, leaving rodent vision an important modeling challenge.Improving rodent models could clarify differences between primate and rodent vision.
  • Task scope: Object and scene classification cover only a small portion of natural vision, motivating embodied tasks such as navigation, manipulation, and visual reasoning.These tasks may better capture visual processing and its relation to other brain areas.

6. Conclusions

CNNs have shifted vision research toward population-level accounts while helping realize the goal of constructing working recognition systems. Their insights are provisional and must be refined through iteration between models and laboratory experiments.

  • Conclusions: CNN research has moved emphasis from interpretable single-neuron responses toward population-level representations and transformations supporting visual tasks.This shift reflects the use of models that perform visual computations rather than only describe isolated responses.
  • Conclusions: Working recognition systems provide an effective measure of progress toward understanding object recognition, making CNNs successful visual-system models.The review connects this success to earlier goals for biologically inspired artificial recognition systems.
  • Conclusions: CNNs alone cannot establish insights about biological vision; progress depends on iterative exchange between computational experiments and laboratory experiments.Insights from CNNs should shape future experiments, which should inform subsequent models.
Loading 2001.07092v2…