Source-linked AI summary
Deep Neural Networks Reveal a Gradient in the Complexity of Neural Representations across the Brain's Ventral Visual Pathway
Umut Güçlü, Marcel A. J. van Gerven
TL;DR
The paper asks how increasingly complex visual features are represented across downstream human ventral-stream regions. It uses deep CNN layers with voxel-wise response models to map cortical representations and decode stimuli from BOLD responses. The results reveal an explicit complexity gradient, feature-specific cortical organization, and improved decoding performance.
Problem
The progression of feature complexity across multiple human ventral-stream regions had not been properly quantified, and downstream feature representations remained debated.
Method
The study uses deep CNN feature representations and voxel-specific linear response models to predict BOLD responses and map feature complexity across cortex.
Results
Downstream ventral-stream areas correspond to increasingly deep CNN layers and increasingly complex stimulus features, while combined decoding identifies stimuli with 100% accuracy from 120 candidates.
Takeaways & Limitations
CNN representations map low-level properties mainly to early visual areas and semantically meaningful object parts and objects mainly to posterior inferior temporal areas.
Abstract
from arXiv · showhide
Converging evidence suggests that the mammalian ventral visual pathway encodes increasingly complex stimulus features in downstream areas. Using deep convolutional neural networks, we can now quantitatively demonstrate that there is indeed an explicit gradient for feature complexity in the ventral pathway of the human brain. Our approach also allows stimulus features of increasing complexity to be mapped across the human brain, providing an automated approach to probing how representations are mapped across the cortical sheet. Finally, it is shown that deep convolutional neural networks allow decoding of representations in the human brain at a previously unattainable degree of accuracy, providing a more sensitive window into the human brain.
1 Introduction
The ventral visual pathway is thought to encode increasingly complex features downstream, but this progression has not been quantitatively established across multiple human regions. Deep CNN representations provide a way to measure and map this complexity gradient across cortex.
- Downstream ventral-stream neurons are associated with larger receptive fields, greater transformation invariance, and more complex stimulus selectivity.
- The progression of feature complexity across multiple human ventral-stream regions had not been properly quantified.
- Exactly which stimulus features are represented in downstream visual areas remained more debated than orientation, location, and spatial frequency in V1.
- Deep CNN layers provide increasingly complex feature representations and state-of-the-art object-recognition performance, motivating their use for modeling BOLD responses.
- The framework predicts voxel responses, maps low-, mid-, and high-level features across the ventral stream, and supports stimulus identification from observed BOLD responses.
2 Framework
The framework combines a deep CNN feature model with a voxel-specific linear response model, using CNN layers as candidate representations of visual stimuli. It is trained and evaluated on natural-image BOLD responses from occipital-cortex voxels.
- The encoding model has a nonlinear feature model that transforms a visual stimulus into layer-specific representations and a linear response model that predicts a voxel response.
- The CNN feature model contains five convolutional and three fully connected layers with nonlinear operations including rectification, normalization, pooling, and softmax transformation.
- The CNN was trained on roughly 1.2 million ImageNet images labeled with 1000 object categories.
- Each voxel received a separate ridge-regression response model using one of eight CNN feature-representation layers as input.
- Response models used 1750 stimulus-response pairs for training and 120 pairs for testing.
- The dataset contained grayscale natural images and peak BOLD responses from 25915 occipital-cortex voxels in one subject.
3 Results
Deep CNN layer assignments revealed a posterior-to-anterior gradient in representational complexity across the ventral visual pathway, with low-layer voxels concentrated early and high-layer voxels downstream. The model also mapped feature selectivity and enabled highly accurate stimulus decoding from multiple voxel responses.
- Voxel encoding: 13% of occipital-cortex voxels had significant prediction accuracy and were assigned to the CNN layer with the lowest cross-validation error.Nonsignificant voxels were discarded before layer assignment analyses.
- Voxel encoding: Prediction accuracy across voxel groups was 0.42, 0.50, 0.39, 0.29, 0.27, 0.24, 0.27 and 0.16 for CNN layers one through eight, respectively.All voxel-group accuracies were significantly above zero; differences were correlated with layer activity and voxel SNR.
- Cortical organization: Voxel-group layer assignments increased from posterior to anterior cortex, while successive groups had stronger partial correlations than non-successive groups.Low-layer representations resembled Gabor wavelets and textures, whereas high-layer representations resembled object parts and objects; Kolmogorov complexity correlated with layer assignment.
- Cortical organization: Mean CNN layer assignments across V1, V2, V4 and LOC were 1.8, 2.3, 3.0 and 4.9, respectively, placing lower layers in earlier and higher layers in downstream visual areas.Prediction accuracies for V1, V2, V4 and LOC were 0.51, 0.46, 0.30 and 0.30, respectively.
- Feature selectivity: Individual feature maps and voxels showed a many-to-many relationship that was mostly confined to single or neighboring visual areas for low- and high-layer features.Layer-one feature maps preferentially predicted early visual voxels, whereas layer-five maps preferentially predicted downstream visual voxels.
- Stimulus decoding: The combined V1+V2+V4+LOC decoder identified the correct stimulus from 120 potential stimuli at 100% accuracy, remaining unchanged when the set increased nearly 16-fold.This exceeded the accuracy of either low-level or high-level decoding model alone.
4 Discussion
Deep neural networks reveal a gradient from simpler to increasingly complex neural representations across the human ventral visual pathway. They also map stimulus features across cortex and support more accurate decoding of complex stimuli than established approaches lacking mid- to high-level features.
- Downstream ventral-stream areas code increasingly complex stimulus features corresponding to increasingly deep CNN layers.Voxel assignments and partial correlations support a hierarchical organization with information flow mainly between neighboring visual areas.
- Low-level stimulus properties are mainly confined to early visual areas, whereas object parts and objects are mostly represented in posterior inferior temporal areas.This feature mapping provides a way to probe how semantic knowledge is represented across the cortical sheet.
- Deep neural networks enable accurate decoding of complex stimuli from observed neural responses.Decoding performance significantly improves on established approaches that do not incorporate mid- to high-level stimulus features.
- The approach demonstrates an explicit human ventral-pathway gradient for object complexity, visualizes deep-network features mapped across cortex, and reaches previously unattainable decoding accuracy.These results position deep neural networks as a sensitive window into human brain representations.