Source-linked AI summary
Neural system identification for large populations separating "what" and "where"
David A. Klindt, Alexander S. Ecker, Thomas Euler, Matthias Bethge
TL;DR
Neural system identification must model nonlinear neuronal responses despite limited data for each cell, while conventional approaches do not exploit similarities across neurons. The paper proposes a CNN with a sparse, factorized readout that separates shared feature learning from neuron-specific location and cell-type estimates. It reports improved performance over state-of-the-art system-identification models on mouse V1 natural-image responses, while identifying location estimation as a performance bottleneck.
Problem
Per-neuron data limits model expressiveness, and conventional system-identification approaches do not exploit shared computations across neurons despite complex nonlinear responses.
Source: §1 · pp. 1–2Method
A CNN with a sparse, factorized readout learns a common feature space while estimating each neuron's receptive-field location and cell type.
Source: §6 · p. 9Results
Takeaways & Limitations
The factorized feature weights can serve as a barcode for functionally classifying cell types.
Source: §6 · p. 9In practice, receptive-field locations are unknown in advance and neurons do not all implement exactly the same nonlinear function.
Source: §3 · p. 3
Abstract
from arXiv · showhide
Neuroscientists classify neurons into different types that perform similar computations at different locations in the visual field. Traditional methods for neural system identification do not capitalize on this separation of 'what' and 'where'. Learning deep convolutional feature spaces that are shared among many neurons provides an exciting path forward, but the architectural design needs to account for data limitations: While new experimental techniques enable recordings from thousands of neurons, experimental time is limited so that one can sample only a small fraction of each neuron's response space. Here, we show that a major bottleneck for fitting convolutional neural networks (CNNs) to neural data is the estimation of the individual receptive field locations, a problem that has been scratched only at the surface thus far. We propose a CNN architecture with a sparse readout layer factorizing the spatial (where) and feature (what) dimensions. Our network scales well to thousands of neurons and short recordings and can be trained end-to-end. We evaluate this architecture on ground-truth data to explore the challenges and limitations of CNN-based system identification. Moreover, we show that our network model outperforms current state-of-the art system identification models of mouse primary visual cortex.
1 Introduction
Neural system identification seeks models of neuronal responses, but existing methods are limited by per-neuron data and do not fully capture nonlinear visual responses. The paper proposes a CNN that shares features across neurons while separating feature learning from estimates of each neuron's location and cell type.
1 Introduction
- Traditional GLMs use a linear filter and point-wise nonlinearity, yet neurons show complex nonlinear responses even in early visual areas.Nonlinearity increases higher in the visual hierarchy.Source: §1 · p. 1
- Neurons in one brain area can compute similar functions at different visual-field locations, motivating shared feature spaces across neurons.The paper relates this what–where separation to functional cell types arranged retinotopically.Source: §1 · pp. 1–2
- Existing nonlinear models still fail to account fully for early visual responses, partly because per-neuron data limits model expressiveness and parameter count.The cited methods include energy models, spike-triggered covariance, LN-LN cascades, convolutional subunit models, and GLMs with handcrafted feature spaces.Source: §1 · pp. 1–2
- The proposed end-to-end CNN separates common feature learning from estimating each neuron's receptive-field location and cell type, and outperforms state-of-the-art models on mouse V1 natural-image responses.The architecture is also evaluated on ground-truth models to examine deep-learning system identification.Source: §1 · p. 2
2 Related work
Earlier neural-response models often fit cells independently or rely on readouts that scale poorly. This work combines convolutional feature sharing with a sparse, factorized readout and evaluates the benefits of sharing on ground-truth models.
2 Related work
- Two-layer convolution-and-pooling models improved on GLMs and spike-triggered covariance, but fit individual cells without exploiting similarities across neurons..Source: §2 · p. 2
- Antolik et al. learned a common feature space without convolution, so features were learned separately by location and fully connected parameter counts grew quadratically with population size.Their model used parameterized difference-of-Gaussian filters followed by two fully connected layers.Source: §2 · p. 2
- McIntosh et al. [20] used a fully connected readout whose parameter count grew quickly and exceeded half the model's parameters even for small populations.The work applied a CNN to retinal data.Source: §2 · p. 2
- Batty et al. estimated neuron locations from spike-triggered averages before learning shared features, a step that can be problematic for nonlinear neurons with weak averages.Their approach used a two-step procedure with recurrent networks for the shared feature space.Source: §2 · p. 2
- The proposed architecture pairs convolutional layers with a sparse readout that factorizes spatial and feature dimensions, estimating receptive-field location and cell type.The readout reduces parameter count while directing computations into the convolutional layers.Source: §2 · pp. 2–3
- Ground-truth evaluations systematically test feature-sharing benefits, especially with many neurons and few observations.The Figure 1 thought experiment illustrates that ten neurons with known, identical receptive-field shapes can provide the equivalent of 10× the data when fit jointly.Source: §2 · p. 3, Figure 1 · p. 3
3 Learning a common feature space
Sharing features lets neurons with similar computations contribute data to a common model. In practice, receptive-field locations are unknown and neurons do not all implement exactly the same nonlinear function.
3 Learning a common feature space
- For ten neurons with identical functions at different known locations, pooling shifted stimuli provides effectively 10× the data of one neuron, while independent fits gain no benefit from their similarity.The shared model reaches the single-neuron performance with one-tenth the data in this thought experiment.Source: §3 · p. 3
- In practice, receptive-field locations are not known in advance and neurons need not implement identical nonlinear functions.The authors expect within-cell-type response diversity to be smaller than diversity across types.Source: §3 · p. 3
4 Feature sharing in a simple linear ground-truth model
The section tests feature sharing in simulated linear populations and describes a CNN whose factorized readout separates receptive-field location from cell-type features. It compares this approach with linear baselines and examines how the readout relates to performance and functional classification.
4 Feature sharing in a simple linear ground-truth model
- The initial experiment uses a population of linear neurons with Poisson-like output noise to investigate the benefits of learning a common feature space.The stimulus is Gaussian white noise, and the GLM baseline reduces to regularized multivariate linear regression.Source: §4 · p. 3
4.1 Convolutional neural network model
- The CNN uses convolutional feature maps and a factorized readout with a spatial mask and feature weights for each neuron.The mask represents receptive-field location, while feature weights represent cell type; both are L1-regularized, and the factorization reduces parameters and aids interpretation.Source: §4.1 · pp. 3–4, Figure 2 · p. 4
4.2 Baseline models
- The baseline models are regularized linear regressions fitted to stimuli cropped around each neuron's receptive field, matching the CNN convolution kernel's parameter count.The study uses both L1 (lasso) and L2 (ridge) regularization and cross-validates their strengths.Source: §4.2 · p. 5
4.3 Performance evaluation
- Performance is measured as fraction of explainable variance explained on held-out test data, evaluated against ground-truth firing rates without observation noise.A perfect model would achieve FEV = 1.Source: §4.3 · p. 5
4.4 Single cell type, homogeneous population
- Ridge regression explained around 60% of explainable variance with 4,000 samples, while a CNN pooling over 10 neurons reached that level with less than a quarter as much data.The CNN consistently outperformed the linear regression models, and its relative improvement grew with the number of pooled neurons before leveling off from 100 to 1,000.Source: §4.4 · p. 5
- With few observations, estimating each neuron's location mask appears to be the bottleneck, and even ground-truth kernels leave mask estimation as an upper performance bound.CNN1000 learned cleaner weights with 256 samples than ridge regression did with 4,096, although ridge achieved higher FEV (65% vs. 55%).Source: §4.4 · pp. 5–6
4.5 Functional classification of cell types
- For two simulated cell types with heterogeneous filters within each type, sparse feature weights make each neuron's selected feature channel indicate its type.The learned kernels represent type averages, although other kernels spanning the same subspace could achieve the same predictive performance.Source: §4.5 · p. 6
5 Learning nonlinear feature spaces
The section evaluates a factorized-readout CNN on synthetic nonlinear populations and mouse V1, testing its predictive performance, recovery of cell properties, and scalability. It outperforms the tested baselines on all three V1 scans and scales to populations of thousands of neurons.
5.1 Ground truth model
- The synthetic nonlinear test used 1,000 VGG-19-derived cells from four feature-map types, with random locations and noise-perturbed responses to cropped ImageNet images.Each type had 250 units, with 32 × 32 possible locations.Source: §5.1 · p. 6
- With relatively few samples, the factorized-readout CNN outperformed all three baselines; the GLM saturated at approximately 20% FEV, while fixed-location masks reduced performance.The fully connected readout [20] reached the factorized readout's performance only asymptotically with more samples; the authors suggest linear spike-triggered-average estimates are unreliable for these nonlinear neurons.Source: §5.1 · p. 7
- The model recovered the four cell types and receptive-field locations: feature weights clustered by type, location masks were recovered, and all cells were correctly classified by their largest feature weight.The reported visualization used 216 samples.Source: §5.1 · p. 7
- The fitted model did not reach 100% performance because its feature space was smaller and its network shallower than the CNN generating the ground truth.Source: §5.1 · p. 7
- As neuron-type diversity increased, the factorized CNN maintained roughly the same performance; the fully connected readout's performance fell quickly and could not be fit beyond 16 feature maps due to GPU memory limits.The CNN with fixed location masks also scaled well, but had lower overall performance.Source: §5.1 · p. 8
- With 216 training samples, the model fit approximately 60,000 recordings from approximately 8,000 neurons on one GPU and achieved 90% FEV.The population comprised 128 neuron types, with 64 units per type.Source: §5.1 · p. 8
5.2 Application to data from primary visual cortex
- For mouse V1, the authors cross-validated CNNs with one to three layers and selected models by validation-set likelihood; test performance was the correlation with average responses across repeats.The scans contained static natural-image responses, and the models used smoothness and group-sparsity regularization of convolution kernels.Source: §5.2 · pp. 8–9
- The factorized-readout CNN outperformed all four baselines on all three mouse V1 scans, including the earlier shared-feature network.Table 1 reports average correlations between model predictions and neural responses on the test set.Source: §5.2 · p. 9, Table 1 · p. 9
6 Discussion
Shared convolutional features can substantially improve neural-response prediction, but performance is bounded by the difficulty of estimating each neuron's visual-field location. The discussion connects this architecture to cell-type classification and recording-design choices, and identifies code and evaluation details.
6 Discussion
- Predictive performance benefits from shared convolutional features until difficulty estimating each neuron's visual-field location imposes an upper bound.A sparse, factorized readout separates location estimation from feature learning while keeping inference of receptive-field location and cell type tractable.Source: §6 · p. 9
- For a very homogeneous population, explaining at least 80% of variance could require either ≈2,000 responses from 10 cells or ≈500 responses from 1,000 cells.The authors suggest performance curves may help decide between longer recordings and sampling more neurons.Source: §6 · p. 9
- Feature weights can serve as a “barcode” for functionally classifying cell types, and the authors are applying the approach to large-scale retina and V1 data.They suggest later visual areas might also exploit approximate equivariance in orientation or direction of motion.Source: §6 · p. 9
Availability of code
- Model-fitting code and figure-reproduction code are available online.The passage gives the project repository URL.Source: Availability of code · p. 9
- Evaluation uses correlation coefficient to match the original study, because FEV estimates are unreliable with few repetitions per image.Source: Availability of code · p. 9