Deep Learning Models of the Retinal Response to Natural Scenes

Lane T. McIntosh; Niru Maheswaranathan; Aran Nayebi; Surya Ganguli; Stephen A. Baccus

Source-linked AI summary

Deep Learning Models of the Retinal Response to Natural Scenes

Lane T. McIntosh, Niru Maheswaranathan, Aran Nayebi, Surya Ganguli, Stephen A. Baccus

arXiv:1702.01825v1Feb 6, 2017q-bio.NCstat.ML

TL;DR

The paper asks how nonlinear retinal circuits encode natural stimuli and whether standard retinal models capture those responses. It trains and analyzes CNNs on retinal responses to natural scenes and white noise, comparing them with LN models and GLMs. CNNs more accurately predict responses, generalize across stimulus classes, and expose internal features consistent with retinal mechanisms.

Problem
Accurate models of neural responses to ethologically relevant natural stimuli remain difficult because multilayered circuits contain nonlinear synaptic and spiking processes.
Method
The study trains CNNs on retinal ganglion-cell responses to natural image sequences and high-resolution white noise, compares them with LN models and GLMs, and examines learned mechanisms.
Results
CNNs markedly outperform LN models and GLMs on white-noise and natural-scene prediction, generalize better across stimulus distributions, and approach retinal response reliability.
Takeaways & Limitations
CNNs can provide accurate, tractable models that reveal computations and mechanisms underlying sensory-circuit responses to natural scenes.
Takeaways & Limitations
GLM comparisons omitted cell-coupling filters, and analyses excluded one minute after stimulus transitions because short CNN filters could not capture tens-of-seconds adaptation.

Abstract

from arXiv · show

A central challenge in neuroscience is to understand neural computations and circuit mechanisms that underlie the encoding of ethologically relevant, natural stimuli. In multilayered neural circuits, nonlinear processes such as synaptic transmission and spiking dynamics present a significant obstacle to the creation of accurate computational models of responses to natural stimuli. Here we demonstrate that deep convolutional neural networks (CNNs) capture retinal responses to natural scenes nearly to within the variability of a cell's response, and are markedly more accurate than linear-nonlinear (LN) models and Generalized Linear Models (GLMs). Moreover, we find two additional surprising properties of CNNs: they are less susceptible to overfitting than their LN counterparts when trained on small amounts of data, and generalize better when tested on stimuli drawn from a different distribution (e.g. between natural scenes and white noise). Examination of trained CNNs reveals several properties. First, a richer set of feature maps is necessary for predicting the responses to natural scenes compared to white noise. Second, temporally precise responses to slowly varying inputs originate from feedforward inhibition, similar to known retinal mechanisms. Third, the injection of latent noise sources in intermediate layers enables our model to capture the sub-Poisson spiking variability observed in retinal ganglion cells. Fourth, augmenting our CNNs with recurrent lateral connections enables them to capture contrast adaptation as an emergent property of accurately describing retinal responses to natural scenes. These methods can be readily generalized to other sensory modalities and stimulus ensembles. Overall, this work demonstrates that CNNs not only accurately capture sensory circuit responses to natural scenes, but also yield information about the circuit's internal structure and function.

1 Introduction

The paper addresses whether models built for retinal responses to artificial stimuli can capture nonlinear responses to natural scenes. It applies CNNs to natural image sequences and finds accurate predictions alongside internal features consistent with retinal mechanisms.

1 Introduction: LN models describe retinal responses to white noise but fail to generalize to natural stimuli.The paper notes that coarse or spatially uniform white noise may not differentially activate nonlinear retinal subunits.
1 Introduction: CNNs cascade spatiotemporal filtering and rectification, matching elementary computations thought to underlie complex sensory-circuit responses.The authors apply CNNs to retinal responses to natural image sequences, extending their use to an early sensory area.
1 Introduction: Deep neural networks markedly outperform LN models and GLMs in predicting retinal responses to white noise and natural scenes.The reported advantages include better generalization across stimulus classes and internal features consistent with sub-Poisson variability, feedforward inhibition, and contrast adaptation.
1 Introduction: CNNs can reveal neural computations and mechanisms within a multilayered circuit under natural stimulation.The paper presents this modeling strategy as potentially useful across neural systems and complex stimulus ensembles.

2 Methods

The study records salamander retinal ganglion-cell responses to natural images and high-resolution white noise, then trains CNNs and compares them with LN and GLM baselines. CNN architectures are optimized probabilistically, regularized, and systematically varied to assess predictive performance.

2 Methods: Retinal ganglion-cell activity was recorded during natural-image sequences and high-resolution spatiotemporal white noise.CNNs predicted all recorded cells simultaneously for each stimulus class, with LN models and GLMs serving as comparison baselines.
2 Methods: CNN parameters were optimized by minimizing Poisson negative log-likelihood using ADAM.The networks used layerwise ℓ2 weight penalties and a final-layer ℓ1 activity penalty.
2 Methods: Increasing CNN depth improved held-out prediction accuracy through three layers, after which performance saturated.The architecture search varied layers, filters, layer types, and convolutional filter sizes.
2 Methods: The networks were trained for 100 epochs with early stopping guided by a validation set.The authors report that filter sizes exceeding 15x15 checkers performed better than smaller filters used in many object-recognition networks.

3 Results

Deep CNNs predict retinal responses to white noise and natural scenes more accurately than LN and GLM models, while generalizing better and revealing retinal-like mechanisms. Their learned structure captures richer natural-scene features, precise timing, sub-Poisson variability, and longer-timescale adaptation.

Performance: CNNs substantially outperformed LN and GLM models when predicting retinal responses to both white noise and natural scenes.CNNs approached retinal reliability for both stimulus classes, with robust gains across explained variance, log-likelihood, and mean squared error.
Performance: A CNN trained on 25 minutes of data achieved better held-out performance than an LN model trained on the full 60-minute recording.LN models were more susceptible to overfitting despite having fewer parameters.
CNN model parameters: Natural-scene models required more active second-layer units and learned a greater diversity of spatiotemporal features than white-noise models.Natural-scene filters also tended toward lower spatial and temporal frequencies.
Generalization across stimulus distributions: CNNs trained on one stimulus class generalized better to an unseen stimulus distribution than LN and GLM models.GLMs performed particularly poorly when generalizing to natural scenes, whereas CNNs provided a more accurate description even when trained on artificial stimuli.
Feedforward inhibition shapes temporal responses in the model: Timed combinations of positive and negative responses across convolutional layers produced precise firing events analogous to retinal feedforward inhibition.The model progressively sharpened responses so its output mimicked the statistics of real firing events.
Noise and adaptation: Injected latent noise enabled CNNs to reproduce the sub-Poisson scaling of retinal spike-count variability.The study also explored recurrent feedback over 10 seconds; recurrent networks learned adaptation over a few seconds while matching CNN prediction performance.

4 Discussion

CNNs provide accurate, interpretable models of retinal responses to natural scenes, outperforming standard retinal models while capturing response properties associated with retinal mechanisms. The approach uses flexible models first and then analyzes their internal activity to understand circuit function.

4 Discussion: CNNs, sometimes augmented with lateral recurrent connections, outperform LN and GLM models on retinal response prediction and generalize across held-out stimuli and stimulus classes.The models learn general retinal-response features rather than fitting only a particular stimulus distribution.
4 Discussion: CNNs capture precise firing timing, adaptive responses to changing stimulus statistics, and biologically realistic sub-Poisson variability that LN models fail to capture.These properties connect model behavior to known retinal response features under natural stimulation.
4 Discussion: Figure 6 visualizes CNN activity across two convolutional layers, spatial filter activations, output predictions, and temporal autocorrelation against recorded firing rates.The figure provides a tractable view of how signals are transformed from stimulus features into predicted responses.
4 Discussion: Recurrent layers added after the CNN capture slowly adapting firing rates over several seconds following step changes in contrast.LSTMs, GRUs, and MUTs showed similar prediction performance to the CNN, while the recurrent network learned multi-second adaptation.
4 Discussion: Natural stimuli are difficult to model because their distributions are hard to describe mathematically and their non-stationary statistics can influence response properties.The paper positions CNNs as a flexible way to model such responses before probing the learned computation.

Supplementary methods

The study recorded salamander retinal ganglion-cell responses to high-resolution white noise and jittered natural images, selecting reliable cells and evaluating models on held-out repeated stimuli. The methods emphasize stimulus resolution, adaptation control, and regularized baseline comparisons.

Supplementary methods: Responses from tiger salamander retinal ganglion cells in three animals were recorded using a 60-channel multielectrode array.The dataset contains responses to both natural-image sequences and high-resolution spatiotemporal white noise.
Supplementary methods: 37 cells exceeded the reliability criterion of a correlation above 0.3, with 70.3% classified as fast OFF-type cells.Cells were retained based on consistency across repeated stimulus blocks.
Supplementary methods: Data collected after transitions between natural scenes and white noise were excluded for one minute to avoid contrast-adaptation effects over tens of seconds.Spikes were binned in 10 ms intervals and smoothed with a 10 ms Gaussian filter.
Supplementary methods: Training used a random 90%/10% train-validation split, while testing used averaged repeated trials from one minute of novel stimuli.The test procedure evaluates predictions against responses to held-out stimuli.
Supplementary methods: Stimuli comprised 35%-contrast binary checkers and jittered natural images sampled from a natural-image database.The natural-image sequence incorporated eye-movement-like jitter statistics.
Supplementary methods: The white-noise stimulus used 50 x 50 checkers spanning 55 µm x 55 µm, allowing differential activation of nonlinear subunits within the receptive-field center.This resolution was higher than that used in prior retinal-modeling studies and created a stronger challenge for nonlinear-response models.
Supplementary methods: Baseline GLMs used stimulus and spike-history filters, whereas LN models used spatiotemporal filters and a parameterized soft rectifying nonlinearity.LN parameters were regularized to reduce overfitting from their large parameter count.