Modeling the dynamics of human brain activity with recurrent neural networks

1 Introduction

• Encoding models combine a nonlinear stimulus-to-feature transformation with a linear feature-to-response transformation to predict brain activity and study sensory representations.Their performance often remains below the noise ceiling, leaving variance that improved encoding models could explain.Source: §1 · p. 1
• Prior work had extensively developed feature models, including visual and semantic representations, while response-model development had progressed relatively slowly.Examples include CNN representations for visual features and constructed or estimated word and phrase representations for semantic features.Source: §1 · pp. 1–2
• Standard response models linearly transform features to responses, although neurovascular coupling and cognitive factors create more complex temporal dependencies.Existing models account for neurovascular dependencies only partly, while remaining dependencies are largely ignored because modeling them can be computationally inefficient or their factors difficult to identify.Source: §1 · p. 2
• The paper uses recurrent neural networks to model arbitrary feature sequences end to end, drawing on internal memory to capture task-relevant temporal dependencies.An in-depth analysis of RNN response models in this encoding setting had not yet been performed.Source: §1 · p. 2
• Recurrent neural network models significantly outperform standard ridge-regression models in predicting BOLD responses to high- and low-level features of natural movies.The authors attribute this to accurately estimating hemodynamic response functions by capturing temporal dependencies in the data.Source: §1 · p. 2

2.1 Data set

• Responses were measured at 1-second repetition time in occipital-cortex voxels and grouped into 16 retinotopically localized areas, though not all areas were identified in every subject.The first six seconds of each block were discarded; the final 45 seconds of training blocks served as validation data.Source: §2.1 · p. 3
• RNN response models use two recurrent nonlinear layers followed by a nonrecurrent linear layer, while ridge-regression models use one nonrecurrent linear layer.The ridge models account for hemodynamic delay either by convolving features with a canonical HRF and derivatives or by concatenating feature sequences lagged by 3–6 seconds.Source: Figure 1 · p. 3

2.2 Problem statement

• The prediction target is the most likely response vector at time t given the stimulus history from time 0 through t.The stimulus has n dimensions and the response has m voxel dimensions.Source: §2.2 · p. 4
• The encoding distribution combines a feature model mapping stimuli into p-dimensional features with a response model mapping those features into m-dimensional responses.The response model produces a vector in R^m.Source: §2.2 · p. 4
• The study pairs two feature models—a low-level scene-description model and a high-level word-embedding model—with recurrent neural network or feedforward ridge-regression response models.The response-model families differ in architecture.Source: §2.2 · p. 4

2.3.1 High-level semantic model

• The high-level semantic model uses pretrained skip-gram word2vec representations to encode words and phrases as continuous distributed vectors.The model was trained on Google News text and contains 300-dimensional representations for three million words and phrases.Source: §2.3.1 · p. 4
• For each second of movie, 20 automatically assigned categories are converted to word2vec vectors and averaged into a 300-dimensional feature vector.The categories were assigned using the Clarifai API.Source: §2.3.1 · p. 4

2.3.2 Low-level visual feature model

• The low-level GIST model represents scenes using spatial-envelope features that capture dominant spatial structure.The paper uses an established GIST implementation and describes prior use of the representation in neural studies.Source: §2.3.2 · pp. 4–5
• For each second, GIST features average energies across 15 frames after filtering 16 image regions with 32 Gabor filters, yielding 512 dimensions.The filters span eight orientations and four scales.Source: §2.3.2 · p. 5

2.4.1 Ridge regression family

• Ridge-regression models predict responses as linear combinations of features and differ in how they account for hemodynamic delay.Source: §2.4.1 · p. 5
• R-C convolves features with the canonical HRF, while R-CTD adds its temporal and dispersion derivatives before linear response prediction.Source: §2.4.1 · p. 5
• R-F uses feature lags at 3, 4, 5, and 6 seconds, concatenating them for linear response prediction.Source: §2.4.1 · p. 5

2.4.2 Recurrent neural network family

• Two-layer RNNs nonlinearly process arbitrary feature sequences and predict responses as linear combinations of second-layer hidden states.Source: §2.4.2 · pp. 5–6
• The LSTM models use 10, 50, or 100 units and maintain cell-state information through input, forget, and output gates; GRUs use 10, 50, or 100 units with update and reset gates.Source: §2.4.2 · p. 6
• RNNs were trained with truncated backpropagation through time and Adam, using dropout and the epoch with the best validation performance as the final model.Source: §2.4.2 · p. 6

• Voxel-specific HRFs are estimated by simulating the RNN response to an impulse, with a pre-zero interval to stabilize the baseline.Source: §2.5 · p. 7
• After baseline subtraction, the impulse response is divided by its maximum and its pre-zero interval is discarded to yield the HRF.Source: §2.5 · p. 7
• Response delay is the HRF's peak time, and undershoot delay is the subsequent time of its minimum.Source: §2.5 · p. 7

3.1 Comparison of the response models

• With W2V, G-100 reached ˜r = 0.16 versus ˜r = 0.12 for the best ridge model, while GIST showed a similar pattern at ˜r = 0.18 versus ˜r = 0.14.For W2V, RNN models with 50 or 100 neurons significantly outperformed all ridge models; for GIST, G-100 was best among RNNs and R-C among ridge models.Source: Figure 2 · p. 8, §3.1.1 · p. 8, §3.1.2 · p. 9

3.1.1 Comparison of the response models in combination with the semantic model

• Among voxels where G-100 and R-C differed by more than r = 0.1, G-100 performed better in 78% (∆˜r = 0.17), while R-C performed better in 22% (∆˜r = 0.14).This threshold-defined subset comprised 30% of the voxels.Source: §3.1.1 · p. 8
• With W2V, noise ceiling controlled performance was highest in TOS (˜r∗ = 0.55) and lowest in pSTS (˜r∗ = 0.14) among the listed areas.Performance was significantly above chance throughout the visual-stream areas.Source: §3.1.1 · pp. 8–9, Figure 3 · p. 9

3.1.2 Comparison of the response models in combination with the low-level feature model

• With GIST, noise ceiling controlled G-100 performance was highest in V1 (˜r∗ = 0.39) and lowest in TOS (˜r∗ = 0.13) among the listed areas.Performance was significantly above chance throughout the areas.Source: §3.1.2 · p. 9, Figure 5 · p. 10

3.2 Comparison of the feature models

• W2V and GIST performance differed significantly in all visual-stream areas except pSTS and V3A, favoring semantic representations downstream and perceptual representations upstream.The largest semantic advantage was in TOS (∆˜r = 0.11), while the largest perceptual advantage was in V1 (∆˜r = 0.10).Source: Figure 6 · p. 11, §3.2 · p. 11

3.3 Estimation of voxel-specific HRFs

• Estimated HRFs had median response and undershoot delays of 6.57 ± 0.02 s and 16.95 ± 0.04 s, respectively, and those delays were correlated (Pearson’s r = 0.45).The overall HRF shape resembled the canonical HRF, but estimated response and undershoot delays varied considerably.Source: Figure 7 · p. 12, §3.3 · p. 12

3.4 Analysis of the internal representations of the G-100 model

• Layer 2 internal-state RDMs correlated more with predicted-response RDMs (0.93) than with feature RDMs (0.21).For layer 1, the corresponding correlations were 0.61 with predicted responses and 0.39 with features.Source: §3.4 · pp. 12–13, Figure 8 · p. 13

4 Discussion

• RNN response models outperform ridge regression, though their nonlinearities can make them harder to interpret when explained variance is not the primary criterion.For comparisons between computational models such as GIST and W2V, the authors say maximizing explained variance is the main criterion of interest.Source: §4 · p. 13

4.1 Testing hypotheses about brain function

• RNN-based encoding models can test representational hypotheses, including whether low-level visual features explain upstream responses and high-level semantic features explain downstream responses.This dissociation complements work mapping different DNN layers to different brain regions.Source: §4.1 · p. 14
• Constraining RNN memory capacity could identify the temporal-dependency scales to which different brain regions are selective.The modeled dependencies may reflect neurovascular coupling or stimulus-induced cognitive processes.Source: §4.1 · p. 14, §4.2 · p. 14

4.2 Capturing temporal dependencies

• Impulse inputs let RNNs learn voxel-specific hemodynamic responses from continuous-design data, while their internal representations reflect long-range feature histories and remain challenging to interpret.Those dependencies are likely induced by stimulus-related and cognitive factors on top of the hemodynamic response.Source: §4.2 · p. 14

4.3 Other uses of RNNs

• RNNs could also serve as feature models or replace standard MVPA models in decoding, where temporal dependencies may affect response amplitudes.The authors expect RNNs to outperform standard MVPA models in the decoding framework.Source: §4.3 · pp. 14–15

4.4 Limitations of RNNs

• RNNs have many free parameters and require substantial training data; very high-dimensional features may therefore need dimensionality reduction, especially with limited data or computation.The authors note dropout variants as one way to combat overfitting.Source: §4.4 · p. 15

4.5 Conclusions

• The authors conclude that RNNs can probe neural correlates of ongoing cognition induced by naturalistic stimuli, with end-to-end training applicable when sensory inputs are coupled to observed neural responses.They contrast this with classical connectionist work focused on RNNs as models of cognitive processing.Source: §4.5 · p. 15