PaperlayerView exact revision on arXiv

Source-linked AI summary

Scaling laws for language encoding models in fMRI

Richard Antonello, Aditya Vaidya, Alexander G. Huth

arXiv:2305.11863v4cs.CLcs.AI

TL;DR

The study asks how language-model scale relates to fMRI encoding performance. It fits voxelwise models from model hidden states and finds logarithmic gains that taper above about 30B parameters.

  • Problem

    The study asks how language-model parameter scale relates to fMRI encoding performance.

    Source: §3.1 · p. 5

  • Method

    It fits voxelwise fMRI encoding models from transformer hidden states and evaluates performance across model sizes.

    Source: §2.3 · p. 3, §3.1 · p. 5

  • Results

    Roughly 4.4% increase in average-subject encoding performance per parameter order of magnitude (r = 0.91) tapers to a plateau above ~30B parameters.

    Source: §3.1 · p. 5

  • Takeaways & Limitations

    Even with as little as an hour of data, encoding models built from larger models tend to outperform their smaller counterparts.

    Source: §4 · p. 10

    With 33,000 data points and FIR delays, models past 30B have more hidden states than training points, lowering encoding performance.

    Source: §3.1 · p. 6

Abstract

from arXiv · show

Representations from transformer-based unidirectional language models are known to be effective at predicting brain responses to natural language. However, most studies comparing language models to brains have used GPT-2 or similarly sized language models. Here we tested whether larger open-source models such as those from the OPT and LLaMA families are better at predicting brain responses recorded using fMRI. Mirroring scaling results from other contexts, we found that brain prediction performance scales logarithmically with model size from 125M to 30B parameter models, with ~15% increased encoding performance as measured by correlation with a held-out test set across 3 subjects. Similar logarithmic behavior was observed when scaling the size of the fMRI training set. We also characterized scaling for acoustic encoding models that use HuBERT, WavLM, and Whisper, and we found comparable improvements with model size. A noise ceiling analysis of these large, high-performance encoding models showed that performance is nearing the theoretical maximum for brain areas such as the precuneus and higher auditory cortex. These results suggest that increasing scale in both models and data will yield incredibly effective models of language processing in the brain, enabling better scientific understanding as well as applications such as decoding.

2 Methods

The methods compare pretrained semantic and speech-model features with fMRI responses, fit voxelwise encoding models, and assess their performance and noise ceiling.

2.1 Language models and speech audio models

  • Semantic models included OPT and LLaMA variants, while acoustic models included HuBERT, WavLM, and Whisper variants.Source: §2.1 · p. 2

2.2 MRI data

  • Three subjects listened to about 20 hours of podcast stories, providing roughly 33,000 training datapoints per voxel; scans were trimmed and responses standardized.Source: §2.2 · pp. 2–3

2.3 Encoding model construction

  • Text-model states use dynamically sized contexts and Lanczos/FIR alignment, while bidirectional audio models use 16-second windows at 100-ms strides and Whisper encoder states.Source: §2.3 · pp. 3–4

2.4 Stacked regression

  • Five-fold held-out predictions support residual-covariance stacking across Whisper layers and LLaMA layer 18; validation retains stacked output only where significantly better.Source: §2.4 · p. 4

2.5 Noise ceiling computation

  • Noise ceilings use averaged responses from a test story repeated 10 times; normalized correlations are reserved for the noise-ceiling analysis, while other correlations remain uncorrected.Source: §2.5 · pp. 4–5

2.6 Compute specifications

  • Ridge regression used about 4,000 node-hours, and feature extraction used about 200 GPU-node-hours.Source: §2.6 · p. 5

3 Results

The results show that semantic and audio encoding performance improves with model and fMRI training-data scale, while some regions approach the estimated noise ceiling. The section also examines long-context artifacts and combines semantic and speech representations.

3.1 Scaling laws for semantic encoding models

  • OPT encoding performance rose roughly 4.4% per parameter order of magnitude, then plateaued above ~30B; LLaMA layers peaked earlier than OPT layers.The authors hypothesize that increased hidden-state size combined with limited training data contributes to the plateau.Source: §3.1 · pp. 5–6
  • Each order-of-magnitude increase in OPT-125M training stories increased average-subject encoding performance by 122%.The authors argue this supports collecting more data from fewer subjects rather than less data from many subjects.Source: §3.1 · p. 6

3.2 Scaling laws for speech audio encoding models

  • Whisper encoding performance increased roughly 32.2% per model-size order of magnitude; increasing Whisper Large training data by an order of magnitude increased performance by 144%.The model-size trend was logarithmic (r = 0.991), as was the training-data relationship (r = 0.988).Source: §3.2 · p. 7
  • Whisper and WavLM performed best in upper-middle or upper layers; unlike WavLM, Whisper performance increased with layer depth.The authors suggest this Whisper pattern may reflect using only the network’s encoder module.Source: §3.2 · p. 7

3.3 Large-scale encoding models

  • OPT-30B’s 33rd layer predicted held-out responses across cortex, with voxelwise correlations as high as r = 0.82.Figure 2 shows predictions against average responses over 10 trials for selected voxels, using 20 hours of training data from one subject.Source: Figure 2 · p. 6, §3.3 · p. 7

3.4 Noise ceiling analysis

  • Angular gyrus and some prefrontal areas retained room for improvement, while precuneus and higher auditory cortex were nearly optimal.Figure 3 maps room for improvement relative to estimated CCmax and also shows CCabs and CCnorm histograms.Source: §3.4 · p. 7, Figure 3 · p. 8

3.5 Long context artifacts

  • Long-context effects persisted for roughly 100 seconds into stories, so the first 100 seconds were excluded from test scoring; artifacts could inflate performance by up to 20%.Effects were mostly localized to low-level speech-processing areas and were most prominent for early language-model layers and speech models.Source: §3.5 · p. 8

3.6 Unifying semantic and speech encoding models with stacked regression

  • Stacked regression combining semantic and Whisper representations improved predictions mainly in primary and early secondary auditory cortex, with benefits also localized to mouth motor cortex.Auditory voxels best predicted by the semantic model were also those most improved by the augmentation.Source: §3.6 · p. 8

4 Discussion & conclusions

Encoding performance reflects a balance between model efficacy and regression conditioning: OPT and LLaMA performance peaks near 30B parameters, while audio models have not reached that plateau. The discussion also describes voxelwise scaling patterns and practical uses of stronger encoding models.

4 Discussion & conclusions

  • Stacking Whisper representations onto the 18th-layer LLaMA-33B semantic model improves selected voxels, with benefits highly localized to auditory and mouth motor cortex.Voxels use the stacked model when it performs better on a validation set.Source: Figure 4 · p. 9
  • Encoding performance peaks near 30B parameters for OPT and LLaMA, as gains from more effective language models compete with poorer regression conditioning from larger hidden states.Audio models have not yet seemed to reach this plateau.Source: §4 · p. 9
  • Higher encoding performance may support more trustworthy predictions, improved conclusions from experiments, language decoding, and fine-grained control of voxel activation through stimulus generation.The discussion presents these as potential uses of better encoding models.Source: §4 · p. 9
  • Deep datasets—many samples from a few subjects—may be more useful for modelling brain activity, and larger language models tend to outperform smaller ones even with about an hour of data.The authors also state that large datasets will be necessary for useful encoding models.Source: §4 · pp. 9–10

A Voxelwise Scaling Laws

  • Voxelwise scaling fits the correlation improvement as a linear function of log2 of the scaled attribute; its slope measures improvement when that attribute doubles.The accompanying flatmaps show parametric and dataset-size scaling across three subjects.Source: A Voxelwise Scaling Laws · p. 14

A.1 Parametric Scaling

  • Parametric scaling examines voxelwise changes as the number of parameters in the feature-extraction model increases.The appendix presents these patterns as flatmaps.Source: §A.1 · p. 14

A.1.1 OPT Model Family

The OPT-family flatmaps characterize voxelwise parametric scaling and indicate where larger semantic models appear most beneficial.

A.1.1 OPT Model Family

  • The flatmaps show voxelwise parametric scaling laws computed with the OPT language-model family.They map the constant of proportionality for encoding performance as model size scales.Source: Figure A.1 · p. 14
  • The mapped coefficient describes the proportionality of encoding performance to model-size scaling.The figure presents this coefficient spatially across cortex.Source: Figure A.1 · p. 14
  • Model-size increases in semantic models seem most beneficial for predicting amodal, postauditory cognitive areas such as prefrontal cortex.Source: Figure A.1 · p. 14

A.1.2 Whisper Model Family

The Whisper-family flatmaps characterize voxelwise parametric scaling, with comparatively smaller model-size improvements in auditory cortex.

A.1.2 Whisper Model Family

  • The flatmaps show voxelwise parametric scaling laws computed with the Whisper audio-model family.They map the constant of proportionality for encoding performance as model size scales.Source: Figure A.2 · p. 15
  • The mapped coefficient describes the proportionality of encoding performance to model-size scaling.The figure presents this coefficient spatially across cortex.Source: Figure A.2 · p. 15
  • Model-size improvements are relatively smaller in auditory cortex, suggesting the simplest models already capture the most useful encoded audio features.Source: Figure A.2 · p. 15

A.2 Dataset Size Scaling

The appendix describes voxelwise dataset-size scaling in terms of the number of stories used to train the encoding model's linear weights.

A.2 Dataset Size Scaling

  • Dataset-size scaling maps voxelwise scaling laws using the number of stories as the dataset-size variable.Source: §A.2 · p. 16
  • The story count refers to stories used to train the linear weights of the encoding model.Source: §A.2 · p. 16
  • The described flatmaps present voxelwise patterns of scaling with dataset size.Source: §A.2 · p. 16

A.2.1 OPT-30B

For OPT-30B, dataset-size scaling benefits the regions whose encoding performance is already best predicted, without significant spatial preference.

A.2.1 OPT-30B

  • Dataset-size scaling benefits the best-predicted regions without significant spatial preference, as shown by flatmaps of encoding-performance scaling proportionality.The figure uses OPT-30B.Source: Figure A.3 · p. 16

A.2.2 Whisper-637M

Whisper-637M dataset-size scaling benefits most well-predicted regions, though some precuneus and auditory-cortex portions benefit less than with OPT. The supplemental figures describe audio-model performance, spatial improvements, and a localized long-context artifact.

A.2.2 Whisper-637M

  • Whisper-637M dataset-size scaling benefits most well-predicted regions, with somewhat smaller benefits in portions of precuneus and auditory cortex than for OPT.The flatmaps show the constant of proportionality for encoding-performance scaling with dataset size.Source: Figure A.4 · p. 17

B Scaling laws for speech audio encoding models

  • Audio-model performance is averaged across auditory-cortex voxels for Whisper, WavLM, and HuBERT, while a separate figure averages HuBERT performance across cortical voxels.The captions identify the plotted scope but do not state which models perform best.Source: Figure B.1 · p. 18, Figure B.2 · p. 18

C Scaling Improvements

  • OPT-30B improves voxelwise encoding performance over OPT-125M in most areas, especially prefrontal and parietal cortex.The comparison covers three subjects and reports percent improvements in CCabs from each model’s best layer.Source: Figure C.1 · p. 19

D Long context artifact effects

  • The long-context artifact is highly localized to primary auditory cortex and can bias encoding-performance measurement if uncorrected.The figure illustrates the effect for an early OPT-30B layer.Source: Figure D.1 · p. 20

E Joint data-parameter scaling results

OPT-30B consistently outperforms OPT-125M even with data from a single story, while larger models show greater marginal benefit from additional data.

E Joint data-parameter scaling results

  • OPT-30B consistently outperforms OPT-125M in the low-data regime of a single story.The comparison uses raw encoding performance from each model’s best layer across three subjects.Source: Figure E.1 · p. 21
  • As model size increases, the marginal benefit of additional data increases, suggesting a conditioning effect from insufficient data in large-scale encoding models.The voxelwise slope histogram is for S03 and includes voxels with CCmax > 0.5.Source: Figure E.2 · p. 21

F Cross-subject results

The supplemental results provide subject-specific OPT-30B voxelwise correlations and show which regions remain poorly explained, alongside stacked-regression comparisons with a LLaMA baseline. They also present estimated optimal voxelwise performance for all three subjects.

F Cross-subject results

G Maximum Correlation Coefficient

H Stacked Regression Center-of-Mass Attributions

Appendix H presents subject-specific flatmaps of stacked-regression attribution centers of mass; Appendix I provides extended model details and notes one perplexity value omitted for computational reasons.

H Stacked Regression Center-of-Mass Attributions

I Extended Model Details

Related papers

Loading 2305.11863v4…