PaperlayerView exact revision on arXiv

Source-linked AI summary

BrainBERT: Self-supervised representation learning for intracranial recordings

Christopher Wang, Vighnesh Subramaniam, Adam Uri Yaari, Gabriel Kreiman, Boris Katz, Ignacio Cases, Andrei Barbu

arXiv:2302.14367v1cs.LGeess.SPq-bio.NC

TL;DR

Neural decoding can be limited by scarce annotations and by the tradeoff between decoder power and explainability. BrainBERT learns contextual representations by reconstructing masked neural spectrograms, then supports linear decoding. The paper reports improved decoding accuracy and data efficiency, generalization to new subjects and electrode locations, and an intrinsic-dimensionality analysis; released weights were trained on passive movie-viewing activity.

  • Problem

    Neural recordings pose a tradeoff between the explainability of linear decoders and the power of nonlinear methods, while annotated training data can be scarce.

    Source: §1 · p. 1

  • Method

    BrainBERT is pretrained to reconstruct masked neural spectrograms, producing reusable representations for linear decoding.

    Source: §1 · p. 2, §2 · p. 2

  • Results

    BrainBERT representations improve decoding accuracy and data efficiency, generalize to new subjects and electrode locations, and support intrinsic-dimensionality analysis of brain computations.

    Source: §1 · p. 2, §4 · pp. 7–8, §6 · p. 9

  • Takeaways & Limitations

    The representations retain the explainability of linear decoders while enabling analysis of intrinsic dimensionality across brain regions.

    Source: §1 · p. 2, §6 · p. 9

    The released pretrained weights use activity recorded during passive movie viewing; substantially different downstream settings may benefit from pretraining BrainBERT from scratch.

    Source: §6 · p. 9

Abstract

from arXiv · show

We create a reusable Transformer, BrainBERT, for intracranial recordings bringing modern representation learning approaches to neuroscience. Much like in NLP and speech recognition, this Transformer enables classifying complex concepts, i.e., decoding neural data, with higher accuracy and with much less data by being pretrained in an unsupervised manner on a large corpus of unannotated neural recordings. Our approach generalizes to new subjects with electrodes in new positions and to unrelated tasks showing that the representations robustly disentangle the neural signal. Just like in NLP where one can study language by investigating what a language model learns, this approach opens the door to investigating the brain by what a model of the brain learns. As a first step along this path, we demonstrate a new analysis of the intrinsic dimensionality of the computations in different areas of the brain. To construct these representations, we combine a technique for producing super-resolution spectrograms of neural data with an approach designed for generating contextual representations of audio by masking. In the future, far more concepts will be decodable from neural recordings by using representation learning, potentially unlocking the brain like language models unlocked language.

1 INTRODUCTION

BrainBERT addresses the tradeoff between decoder power and explainability by learning task-agnostic neural representations for linear decoding. It is pretrained across subjects and supports sample-efficient analysis of new subjects and electrode locations.

1 INTRODUCTION

  • Self-supervised representations balance decoder power and explainability by learning task-agnostic neural features before applying a linear decoder.Linear decoding is interpretable, but interesting signals may not be linearly decodable; nonlinear decoders risk performing the task themselves.Source: §1 · p. 1
  • BrainBERT learns a nonlinear transformation of neural data that enables linear decoders to achieve higher accuracy with fewer examples than raw features.The model is pretrained across subjects and offers off-the-shelf analysis for new subjects and electrode locations, including when data is scarce.Source: §1 · p. 1
  • The model uses 43.7 hours of unannotated recordings across subjects to build reusable contextual neural embeddings.The recordings total 4,551 electrode-hours.Source: §1 · p. 2
  • BrainBERT masks parts of neural spectrograms and learns to reconstruct them from contextual embeddings, adapting superresolution representations and masking to neural signals.A content-aware loss gives more weight to non-zero spectrogram elements, which are often sparse.Source: §1 · p. 2
  • The paper contributes a reusable subject- and electrode-agnostic model, improved linear decoding and generalization, and an intrinsic-dimensionality analysis of brain computations.The analysis uses BrainBERT embeddings to examine computations across brain areas.Source: §1 · p. 2

2 METHOD

BrainBERT uses a Transformer to reconstruct randomly masked time-frequency neural representations during pretraining. Its design adapts the representations, masking, and loss to neural signals, and the pretrained model can support linear classification with or without fine-tuning.

2 METHOD

  • During pretraining, a Transformer encoder receives a masked time-frequency representation and learns to reconstruct its missing portions.A spectrogram prediction head maps the top-layer hidden outputs to a reconstruction of the original signal.Source: §2 · pp. 2–3
  • BrainBERT accepts STFT or superlet representations; superlets vary temporal resolution with frequency, while STFT resolution is fixed across frequencies.Superlets composite Morlet wavelet transforms, and per-frequency z-scoring places frequency bands on equal footing for generic representations.Source: §2 · p. 3
  • The superlet masking scheme increases time-mask width as frequency falls and masks more frequency channels at higher frequencies.For STFT, masking instead uses randomly chosen time and frequency intervals.Source: §2 · p. 4
  • A content-aware loss supplements reconstruction to discourage predicting zero for most masked portions and emphasize spectrogram values far from zero.The motivation is that neural signals contain spikes and short oscillation bursts, while approximately 68% of the z-scored spectrogram is 0 or < 1.Source: §2 · p. 4
  • For classification, BrainBERT features are averaged over a centered 244 ms window and passed to a linear classifier, with weights either frozen or fine-tuned.Fine-tuning uses more compute resources but often improves performance.Source: §2 · p. 4
  • Pretraining pools 5-second segments from all subjects and electrodes, while downstream training uses supervised data with a classification head.The recordings come from 26 sessions of 10 subjects with 1,688 electrodes in total.Source: §2 · pp. 3–4

3 EXPERIMENTS

The experiments evaluate four decoding tasks using held-out sessions and compare BrainBERT with raw-signal and spectrogram baselines. The setup also tests frozen-weight performance and controls electrode selection using a linear baseline.

3 EXPERIMENTS

  • The evaluation uses seven held-out sessions, with all ten subjects represented in pretraining and no movie overlap between pretraining and held-out sessions.Annotated examples are drawn from held-out recordings; each uses either 0.25 or 5 seconds of context, depending on classifier architecture.Source: §3 · pp. 4–5
  • Four tasks span audio volume, speech versus non-speech, word pitch, and sentence onsets, with BrainBERT pretrained without task knowledge.The tasks range from low-level perception to higher-level classification.Source: §3 · p. 5
  • Baselines include linear classifiers on raw neural signals and STFT or superlet spectrograms, plus a five-layer feed-forward network.The raw-signal linear baselines use 250 ms or 5 seconds of input.Source: §3 · pp. 5–6
  • With BrainBERT weights frozen, a linear decoder reaches the deep network's performance, while superlet-based BrainBERT generalizes better than STFT-based BrainBERT.The setup tests performance without fine-tuning the pretrained model.Source: Table 2 · p. 6
  • For each task, the comparison fixes the top 10 electrodes selected by the 5-second time-domain linear decoder, and evaluates models per task and electrode.This electrode set is held constant across baselines, ablations, and BrainBERT comparisons.Source: §3 · p. 6

4 RESULTS

BrainBERT improves neural decoding across tasks, including with less labeled data and on new subjects with new electrode locations. Its embeddings also support task-agnostic analysis of intrinsic dimensionality across brain regions.

Improving decoding accuracy

  • Across four tasks, a linear decoder using BrainBERT achieved AUC 0.83, versus 0.63 for the baseline deep network and 0.59 for a linear decoder on unprocessed data.The comparison used the ten electrodes selected by linear decoding on the original data, putting BrainBERT at a disadvantage.Source: §4 · p. 6

Decoding accuracy without fine-tuning

  • Without fine-tuning, BrainBERT averaged AUC 0.63, on par with the baseline deep network; STFT performed best with fine-tuning, while superlets appeared more robust without it.Without fine-tuning, only the linear decoder's parameters were updated.Source: §4 · pp. 6–7

Improving decoding accuracy

  • Across 947 held-out electrodes, BrainBERT embeddings improved sentence-onset decoding and more clearly highlighted language-related superior temporal and frontal regions than raw recordings.Figure 3 shows electrode-level classification performance, with chance AUC 0.5.Source: Figure 3 · p. 7

Generalizing to new subjects

  • For new subjects with novel electrode locations, including the subject in pretraining made a negligible performance difference, and both variants outperformed the linear-decoding baseline.This held across four decoding tasks using the held-out subject's ten best electrodes.Source: Figure 4 · p. 7, §4 · p. 7

Ablations

  • With STFT and fine-tuning, pretrained BrainBERT gained an average 0.23 AUC over its randomized-weight ablation, indicating that the weights matter beyond the structural prior.The comparisons in tables 1 and 2 test BrainBERT with randomly initialized weights against pretrained BrainBERT.Source: §4 · p. 7
  • With superlets and fine-tuning, the content-aware loss and adaptive mask each gained 0.01 AUC; their cumulative impact was significant but second-order relative to BrainBERT itself.The content-aware loss improved three of four tasks and the adaptive mask improved two of four, each by about 5%.Source: §4 · p. 7

Improved data efficiency

  • 150 examples matched the best baseline model's performance at 1,000 examples, and BrainBERT could further improve when more examples were available.This learning-curve comparison used one electrode in the left superior temporal gyrus.Source: Figure 5 · p. 8, §4 · p. 8

Intrinsic dimension

  • Electrode-level intrinsic dimension, estimated as the PCs needed to account for 95% of variance, varied smoothly; high-dimension electrodes clustered mainly in frontal and temporal lobes.Among the top 10th-percentile electrodes, mean intrinsic dimension was highest in the supramarginal gyrus, lateral orbitofrontal cortex, and amygdala; the analysis was task-agnostic.Source: §4 · p. 8

5 RELATED WORK

The paper adapts self-supervised representation learning to intracranial neural recordings, a modality distinguished by its temporal resolution and spatial precision. It also uses BrainBERT embeddings to investigate intrinsic dimensionality as a way to study neural computation.

Self-supervised representations

  • Self-supervised masked-spectrogram modeling learns representations from audio and other domains, and the paper adapts this approach to neural recordings.Related work includes language, audio, and image applications.Source: §5 · pp. 8–9

Intrinsic dimensionality

  • BrainBERT embeddings place electrodes with similar computations on similar data at similar intrinsic dimensionality, suggesting a data-driven metric for identifying functional regions and their relationships.Figure 6 reports that the highest-ID electrodes are mainly in frontal and temporal lobes.Source: Figure 6 · p. 9

Neural decoding

  • BrainBERT applies representation learning to intracranial signals, which directly measure electrophysiological activity from small neuronal populations with high temporal resolution and spatial precision.The paper contrasts these signals with fMRI and EEG modalities.Source: §5 · p. 9

6 CONCLUSION

The paper presents self-supervised neural representations to improve decoding accuracy and data efficiency while retaining linear decoders’ explainability. It also identifies task-agnostic embeddings as a basis for investigating cognitive processes, while noting that the released pretrained weights are tied to passive movie viewing.

Conclusion

  • The self-supervised representations improve neural-decoding accuracy and data efficiency while retaining linear decoders’ explainability.The conclusion connects decoding to neuroscience results that hinge on whether a task is linearly decodable.Source: §6 · p. 9
  • Task-agnostic embeddings can be analyzed to investigate cognitive processes, including possible changes in representations across brain states.The conclusion proposes studying sleep and other brain-state changes in a data-driven manner.Source: §6 · p. 9
  • The authors plan to refine BrainBERT by training larger variants on continuous 24/7 recordings from numerous subjects.Source: §6 · p. 9
  • Released pretrained weights were trained on passive movie viewing, and the authors recommend pretraining from scratch when a downstream use case departs significantly from that setting.Source: §6 · p. 9

ETHICS STATEMENT

This material covers ethics and the methods and implementation details for time-frequency transforms, masking, baselines, preprocessing, and pretraining. It describes the study’s consent and approval alongside practical choices for constructing and training BrainBERT.

ETHICS STATEMENT

  • Experiments were IRB-approved, subjects consented, and electrode locations were determined exclusively by clinical considerations.Source: ETHICS STATEMENT · p. 10

A SUPERLETS

  • Superlets combine Morlet transforms to improve time and frequency resolution together, capturing self-similar neural oscillations across frequencies.Morlet wavelets trade temporal and frequency resolution; their temporal resolution is lower at low frequencies.Source: A SUPERLETS · p. 13

B TIME-FREQUENCY REPRESENTATION PARAMETERS

  • The superlet representation uses c1 = 1 and orders o = 3−30, with frequencies from 0.1 to 200 Hz, then decimates by 50.Five columns are removed from each end to account for edge effects.Source: B TIME-FREQUENCY REPRESENTATION… · p. 13

C MASKING TIME AND FREQUENCY

  • Adaptive masking adjusts temporal-mask width inversely with frequency and masks more frequency channels at higher frequencies.The parameters are selected to roughly match STFT mask widths.Source: C MASKING TIME AND FREQUENCY · p. 14

D MASKING PROCEDURE

  • Mask intervals may be left unaugmented, replaced with a random signal slice, or zero-filled; these choices aim to align train and test distributions and prevent identity learning.The procedure uses pmask = 0.05, pID = 0.1, and preplace = 0.1.Source: D MASKING PROCEDURE · p. 14

E BASELINES

  • Time-domain baselines include linear networks on 5-second or 0.25-second inputs and a five-layer deep network on 5-second inputs.Time-frequency linear baselines use STFT or superlet representations averaged over an approximately 244 ms interval.Source: E BASELINES · p. 15

F PREPROCESSING

  • Pretraining data were high-pass filtered, line noise removed, and electrodes Laplacian re-referenced; only electrodes with neighbors were included, and visibly corrupted recordings were removed.The resulting pretraining set used 1,249 electrodes, about 74% of the total, and 4,551 electrode-hours.Source: F PREPROCESSING · p. 15

G PRETRAINING PARAMETERS

  • The encoder has six layers, hidden dimension 768, and 12 attention heads; pretraining uses LAMB at learning rate 1e−4 for 500,000 steps.The best-validation weights are retained; the prediction head maps to 40 frequency channels.Source: G PRETRAINING PARAMETERS · pp. 15–16

H FINE-TUNING TRAINING PARAMETERS

Fine-tuning uses a linear sigmoid classifier with binary cross-entropy, and model selection is based on validation accuracy.

H FINE-TUNING TRAINING PARAMETERS

  • The linear sigmoid classifier is trained with binary cross-entropy for 1,000 updates, with test accuracy reported from the weights achieving the best validation performance.Fine-tuning uses AdamW at learning rates 1e−3 for the classifier and 1e−4 for BrainBERT; frozen BrainBERT uses 1e−3.Source: H FINE-TUNING TRAINING PARAMETERS · p. 16

I INTRINSIC DIMENSION

The section estimates intrinsic dimension from BrainBERT embeddings using PCA and identifies the brain regions with the highest mean intrinsic dimension.

I INTRINSIC DIMENSION

  • Intrinsic dimension is the number of PCA dimensions whose explained-variance ratio exceeds the threshold β.Source: I INTRINSIC DIMENSION · p. 16
  • For each held-out session, 5-second intervals are embedded and averaged across time before intrinsic dimension is estimated per electrode.The procedure uses PCA of N = 200 with threshold β = 0.95.Source: I INTRINSIC DIMENSION · p. 16
  • The supramarginal gyrus, lateral orbitofrontal cortex, and amygdala have the highest mean intrinsic dimension.The analysis uses PCA with N = 200 and a 0.95 explained-variance threshold.Source: I INTRINSIC DIMENSION · pp. 16–17

I.1 TASKS

The tasks use transcript-aligned brain activity to define positive examples for speech timing and vocal properties, with non-speech intervals serving as negatives for two tasks.

Sentence onset

  • Sentence-onset positives are transcript-aligned brain-activity intervals around sentence onsets, each embedded in 5 seconds of context.Source: §I.1 · p. 17

Speech vs. non-speech

  • Speech positives use 5-second contexts around words, while negatives are 1-second intervals without overlapping speech audio, embedded in 5 seconds of context.Source: §I.1 · p. 17

Volume

  • For volume classification, word-level RMS over 500 milliseconds from onset defines high- and low-volume labels at one standard deviation above and below the mean.Source: §I.1 · p. 17

Pitch

  • For pitch classification, word pitch is extracted with librosa piptrack over a Mel-spectrogram, and labels mark values one standard deviation above or below the mean.The spectrogram uses a 48,000 Hz sampling rate, a 2048-sample FFT window, a 512-sample hop, and 128 mel filters.Source: §I.1 · p. 17

J DATASET STATISTICS

The supplementary material reports task and subject data statistics, electrode localization procedures, pretraining reconstruction losses, and additional electrode and feature analyses.

J DATASET STATISTICS

  • The dataset-statistics listing names sentence onset, speech versus non-speech, volume, and pitch tasks and their class labels.Source: J DATASET STATISTICS · p. 18
  • Held-out examples are counted by task and subject, with sentence-onset and speech-versus-non-speech examples balanced between classes.Non-speech examples are 1-second intervals that do not overlap any movie word-audio.Source: Table 3 · p. 18
  • The subject-statistics table reports an average of 4.3 hours of recording data per subject.Source: Table 4 · p. 18

K ELECTRODE VISUALIZATION

  • Electrodes were assigned to 94 Desikan-Killiany regions after MRI and fluoroscopy registration, with atlas alignment manually verified for each subject.For visualization, gray-matter and boundary electrodes were projected to the pial surface and mapped to an average brain.Source: K ELECTRODE VISUALIZATION · p. 19

L PRETRAINING PERFORMANCE

  • BrainBERT achieved content-aware reconstruction loss of 0.81 with superlet inputs and 0.42 with STFT inputs during pretraining.The corresponding L1 components were 0.41 and 0.20, respectively.Source: Figure 9 · p. 19

M SUPPLEMENTARY FIGURES

  • The STFT-feature intrinsic-dimension distribution is tightly grouped around a few values, unlike the BrainBERT representation distribution.Source: Figure 10 · p. 20
  • The locations of test electrodes are shown after selecting each task's top 10 electrodes using a linear classifier trained over all electrodes.Source: Figure 11 · p. 20

N SUPPLEMENTARY RESULTS

Supplementary analyses describe the held-out performance comparisons, layerwise decoding pattern, and subject-age and held-one-out evaluations.

N SUPPLEMENTARY RESULTS

  • Performance comparisons in Table 5 use fixed task-specific electrode subsets selected from the top 10 linear-classifier electrodes, with the top 20 also reported for comparison.Source: Table 5 · p. 21
  • Across tasks, frozen-feature decoding performance rises with Transformer depth and peaks at the second-to-last layer.The reported work uses last-layer features.Source: Figure 12 · p. 21
  • The age analysis reports no dependence of transfer ability on age (Pearson’s r = −0.27, p-value 0.451).The plotted y-axis is the across-task average AUC-ROC delta between models pretrained with and without a subject.Source: Figure 13 · p. 21
  • The held-one-out table reports close performance matches between BrainBERT trained with and without each subject, with some leave-one-subject-out versions performing better.The table also compares fine-tuning results with the linear classifier performance for that subject.Source: Table 6 · p. 22

Related papers

Loading 2302.14367v1…