PaperlayerView exact revision on arXiv

Source-linked AI summary

iBrain: A Unified Foundation Model Reading the Brain from Surface to Spikes

Ying Chen, Tiou Wang, Zhifeng Yue

arXiv:2609.06960v1cs.AI

TL;DR

Existing neural foundation models generally treat invasive recording paradigms separately, leaving joint learning across heterogeneous iEEG and spiking signals underexplored. iBrain addresses this gap with signal-specific encoders, a shared spatiotemporal Transformer, and self-supervised pretraining on more than 7,000 hours of recordings. It achieves the strongest performance on seven of eight downstream benchmarks and supports transfer across recording settings.

  • Problem

    Joint representation learning across heterogeneous invasive signals remains underexplored despite their complementary neural measurements and distinct observation scales.

  • Method

    iBrain jointly pretrains signal-specific encoders and a shared spatiotemporal backbone using masked reconstruction and channel-view alignment on over 7,000 hours of iEEG and spike recordings.

  • Results

    iBrain achieves the strongest performance on seven of eight downstream benchmarks spanning motor decoding and language-related tasks.

  • Takeaways & Limitations

    Joint pretraining across heterogeneous invasive recordings yields representations effective across diverse recording settings and downstream tasks.

  • Takeaways & Limitations

    Performance gains from increasing pretraining duration become smaller beyond 2,000 hours, so recording duration alone may not suffice for neural representation learning.

Abstract

from arXiv · show

Invasive neural recordings provide high-fidelity measurements of brain activity, with signals such as intracranial EEG (iEEG) and intracortical spiking activity capturing neural dynamics at different spatial and temporal scales. Yet existing neural foundation models have largely been developed independently for different invasive recording paradigms, leaving joint pretraining across heterogeneous invasive signals underexplored. In this work, we introduce iBrain, a unified foundation model that jointly learns from iEEG and spiking activity. iBrain employs signal-specific encoders to accommodate their distinct signal characteristics and a shared spatiotemporal Transformer backbone to model dependencies across recording channels and time. We pretrain iBrain on over 7,000 hours of heterogeneous neural recordings using masked signal reconstruction and channel-view alignment, promoting contextual modeling of neural dynamics and robustness across different channels. iBrain consistently outperforms single-signal pretraining baselines and achieves state-of-the-art performance on multiple benchmarks. Further experiments demonstrate that iBrain exhibits transferability and data efficiency across diverse recording settings. These results highlight the potential of joint pretraining on heterogeneous invasive neural recordings to support scalable neural modeling and transferable representations across recording settings and downstream tasks.

Introduction

Existing invasive neural models typically specialize in one recording paradigm, limiting transferability, while iBrain jointly learns from iEEG and spiking activity in a unified framework. It combines signal-specific encoders, shared representation learning, and large-scale heterogeneous pretraining for diverse downstream tasks.

  • Prior pretrained neural models have shown benefits for decoding, data-efficient adaptation, and cross-context generalization, motivating broader heterogeneous-signal modeling.
  • Invasive recordings capture complementary neural dynamics across iEEG field activity and localized intracortical spiking activity.
  • iBrain combines signal-specific encoders with heterogeneous-signal reconstruction and channel-view alignment to learn robust representations.
  • iBrain jointly learns iEEG and spiking activity within a shared representation space for diverse downstream tasks.
  • The pretraining corpus contains over 7,000 hours of heterogeneous invasive neural recordings and supports evaluation across motor-decoding and language-related benchmarks.

Related Work

Prior work has adapted foundation-model pretraining to invasive recordings and explored unified modeling across neural modalities. However, jointly modeling invasive signals with distinct forms and observation scales remains underexplored.

  • BrainBERT uses masked self-supervised pretraining on unlabeled iEEG to learn reusable Transformer representations for data-efficient decoding.
  • Unified neural modeling has been explored across fMRI, EEG, and MEG, but invasive recordings remain heterogeneous in acquisition, spatial layout, and physiological content.
  • Joint modeling of heterogeneous invasive signals with distinct signal forms and observation scales remains underexplored.

Method

iBrain combines signal-specific encoders and decoders with a shared spatiotemporal attention backbone to jointly model iEEG and spike recordings. Pretraining uses masked reconstruction and channel-view alignment to learn contextual and channel-robust representations.

  • Signal-specific Tokenization and Encoding: Signal-specific encoders convert iEEG waveform patches and binned spike counts into channel–time tokens for shared processing.The iEEG encoder uses temporal convolution with a residual linear pathway, while the spike encoder uses an MLP.
  • Shared Spatiotemporal Attention: The shared spatiotemporal backbone alternates channel attention and temporal attention to model dependencies across recordings and time.Learnable signal-type and temporal-position embeddings accompany the token representations.
  • Self-supervised Pretraining Objectives: Masked reconstruction decodes contextualized tokens back into signal-specific patch spaces, with losses computed only on masked valid channels or units.Separate reconstruction objectives reflect the differing statistical properties of iEEG waveforms and spike counts.
  • Self-supervised Pretraining Objectives: Channel-view alignment processes overlapping channel or unit subsets of the same segment and encourages stable segment representations under subsampling.The two views share the signal-specific encoder and ST backbone before global pooling and alignment.
  • Pretraining Schedule: iEEG and spike minibatches alternate at a 1:1 step ratio, updating signal-specific modules with their own inputs and the shared backbone with both signal types.This schedule trains shared parameters across unpaired multimodal data.

Experiments

iBrain is evaluated using a heterogeneous corpus spanning diverse invasive recordings and representative downstream baselines. The experimental setup includes large-scale self-supervised pretraining and comparisons across iEEG and spike-based tasks.

  • Pretraining Data: The pretraining corpus spans diverse species, recording interfaces, brain regions, and behavioral contexts, using iEEG and spike recordings.Recordings are divided into non-overlapping one-second windows, with iEEG resampled to 500 Hz and spikes represented as binned spike-count sequences.
  • Pretraining Setup: iBrain uses a six-layer Transformer with signal-specific reconstruction and channel-view alignment during pretraining.The backbone has hidden dimension 256, eight attention heads, feed-forward dimension 1024, and dropout 0.1.
  • Evaluation Protocol: Evaluation follows subject-specific fine-tuning on subsets of recordings and testing on held-out recordings.Baselines are tailored to iEEG and spike signal families, including Brant, BrainBERT, TOTEM, and PopT prediction heads for iEEG benchmarks.
  • Evaluation Protocol: Table 1 compares iBrain with representative baseline methods on downstream benchmarks.

Comparison with Baselines

iBrain outperforms representative baselines across the evaluated iEEG language tasks and achieves top performance on most spike-based benchmarks. The results support strong performance across heterogeneous invasive decoding settings.

  • iEEG Benchmarks: iBrain improves the strongest baseline on all four Brain TreeBank iEEG tasks: Pitch 0.60 to 0.68, Volume 0.75 to 0.89, Onset 0.81 to 0.86, and Speech 0.79 to 0.89.
  • Spike Benchmarks: iBrain obtains the best results on MC-Maze, Area2-Bump, and Perich T-CO among spike-based benchmarks.It does not achieve the best result on Perich T-RT, indicating that performance remains weaker for some random-target reaching settings.
  • Overall Results: iBrain achieves top performance on seven of the eight evaluated benchmarks.
  • Pretraining Comparisons: Table 2 compares iBrain under iEEG-only, spike-only, and joint iEEG–spike pretraining.

Joint Pretraining

Joint pretraining converges stably while combining iEEG and spike recordings. It achieves the best overall transfer results across language-related and motor decoding task groups compared with single-signal pretraining.

  • Training Dynamics: The total joint-pretraining loss decreases steadily, with iEEG reconstruction declining, spike reconstruction remaining low and stable, and both alignment losses staying small.This training behavior indicates stable optimization across the two signal types and their alignment objectives.
  • Downstream Transfer: Joint iEEG + Spike pretraining achieves the best overall results across both language-related and motor decoding task groups.It improves over iEEG-only pretraining on Speech and Volume and over spike-only pretraining on MC-Maze, Area2-Bump, Perich T-CO, and Perich T-RT.
  • Training Dynamics: Figure 3 presents the joint pretraining loss curves.

Few-shot Evaluation

iBrain maintains competitive neural decoding performance with limited labeled trials, improving as more labels become available. Half of the labeled data nearly matches full-data performance on both evaluated tasks.

  • Label Efficiency: On MC-Maze, average R2 rises from 0.817 with 10% labels to 0.863, 0.896, and 0.914 with 20%, 50%, and 100% labels.
  • Label Efficiency: On Area2-Bump, average R2 increases from 0.588 with 10% labels to 0.755, 0.860, and 0.903 with 20%, 50%, and 100% labels.
  • Label Efficiency: Only 50% of labeled trials leaves performance gaps of 0.018 on MC-Maze and 0.043 on Area2-Bump relative to full-data training.
  • Evaluation Design: Figure 4 reports average R2 on MC-Maze and Area2-Bump across different fractions of labeled training trials.
  • Ablation Context: Table 3 reports mean R2 for model ablations under both fine-tuning and frozen-backbone settings.

Model Ablation

Ablations show that channel-view alignment and both channel and temporal attention contribute to iBrain’s downstream decoding performance. Their effects vary across benchmarks and evaluation settings.

  • Channel-view alignment: Removing channel-view alignment lowers performance across all settings, with the largest drop on frozen-backbone MC-Maze, where R2 falls from 0.904 to 0.872.This pattern suggests that channel-view alignment improves transferability under frozen-backbone evaluation.
  • Spatiotemporal attention: Removing either channel or temporal attention reduces performance, supporting the importance of modeling cross-channel interactions and temporal dependencies.Channel attention consistently affects both benchmarks, while temporal attention has a larger effect on Area2-Bump than MC-Maze.
  • Spatiotemporal attention: Channel and temporal attention produce distinct degradation patterns across benchmarks and evaluation settings.The differing effects indicate complementary contributions to downstream decoding.

Scaling Analysis

Increasing balanced iEEG-and-spike pretraining data from 500 to 7,160 hours improves downstream R2, but gains become smaller beyond 2,000 hours. The results suggest that future scaling may require greater diversity, not duration alone.

  • Performance scaling: R2 increases from 0.882 to 0.914 on MC-Maze and from 0.887 to 0.903 on Area2-Bump as pretraining data grows from 500 to 7,160 hours.Each scale contains equal durations of iEEG and spike recordings.
  • Performance scaling: Downstream performance improves monotonically with increasing pretraining data, but gains become smaller beyond 2,000 hours.The scaling behavior is less pronounced than often observed in general-domain foundation models.
  • Implications: Future scaling may benefit from expanding subject, task, and acquisition diversity in addition to increasing overall data volume.The paper suggests that redundancy within subjects and sessions may cause effective diversity to grow more slowly than recording duration.

Encoder Representations

Fine-tuned iBrain representations organize MC-Maze trials by movement direction more clearly than raw spike-count signals. Similar movement angles cluster locally, while different angle ranges become more separated.

  • Representation structure: Raw spike-count signals are highly entangled, whereas fine-tuned iBrain representations show clearer separation across movement directions.The comparison uses t-SNE on the held-out MC-Maze test split, grouped into eight movement-angle ranges.
  • Representation structure: Samples with similar movement angles tend to cluster locally, while different angle ranges are more clearly separated after fine-tuning.This organized distribution contrasts with the substantial overlap among raw spike-count samples.

Conclusion

iBrain unifies cortical-surface and deep-brain iEEG with intracortical spiking through signal-specific encoders, shared spatiotemporal modeling, and channel-view alignment. It achieves strong performance across diverse tasks while indicating that future gains may depend on recording diversity as well as scale.

  • Unified model: iBrain jointly learns from cortical-surface and deep-brain iEEG and intracortical spiking using signal-specific encoders and a shared spatiotemporal Transformer.Channel-view alignment and reconstruction objectives support representations across diverse recording settings.
  • Results: iBrain achieves the strongest performance on seven of eight downstream benchmarks spanning motor decoding and language-related tasks.The paper also reports that joint pretraining improves upon signal-specific pretraining.
  • Implications: Ablations confirm complementary contributions from spatiotemporal modeling and channel-view alignment, while few-shot and scaling analyses show label-efficiency and data-scaling benefits.The scaling analysis also suggests that future gains may depend on greater diversity across subjects, tasks, and acquisition settings rather than recording duration alone.
Loading 2609.06960v1…