PaperlayerView exact revision on arXiv

Source-linked AI summary

MEG-XL: Data-Efficient Brain-to-Text via Long-Context Pre-Training

Dulhan Jayalath, Oiwi Parker Jones

arXiv:2602.02494v2cs.LGq-bio.NC

TL;DR

Brain-to-text decoders usually use brief neural-data windows, limiting access to linguistic context and requiring extensive subject-specific recordings. MEG-XL pre-trains on 2.5-minute MEG samples and improves data-efficient word decoding, reducing the subject-specific data bottleneck.

  • Problem

    Brain-to-text decoders typically use milliseconds-to-seconds samples, while clinical users such as paralysed patients cannot provide extensive training recordings.

  • Method

    MEG-XL pre-trains on 2.5-minute MEG samples, 5–300× longer than prior pre-trained models and corresponding to 191k tokens.

  • Results

    MEG-XL generalises better than supervised and brain-foundation-model alternatives in low-data word decoding, reducing required data from dozens to 1–2 hours.

  • Takeaways & Limitations

    Long-context pre-training helps models acquire priors that exploit extended neural context and transfer better to word decoding, with diminishing returns after 100s.

  • Takeaways & Limitations

    Evaluation focuses on perceived speech and a relatively limited 50-word vocabulary, with performance decreasing as vocabulary size increases.

Abstract

from arXiv · show

Clinical brain-to-text interfaces are designed for paralysed patients who cannot provide extensive training recordings. Pre-training improves data-efficient generalisation by learning statistical priors across subjects, but these priors critically depend on context. While natural speech might unfold gradually over minutes, most methods pre-train with only a few seconds of context. Thus, we propose MEG-XL, a model pre-trained with 2.5 minutes of MEG context per sample, 5-300x longer than prior work, and equivalent to 191k tokens, capturing extended neural context. Fine-tuning on the task of word decoding from brain data, MEG-XL matches supervised performance with a fraction of the data (e.g. 1hr vs 50hrs) and outperforms brain foundation models. We find that models pre-trained with longer contexts learn representations that transfer better to word decoding. Our results indicate that long-context pre-training helps exploit extended neural context that other methods unnecessarily discard. Code, model weights, and instructions are available at https://github.com/neural-processing-lab/MEG-XL .

1. Introduction

MEG-XL addresses the mismatch between brief neural-data windows and the longer context carried by speech-related brain activity by pre-training on 2.5-minute MEG samples. The resulting long-context priors improve data-efficient transfer to word decoding and provide evidence that longer pre-training contexts yield better representations.

  • Motivation: Prior brain-to-text decoders typically use millisecond-to-second brain-data windows, ignoring long-range context present in neural recordings.Short windows are commonly chosen to match the unit being decoded, such as a phoneme, syllable, or word.
  • Motivation: 50% improvement in word decoding accuracy was reported when sentence-level MEG context replaced isolated-word classification.The result comes from prior non-invasive decoding work using neural responses to all words in a perceived sentence.
  • MEG-XL: 2.5-minute MEG samples make MEG-XL contexts 5–300× longer than prior pre-trained models, corresponding to 191k tokens.The model uses masked token prediction across recordings from hundreds of subjects and hundreds of hours of diverse tasks.
  • Results: Fine-tuning across three MEG speech datasets generalises to new subjects with limited data better than the state-of-the-art supervised approach.MEG-XL outperforms brain foundation-model alternatives in the shallow low-data regime while remaining competitive with deep subject recordings.
  • Interpretation: Longer pre-training contexts produce better brain-to-text representations and improve zero-shot prediction of masked brain signals from unseen datasets.The reported mechanism is the emergence of selective, hierarchical attention, progressing from local processing in early layers to global integration in later layers.

2. Method

MEG-XL tokenizes multichannel MEG recordings into discrete residual codes and models long temporal contexts with an efficient criss-cross transformer. Masked-token pre-training reconstructs contiguous signal blocks from surrounding temporal and spatial information.

  • 2.1. Tokenizer: MEG signals are independently tokenized per sensor with BioCodec using six residual quantization levels and a vocabulary of 256 codes.The tokenizer downsamples time by r = 12 and represents each recording as discrete codes across sensors, timesteps, and quantization levels.
  • 2.1. Tokenizer: Tokenization compresses the temporal input and supplies discrete targets for masked-token self-supervision.These two roles allow the transformer to process longer contexts while predicting structured residual codes.
  • 2.2. Model Architecture: Token embeddings concatenate RVQ codebook vectors, project to the model dimension, and combine with sensor position, orientation, and type embeddings.Position and orientation use Gaussian Fourier features, while sensor type uses a learned embedding; the resulting embeddings are summed before transformer processing.
  • 2.2. Model Architecture: The eight-layer transformer factorizes attention into temporal and spatial operations across sensors and timesteps.This criss-cross design reduces full-attention cost from O((CT′)^2) to O(C·T′^2 + T′·C^2), enabling long-context modelling without compressing sensor channels.
  • 2.3. Masked Prediction Objective: Pre-training masks random contiguous 3-second blocks across sensors until 40% of tokens are masked, then predicts each masked RVQ code from visible context.The objective averages cross-entropy over masked positions and quantization levels; padded channels are excluded through sensor masks.

3. Experiments

Experiments evaluate MEG-XL on contextual word decoding across three MEG datasets and compare it with supervised, randomly initialized, and brain foundation-model baselines. Results show strongest benefits in data-constrained, shallow multi-subject settings, while longer pre-training contexts improve transferable representations and masked neural-signal prediction.

  • Experimental setup: MEG-XL is fine-tuned end-to-end for contextual word decoding across three MEG speech datasets spanning shallow multi-subject, deep single-subject, and intermediate regimes.Inputs concatenate 50 three-second word-locked neural windows into 150 seconds, and predictions use semantic word embeddings with nearest-neighbor cosine retrieval.
  • Data efficiency compared to supervised learning: On LibriBrain, MEG-XL and d’Ascoli et al. (2025) perform similarly until approximately 2.5 hours, after which the supervised method pulls ahead.The comparison indicates that deep single-subject recordings can eventually favor learning from scratch over pre-training.
  • Comparison to foundation models: MEG-XL shows the best generalisation under data constraints and is jointly best with all training data across six pre-trained-model comparisons.At 13% of training data, most baselines collapse to chance; with full data, MEG-XL matches or exceeds alternatives on LibriBrain and MEG-MASC and remains within 1.1 percent of BrainOmni on Armeni.
  • Comparison to foundation models: 47–57% accuracy against a 20% random baseline demonstrates that MEG-XL remains effective in the low-data regime where most baselines remain near chance.The paper identifies this regime as especially relevant for transfer to new users with limited subject data.
  • Data efficiency compared to supervised learning: MEG-XL generally outperforms d’Ascoli et al. (2025) when subject data is limited, with gains persisting across shallow multi-subject datasets.On MEG-MASC, improvements exceed 25% at certain points across 54 hours of training data; on Armeni, accuracy is roughly 10% above the supervised method until around 15 hours.
  • Long-context representation quality: Longer pre-training contexts yield better word-decoding representations, with diminishing returns after 100 seconds and little benefit from inference context beyond pre-training context.Linear probes use frozen backbones; full-context and matched-context conditions show that long-context utilisation is learned during pre-training.
  • What long-context pretraining teaches: Longer-context pre-training consistently improves zero-shot masked neural-signal prediction on unseen datasets, with scaling continuing through 150 seconds.Attention analyses associate this with local-to-global hierarchical processing and lower attention entropy, indicating more selective temporal attention.

4. Discussion

Long-context pre-training reduces subject-specific data requirements for contextual word decoding, although supervised within-subject training remains strongest with extensive data. The benefits are linked to learned use of distant context but remain bounded by vocabulary and unresolved questions about learned structure.

  • Pre-training with long neural context reduces total data requirements from dozens of hours to 1–2 hours and requires only tens of minutes per subject.This expands applicability to subjects unable to provide long training recordings.
  • Models with longer pre-training windows generalise better in word decoding, with diminishing returns after 100s of context.Zero-shot masked-brain-activity prediction continues scaling to 150s without saturation.
  • Long-context pre-training teaches models to attend selectively and process near-to-far context hierarchically.Its proposed advantage is learning when and how to use distant context rather than merely acquiring the ability to attend to it.
  • The evaluation focuses on perceived speech and a relatively limited 50-word vocabulary, with performance decreasing for larger vocabularies.Understanding the learned structure behind the attention patterns remains unresolved.
  • Long neural context requires appropriate long-context statistical priors for data-efficient generalisation; supplying long inputs alone is insufficient.The discussion compares this requirement with language-model pre-training on long documents for coherence.

Impact Statement

The work targets non-invasive speech decoding with possible applications for people who have lost the ability to speak, while clinical deployment and governance remain unresolved.

  • The paper presents non-invasive speech decoding as a potential technology for individuals who have lost the ability to speak.
  • Clinical deployment remains distant because performance is below communication-aid requirements and substantial work remains.
  • Neural decoding raises privacy concerns because it infers mental content from brain activity.The paper calls for norms covering consent, data ownership, and assistive versus surveillant applications.

A.1. Preprocessing

The preprocessing pipeline filters and downsamples MEG recordings to reduce timepoints and enable longer contexts, while acknowledging a potential aliasing risk.

  • Recordings are high-pass filtered at 0.1Hz, low-pass filtered at 40Hz, and resampled to 50Hz.The pipeline follows prior word-decoding preprocessing practice.
  • Resampling reduces the number of timepoints, allowing the model to use longer contexts.
  • The preprocessing technically risks creating aliasing artefacts, with Nyquist-compliant resampling results reported in Appendix C.
  • The 0.1Hz high-pass filter removes slow drift and scanner artifacts without removing statistical dependencies between neural responses separated by minutes.Frequency content and dependency timescale are treated as distinct.

A.2. Pre-training Datasets

MEG-XL is pre-trained on continuous 2.5-minute MEG windows drawn from three datasets spanning ageing, language, and extended natural-speech data.

  • The pre-training samples are continuous 2.5-minute windows of MEG data.
  • CamCAN contributes data from approximately 700 healthy subjects across rest, sensorimotor, and passive sensory tasks.
  • MOUS contributes reading and listening data from 104 subjects in a Dutch language study.
  • SMN4Lang contributes extended natural-speech listening data from 12 subjects in Mandarin Chinese.

A.3. Foundation Model Baselines

The evaluation compares MEG-XL with several brain foundation models and task-specific baselines under a common fine-tuning setup, while accounting for modality and context-length constraints.

  • Evaluation setup: Baseline backbones receive 2.5-minute MEG samples and are fine-tuned end-to-end with an MLP head predicting target embeddings.Word-aligned backbone embeddings are time-pooled, flattened, concatenated, and independently classified by the MLP head.
  • Foundation-model baselines: BioCodec is evaluated as a single-channel tokenizer, with two linear transformers resolving spatial features across MEG channels.The resulting embeddings are pooled over time before downstream prediction.
  • Foundation-model baselines: BIOT required a trainable channel-reduction projection and a 24-second neural context because its pre-training supported at most 18 channels and its positional embeddings had limited length.
  • Foundation-model baselines: EEGPT also required a trainable projection to reduce MEG channels because it was pre-trained for at most 58 channels.
  • Foundation-model baselines: BrainOmni accepts sensor position, orientation, and type information directly, whereas LaBraM required a 15-second context because its learned time embeddings limit context length.

B. Larger Retrieval Set Results

The appendix repeats comparisons with a larger 250-word retrieval vocabulary and reports a Nyquist-compliant preprocessing alternative constrained by GPU memory.

  • Nyquist-compliant preprocessing: Resampling at 100Hz avoids aliasing by satisfying the Nyquist criterion for a 40Hz low-pass filter.
  • Nyquist-compliant preprocessing: 75s replaces 150s of context in the Nyquist-compliant configuration because GPU VRAM limits the longer setting.Table 5 compares the two context configurations.

D. Token-Matched Neural Context Scaling

The token-matched analysis addresses confounding between context length, total training information, and optimisation steps, while documenting a tokenizer trade-off for MEG reconstruction.

  • Token-matched comparison: Context-length comparisons are confounded because fixed training steps change total information exposure, whereas fixed data changes the number of optimisation steps.Non-linear optimisation makes either comparison imperfect, motivating a principled compromise.
  • Tokenizer choice: BioCodec was selected over BrainTokenizer because it reconstructed MEG data at 50Hz with lower error, despite producing more tokens by retaining the channel dimension.

F. Analysing Temporal Attention Heads

The analysis characterises temporal attention using attention distance and entropy, alongside token-matched context-scaling results that test whether longer contexts improve downstream decoding.

  • Attention metrics: Temporal attention weights are computed as A = softmax(QK^⊤/d), where d is the head dimension.
  • Attention metrics: Mean attention distance measures how far each query position attends on average, with values converted from timesteps to seconds using a 0.24-second factor.The conversion uses tokenizer downsampling ratio r = 12 and sample rate f = 50.
  • Context scaling: Token-matched pre-training shows larger contexts systematically improving downstream performance, with diminishing returns and possible regression after 150s.The results are noisier because three pre-training seeds and linear probes were used for each context length instead of five.
  • Attention metrics: Attention entropy measures distribution uniformity: higher values indicate broader attention, while lower values indicate concentration on fewer positions.Metrics aggregate results across 100 held-out MEG segments and five random seeds per context length.
Loading 2602.02494v2…