Source-linked AI summary
Decoding speech perception from non-invasive brain recordings
Alexandre Défossez, Charlotte Caucheteux, Jérémy Rapin, Ori Kabeli, Jean-Rémi King
TL;DR
Decoding natural speech from noisy, variable non-invasive brain recordings remains challenging, while the brain representations most suitable for language decoding are poorly known. The paper addresses these gaps with a multi-participant model using pretrained speech representations and reports accurate identification of perceived speech segments from MEG.
Problem
Natural speech decoding from non-invasive brain recordings is limited by noisy, highly variable signals and uncertainty about the acoustic, phonetic, lexical, and semantic representations used by the brain.
Method
The model combines a common convolutional architecture trained across participants with self-supervised speech representations and a contrastive decoding objective.
Results
Up to 41% accuracy out of more than 1,000 possibilities was achieved from 3 seconds of non-invasive recordings, reaching 80% in the best participants.
Takeaways & Limitations
The decoder can identify perceived words and phrases absent from training, while its predictions primarily reflect high-level lexical and contextual semantic representations.
Takeaways & Limitations
The approach addresses speech perception in healthy volunteers, so decoding intended speech production in clinical settings still requires further challenges to be addressed.
Abstract
from arXiv · showhide
Decoding speech from brain activity is a long-awaited goal in both healthcare and neuroscience. Invasive devices have recently led to major milestones in that regard: deep learning algorithms trained on intracranial recordings now start to decode elementary linguistic features (e.g. letters, words, spectrograms). However, extending this approach to natural speech and non-invasive brain recordings remains a major challenge. Here, we introduce a model trained with contrastive-learning to decode self-supervised representations of perceived speech from the non-invasive recordings of a large cohort of healthy individuals. To evaluate this approach, we curate and integrate four public datasets, encompassing 175 volunteers recorded with magneto- or electro-encephalography (M/EEG), while they listened to short stories and isolated sentences. The results show that our model can identify, from 3 seconds of MEG signals, the corresponding speech segment with up to 41% accuracy out of more than 1,000 distinct possibilities on average across participants, and more than 80% in the very best participants - a performance that allows the decoding of words and phrases absent from the training set. The comparison of our model to a variety of baselines highlights the importance of (i) a contrastive objective, (ii) pretrained representations of speech and (iii) a common convolutional architecture simultaneously trained across multiple participants. Finally, the analysis of the decoder's predictions suggests that they primarily depend on lexical and contextual semantic representations. Overall, this effective decoding of perceived speech from non-invasive recordings delineates a promising path to decode language from brain activity, without putting patients at risk for brain surgery.
1 Introduction
Decoding language has advanced with invasive recordings but remains limited by surgery, noisy non-invasive signals, and narrow decoding targets. This work proposes a multi-participant, self-supervised approach to decode speech from non-invasive brain activity.
- Invasive brain-computer interfaces have decoded elementary speech features, but their recordings require brain surgery and are difficult to maintain chronically.
- Non-invasive MEG and EEG are safe and potentially wearable, yet their signals vary greatly across sessions and individuals.
- Language decoding has largely remained limited to invasive recordings or impractical tasks targeting spectrograms, letters, phonemes, or small word sets.
- The proposed model uses one architecture across many participants and self-supervised deep speech representations to decode perceived speech from non-invasive recordings.
- The study identifies contrastive learning, pretrained speech models, and multi-subject architecture as important choices for efficient continuous EEG and MEG processing.
2 Method
The method aligns brain-derived representations of 3-second MEG/EEG windows with pretrained or learned speech representations using contrastive learning, enabling segment-level zero-shot decoding. Its architecture combines spatial and subject-specific processing with convolutional decoding, while prediction analyses test lexical and semantic features.
- 2.1 Problem formalization: The model maps 3-second M/EEG windows X to brain representations Z that align with speech representations Y through a contrastive objective.At evaluation, it assigns probabilities to candidate speech segments using representation similarity.
- 2.2.1 Contrastive loss: Unlike direct regression, contrastive learning trains the decoder to distinguish different speech segments rather than reconstruct every spectrogram dimension equally.The objective uses batch elements as negative samples during training and test-set segments as negatives during evaluation.
- 2.2.2 Brain module: The brain module combines spatial attention, a subject-specific 1x1 convolution, and five residual dilated convolutional blocks to process raw M/EEG signals.The subject layer is learned separately for each participant to leverage inter-subject variability.
- 2.2.3 Speech module: The speech module replaces low-level Mel spectrograms with latent representations from either end-to-end Deep Mel or pretrained wav2vec 2.0.Because the pretrained wav2vec 2.0 approach was more efficient, subsequent analyses focus on it.
- 2.2.3 Speech module: The decoder’s predictions are analyzed using phonetic, frequency, part-of-speech, word-embedding, and phrase-embedding features.These features span dimensions from phonetic descriptors to 1,024-dimensional phrase embeddings.
3 Results
The model decodes speech segments from non-invasive M/EEG recordings, with performance shaped by speech representations, architectural components, and training across participants.
- 3.1 Accurately decoding speech from M/EEG recordings: 70.7% top-10 accuracy was achieved on average across MEG subjects for identifying the correct segment among more than 1,000 possibilities.Top-1 accuracy reached 41.3%; more than half of samples ranked the true segment first or second.
- 3.1 Accurately decoding speech from M/EEG recordings: EEG decoding reached 17.7% and 25.7% top-10 accuracy across the two analyzed datasets, exceeding the random baseline.The random model achieved less than 1% top-10 accuracy on the same MEG datasets.
- 3.3 ‘Speech module’ evaluation.: Contrastive learning improved decoding over the regression baseline by 2% to 42.7%, while pretrained wav2vec 2.0 representations performed best among tested speech targets.Targeting latent speech representations further supplemented the contrastive-learning gains.
- 3.4 ‘Brain module’ evaluation.: Removing skip connections, spatial attention, initial or final convolutional layers systematically decreased performance, while clamping was essential for training.Additional experiments found the end-to-end architecture robust to M/EEG artefacts and requiring little preprocessing.
- 3.4 ‘Brain module’ evaluation.: Decoding performance steadily increased as the model was trained with more subjects on the two MEG datasets.The ablations also showed that the subject layer was important, although its gain was modest compared with the introduced subject embedding.
- 3.5 Prediction analysis.: Prediction analyses examined whether decoder probabilities corresponded to phonetic, lexical, syntactic, frequency, word-embedding, or phrase-embedding features.A ridge regression with five-split cross-validation measured Pearson R correspondence between each feature and the probability assigned to the true word.
4 Discussion
The model decodes perceived speech from non-invasive brain recordings, but clinical speech-production decoding still requires substantial extensions. Its performance reflects scalable multi-participant modeling and pretrained speech representations, while comparisons and EEG-to-MEG differences define important boundaries.
- 4.1 Efficiently extracting brain signals.: Minimal preprocessing and a shared architecture support decoding across participants, devices, and stimuli, with performance increasing as more participants join training.The authors report that advanced M/EEG preprocessing offers no major advantage for this task, while subject-specific modeling improves decoding.
- 4.2 How is language represented in the brain?: Pretrained speech representations and contrastive learning outperform spectrogram-based alternatives, while predictions primarily reflect lexical and contextual semantic features.The authors caution that the nature and structure of these high-level representations remain unresolved.
- 4.3 Comparison to previous works: The approach distinguishes several hundred words and can generalize to words absent from training, exceeding the small vocabularies typical of earlier non-invasive studies.For the Gwilliams dataset, performance reaches 40% despite nearly 36% of words never appearing during training.
- 4.3 Comparison to previous works: Perceived-word decoding can approach invasive benchmarks in restricted 50-word settings, but comparisons require caution because perceived and produced words have different brain bases.The authors describe performance as potentially relevant to future speech-production applications without claiming equivalence between tasks.
- 4.4 Remaining steps to decode speech production in the clinics: Clinical deployment remains bounded by the need to adapt the model to speech production, replace fixed candidate sets with generative decoding, and address substantial EEG–MEG differences.MEG adaptation to clinical environments also faces noisy magnetic settings and limited deployment of room-temperature systems.
5 Data Availability
The study uses four public datasets with distinct licensing arrangements, including RU-DI-HD-1.0, CC0 1.0 Universal, and CC BY 4.0.
- The Schoffelen dataset was provided partly by the Donders Institute under a RU-DI-HD-1.0 licence.
- The Gwilliams and Broderick datasets are available under CC0 1.0 Universal.
- The Brennan and Hale dataset is available under a CC BY 4.0 licence.
- All audio files were provided by the authors of their respective datasets.
6 Code Availability
The complete dataset-processing, model-training, evaluation, and method source code is publicly available through GitHub, with a fixed release archived on Zenodo.
- The complete source code for processing datasets, training and evaluating models, and implementing the method is available at github.com/facebookresearch/brainmagick.
- The GitHub code is provided under the CC-NC-BY 4.0 license.
- A fixed version of the code is archived on Zenodo under DOI 10.5281/zenodo.8114374.
8 Statement of contributions.
The contribution statement assigns leadership, dataset curation, training-pipeline development, model training, evaluation, and analysis responsibilities across the author team.
- Alexandre Défossez and Jean-Remi King led the project, with King responsible for data curation.
- Jérémy Rapin, Ori Kabeli, and Alexandre Défossez built the training pipeline.
- Ori Kabeli and Alexandre Défossez handled model training and hyper-parameter search.
- Charlotte Caucheteux built the speech module and evaluation pipeline.
- Alexandre Défossez, Charlotte Caucheteux, Ori Kabeli, and Jean Remi King conducted in-depth data and results analysis.
A.1 Datasets
The paper’s dataset materials include licensing information for the Schoffelen, Gwilliams, Broderick, and Brennan–Hale datasets, while the cited table reports segment-level top-1 accuracy.
- The Schoffelen dataset was provided partly by the Donders Institute under a RU-DI-HD-1.0 licence.
- The Gwilliams and Broderick datasets are available under CC0 1.0 Universal.
- The Brennan and Hale dataset is available under a CC BY 4.0 licence.
- Table A.1 reports segment-level top-1 accuracy related to Table 2.
A.2 Impact of clamping
Clamping rescales each recording to reduce the influence of unusually large electromagnetic perturbations. The appendix reports the resulting top-10 segment-level accuracy.
- Clamping maps the robust-scaled data range [-1, 1] to the [0.25, 0.75] quantile range independently for each recording.The resulting M/EEG signals are expected to have a scale of approximately 1.
- Table A.2 reports top-10 segment-level accuracy for the clamped model.
A.3 Comparison with Autoreject
The appendix compares the model with Autoreject preprocessing, which detects and corrects corrupted channels using spatial neighborhoods. It also presents representative segment-level prediction visualizations for a median-scoring subject.
- Autoreject detects and corrects corrupted channels based on their spatial neighborhood before model training.It can also reject highly corrupted samples, but sample rejection is excluded because it would change the test-set definition.
- Figure A.1 shows predictions for five representative speech segments using the subject with median decoding scores across the cohort.The top and bottom segments are respectively the easiest and hardest to decode.
- The comparison is labeled as a method-level comparison with Gwilliams2022 using 16 recordings.
A.4 Impact of EEG/MEG time offset
The appendix examines timing alignment alongside several decoding analyses. It reports a small effect of the speech–brain delay choice and additional results for Mel features, optimization, zero-shot words, isolated words, and attention patterns.
- A 150 ms speech–brain delay is used, while setting the delay to 0 reduces top-10 accuracy by only 0.5% on the Gwilliams2022 dataset.The authors describe the overall impact of this parameter as small.
- Using more Mel frequency bands produces a small accuracy increase across the evaluated models.The minimum and maximum frequencies remain fixed in this analysis.
- Larger batch sizes generally improve performance, whereas larger learning rates produce instabilities in the reported Gwilliams et al. evaluation.The main study uses batch size 256 and learning rate 3e−4.
- For MEG, zero-shot word decoding is remarkably close to decoding words present during training, whereas EEG zero-shot performance falls dramatically.This supports decoding words never present during training more strongly for MEG than EEG.
- 22.7% zero-shot top-1 accuracy is achieved with a 50-word test vocabulary, with subjects peaking at 42.9% for isolated words.The evaluation uses segments from -300 ms to +500 ms relative to word onset.
- Figure A.2 compares single-word decoding with vocabularies of 1,000 and 50 words.