Source-linked AI summary
LibriBrain100: One Hundred Hours of Broad and Deep MEG Data for Neural Speech Decoding at Scale
Francesco Mantegna, Dulhan Jayalath, Gereon Elvers, Tasha Kim, Benjamin Ballyk, Alex Fung, SungJun Cho, Teyun Kwon, Luisa Kurth, Miran Özdogan, Gilad Landau, Pratik Somaiya, Natalie Voets, Mark Woolrich, Oiwi Parker Jones
TL;DR
Neural speech decoding lacks the shared data and evaluation infrastructure needed for reliable progress, while practical BCIs require methods that work with limited subject-specific data. LibriBrain100 addresses these needs by combining deep single-subject and broad multi-subject MEG recordings with standardised word-classification evaluation. The dataset supports strong within-subject decoding and cross-subject transfer with substantially reduced fine-tuning data, although it remains limited to passive listening and does not include a brain-to-text baseline.
Problem
Neural speech decoding lacks shared data, standard benchmarks, and common evaluation protocols, while real-world BCIs require minimal per-user data collection.
Method
LibriBrain100 combines ∼80 hours from subject 0 with ∼40 minutes from each of 32 additional subjects and evaluates word classification using standard splits and MEG-XL fine-tuning.
Results
Including subject 0’s ∼80 hours improves cross-subject generalisation by about 15 percentage points, while reducing fine-tuning data causes minimal degradation.
Takeaways & Limitations
Deep single-subject data can improve decoding for many individuals with much less subject-specific data, supporting investigation of data-efficient speech BCI generalisation.
Takeaways & Limitations
All data were collected during passive listening to continuous speech, and the release deliberately omits a brain-to-text baseline.
Abstract
from arXiv · showhide
We introduce LibriBrain100, a large-scale MEG dataset for speech decoding designed from the ground up for reproducible, standardised evaluation. LibriBrain100 more than doubles the size of the original LibriBrain release, resulting in over 100 hours of high-quality MEG acquired while subjects listened to naturalistic continuous speech. With $\sim$80 hours from a single subject, LibriBrain100 sets a new record for deep, within-subject neural data (8$\times$ more than the next comparable dataset and roughly 80$\times$ more than other datasets). To demonstrate the payoff of this depth-first design, we evaluate on a word-classification benchmark---an increasingly well-established stepping stone towards the open challenge of noninvasive brain-to-text decoding. Using an existing decoding model, we achieve state-of-the-art performance---validating both the quality of the recordings and the value of within-subject data at scale. Because collecting 80 hours of data per user is impractical for real-world applications, we also collected $\sim$40 minutes of additional data from each of 32 subjects. Using the same word-classification benchmark, we demonstrate the value of broad multi-subject data: supervised finetuning of a pre-trained model can substantially compensate for limited per-subject data. We provide standard train, validation, and test splits, all reproducible through an open-sourced Python library that supports easy downloading, optional preprocessing, and data loading for common deep learning frameworks. In addition, the dataset and evaluation infrastructure are being released alongside an open machine-learning competition with a public leaderboard for standardised benchmarking. Ultimately, our hope is that LibriBrain100 will accelerate progress towards practical non-invasive brain-computer interfaces, capable of restoring communication to people living with severe paralysis.
1 Introduction
LibriBrain100 targets the lack of shared infrastructure for neural speech decoding by extending LibriBrain in depth, breadth, stimulus diversity, and reproducible evaluation. It evaluates word classification within and across subjects using standardised benchmarks and open tooling.
- Standard benchmarks, shared data, and common evaluation protocols are needed to determine what neural speech decoding works and by how much.
- LibriBrain100 extends within-subject recordings from ∼50 to ∼80 hours and adds stimuli designed to address phonetic and semantic decoding.The added stimuli include TIMIT, MOCHA-TIMIT, and podcast narratives.
- Word classification extends the evaluation curriculum toward representations that are both phonetic and semantic and closer to brain-to-text goals.
- 32 additional subjects contribute ∼40 minutes each, enabling cross-subject generalisation and data-efficient benchmarking.
- The release provides standard splits, baselines, linguistic annotations, and Python tooling for reproducible within-subject and cross-subject evaluation.The resources support streamlined deep-learning integration and a planned open machine-learning competition.
2 Related Work
Prior MEG speech-decoding datasets differ mainly in breadth and depth. LibriBrain100 preserves an unusually deep within-subject design while adding substantial subject breadth and more diverse stimuli.
- LibriBrain100 has ∼8× more deep within-subject data than the next non-LibriBrain dataset and ∼80× more than the rest.
- MOUS, MEG-MASC, and Le Petit Prince use breadth-first designs with 27–96 subjects but under 2 hours per subject.
- Armeni and LibriBrain use depth-first designs with 10 and 52.3 hours per subject, respectively, but only 1–3 subjects.
- LibriBrain100 combines ∼80 hours from one subject with ∼40 minutes from each of 32 additional subjects, spanning 0.6–80.0 hours per subject.It has the third-highest number of subjects among the six datasets in Table 2.
- Its stimulus set adds TIMIT, MOCHA-TIMIT, and 30 podcasts from The Moth to Sherlock Holmes readings, supporting control of sound- and meaning-based activity.
3 The LibriBrain100 Dataset
LibriBrain100 provides over 100 hours of annotated MEG speech-listening data, combining a deep single-subject component with a broad multi-subject component and reproducible access infrastructure.
- Dataset composition: Over 100 hours of MEG recordings from 33 subjects are paired with time-locked linguistic annotations for supervised decoding tasks.The recordings were collected during connected-speech listening using a 306-sensor MEG system.
- Dataset composition: Approximately 80 hours come from subject 0, while 32 additional subjects contribute approximately 20 minutes across each of two sessions.This structure prioritises within-subject depth while providing more practical amounts of data for additional subjects.
- Stimuli: The stimulus collection combines the complete Sherlock Holmes canon with TIMIT, MOCHA-TIMIT, and 30 podcast stories from The Moth.These corpora were selected to expand phonetic and semantic coverage while retaining substantial naturalistic speech data.
- Reproducible evaluation: Standard training, validation, and testing splits are provided, including established TIMIT partitions and newly created splits where no precedent existed.Sherlock sessions 11 and 12 are held out for validation and testing, respectively.
- Formats and access: The release offers raw BIDS/FIF data and minimally preprocessed float32 HDF5 data for machine-learning use.The raw and serialised formats are accompanied by annotation files containing timestamps for linguistic events.
- Formats and access: The pnpl Python library provides task-driven dataset classes that return event-aligned MEG windows and supports selective downloading by subject and corpus.Selectors include deep subject-0 data, broad subjects 1–32, individual subjects, and named corpora.
4 Decoding Experiments
The experiments evaluate word classification with MEG-XL in within-subject and cross-subject settings. Results show benefits from combining deep single-subject data with broad multi-subject data, while performance remains robust with reduced fine-tuning data.
- Word classification maps multichannel MEG recordings to words from a fixed 50-word vocabulary, evaluated using top-10 balanced accuracy.
- MEG-XL is a pretrained backbone using unsupervised pretraining followed by supervised fine-tuning for cross-subject MEG decoding.
- Additional data from 32 subjects generally improves Subject 0 performance on held-out speech distributions, although Sherlock performs marginally better without it.
- 15 percentage points: including Subject 0’s approximately 80 hours improves generalisation across subjects with approximately 40 minutes each.
- 25% training data, equivalent to about 10 minutes of recordings, preserves robust generalisation without significant performance differences across data percentages.
5 Discussion
LibriBrain100 expands the original dataset in depth, breadth, stimulus diversity, and reproducibility infrastructure. Its remaining scope boundaries are passive listening and the absence of a brain-to-text baseline.
- ∼80 hours from one subject and ∼40 minutes from each of 32 additional subjects support both deep within-subject decoding and cross-subject generalisation research.
- The release adds 32 subjects, phonetic and semantic stimuli, serialised HDF5 preprocessing, raw BIDS data, standard splits, and shared evaluation infrastructure.
- All data were collected during passive listening, leaving attempted or imagined speech paradigms outside the current dataset’s scope.
- LibriBrain100 omits a brain-to-text baseline because current open-ended decoding methods were judged insufficiently stable for community benchmarking.
- The open competition, public leaderboard, and Python library are designed to support cumulative, standardised comparison across decoding methods.
Appendices / Supplemental Materials
Figure 5 presents complementary views of LibriBrain100’s composition: recording hours by subject and corpus, and a treemap showing duration across corpora and sessions.
- Figure 5: Dataset proportions: Figure 5 offers complementary perspectives on dataset composition, with Table 1 providing another summary.
- Figure 5: Dataset proportions: Panel (a) shows recording hours by subject and corpus using stacked bars and bubbles whose areas are proportional to hours.Subject 0 is subdivided by linguistic material, while smaller circles represent other subjects.
- Figure 5: Dataset proportions: Panel (b) shows the full dataset as a treemap, with rectangle area proportional to duration and subdivisions for corpora and MEG sessions.A multicolour outline identifies the only two audiobook sessions recorded from multiple subjects.
A.2 Comparison with Existing Datasets
LibriBrain100 combines unusually high per-subject depth with substantially broader coverage than the original LibriBrain release, while incorporating diverse speech materials and reproducible dataset infrastructure.
- Comparison with Existing Datasets: 104 total hours and 0.6–80 hours per subject place LibriBrain100 highest by far in dataset depth, with 33 subjects overall.It has 33 subjects, within the reported range of 1–96 across compared datasets.
- Infrastructure: The pnpl library provides loaders, reproducible splits, and downloading support for the compared datasets to support machine learning at scale.
- Stimulus materials: The dataset spans Sherlock Holmes audiobooks, TIMIT, MOCHA-TIMIT, and The Moth podcast, combining narrative, phonetic, articulatory, and semantic stimulus properties.TIMIT provides controlled phonetic coverage and speaker variation; MOCHA-TIMIT targets connected-speech processes; Moth stories provide naturalistic semantic diversity.
- Audiobooks: The Sherlock Holmes materials use nine public-domain books, with books 1–8 read by David Clarke and book 9 by Thomas A. Copeland.Each MEG session corresponds to a standalone audiobook chapter, with text manually aligned to spoken audio.
- Podcasts: The Moth subset contains 30 stories totaling 6 hours, selected from 77 stories using a criterion intended to maximise semantic coverage.These stories combine coherent autobiographical narratives with spontaneous speech and audience responses.
B.3 Experimental Design & Procedure
The experiments used controlled auditory presentation in the MEG scanner, with different stimulus cohorts and comprehension checks tailored to the available materials and participants.
- Procedure: Auditory stimuli were delivered binaurally through non-metallic air-tube earphones at approximately 70 dB SPL after visually presented instructions.Participants initiated each experiment via button press while seated inside the MEG scanner.
- Multiple-subject cohort: Subjects 1–32 listened to Chapters 11 and 12 of A Study in Scarlet in an approximately one-hour session with a short break between chapters.They answered five four-alternative comprehension questions after each chapter.
- Subject 0: Subject 0 heard the full Sherlock Holmes canon plus TIMIT, MOCHA-TIMIT, and Moth podcasts across multiple recording sessions.Comprehension assessments varied by stimulus type, including chapter-, sentence-, and episode-level questions.
- Comprehension checks: Subject 0’s TIMIT and MOCHA-TIMIT sessions used approximately 20 questions per session, while podcast episodes used five questions probing details, event sequences, and overall meaning.Audiobook chapters used one two-alternative question, whereas the other assessments generally used four alternatives.
B.4 Data Acquisition
Data acquisition combined head-position monitoring, common-reference realignment, detailed linguistic annotations, forced alignment, and corpus-specific reproducible train, validation, and test splits.
- Data Acquisition: Head shape was digitised with fiducial landmarks and approximately 300 scalp, forehead, and nose points, while five HPI coils enabled continuous head-position monitoring.
- Preprocessing: Continuous HPI recordings supported head-movement correction and realignment of all participants to a common reference head position.The procedure was applied consistently across subjects.
- Annotations: Session annotations record onset times and durations for silence, words, phonemes, and phoneme positions within words.
- Forced alignment: Gentle forced alignment was used to align audio and transcripts and generate phoneme-level ARPABET annotations across all linguistic materials.ARPABET provides 39 standardised phoneme categories but does not represent finer allophonic or diphthong detail.
- Quality control: Manual correction addressed forced-alignment failures involving proper names, out-of-vocabulary items, multilingual quotations, atypical prosody, and spontaneous podcast speech.
- Audiobooks: Audiobook splits reserve subject 0’s book 1 sessions 11 and 12 for validation and testing, while subjects 1–32 use separate halves of session 11 and independent session 12 testing.
- TIMIT: TIMIT uses 24 core test speakers and 50 development speakers for testing and validation while excluding repeated sentence IDs and overlapping speakers.
- MOCHA-TIMIT: MOCHA-TIMIT avoids sentence overlap by training on sets A and D, validating on the first half of B, and testing on non-overlapping sentences from C.
C.1 Linguistic Variability
LibriBrain100 broadens linguistic variability across acoustic, phonetic, and semantic dimensions, enabling robustness evaluation across heterogeneous speech inputs.
- Acoustic variability: Speaker embeddings capture pitch, timbre, and vocal-tract characteristics, providing a compact representation of speaker-dependent acoustic variation.Combining corpora expands coverage of pitch- and timbre-related dimensions relevant to auditory-cortex responses.
- Acoustic variability: TIMIT spans a broader speaker-embedding region than the other corpora, partly because it contains many speakers.The combined corpora provide richer sampling of speaker variability for evaluating decoding robustness.
- Phonetic variability: The corpora differ in phoneme distributions and realizations, creating a test of performance across shifts in phonetic statistics.TIMIT is engineered for balanced phoneme coverage, including relatively rare sounds.
- Phonetic variability: Approximately 200,000 phoneme tokens provide broader coverage of acoustic–phonetic patterns across TIMIT, podcasts, and audiobooks.TIMIT increases rare-phoneme occurrences through repeated sentences designed for balanced phoneme coverage.
- Semantic variability: Audiobooks, TIMIT, and podcasts provide distinct lexical-semantic structures, ranging from constrained narratives to broader or less coherent content.These differences add semantic variability beyond acoustic and phonetic variation.
D.1 Neural Variability
Naturalistic speech listening produces neural variability within and across subjects, while preserving shared MEG signatures that decoding models can exploit.
- Shared neural signatures: Speech produces stronger bilateral temporal amplitude, greater low-frequency phase coherence, and enhanced delta–theta power than non-speech.These signatures were assessed using RMS amplitude, ITC phase consistency, and PSD oscillatory power.
- Within-subject variability: Session-level differences in amplitude, phase locking, and low-frequency power vary substantially despite common within-subject speech responses.Some sessions show clean separations, whereas others show attenuated or noisier patterns.
- Between-subject variability: Across participants, shared speech-related responses coexist with considerable individual differences in how strongly the effects are expressed.The preserved group-level pattern includes bilateral auditory-cortex amplitude, low-frequency phase coherence, and delta–theta power.
- Implications for decoding: Variability reflects attention, fatigue, engagement, physiology, and recording conditions rather than being merely noise to eliminate.The dataset uses this variability to test robustness across sessions, linguistic contexts, and individuals.
E.1 Generalisation of Supervised and Pre-trained Models to Deep and Broad Data
Deep single-subject data and shallow multi-subject data favor different modeling strategies: supervised learning excels on deep data, while pretrained self-supervised models generalize better across broad subjects.
- Model generalisation: Supervised word decoding performs better on deep Subject 0 data, whereas fine-tuned MEG-XL generalizes better on shallow Subjects 1–32 data.The comparison uses d’Ascoli’s supervised model and the pretrained self-supervised MEG-XL model; random chance is 0.2.
- Deep-data regime: With sufficient deep data, rich in-domain information can supersede pretrained statistical priors.This pattern explains why supervised modeling performs best in the deep-data regime.
- Broad-data regime: On shallow multi-subject data, pretrained multi-subject priors assist generalization to new subjects.Fine-tuning MEG-XL is therefore advantageous for the broad component.
- Information transfer: OVMI benchmarks compare LibriBrain100 decoding with a prior implanted speech BCI using a 50-word vocabulary.The comparison is intended as a reference for future benchmarks, but heard speech does not represent intended speech communication.
F Additional Decoding Experiment Details
The additional decoding details specify computational cost, vocabulary design, evaluation outputs, and release practices alongside ethical safeguards.
- Compute: Fine-tuning MEG-XL with 100 hours of data took approximately 20 hours per run on an NVIDIA H100 GPU.Experiments used individual H100 GPUs and 64 GiB of CPU memory.
- Vocabulary: The word-classification vocabulary contains 50 words spanning function words, negation, auxiliaries, common verbs, and selected content words.The vocabulary supports short assistive-communication utterances such as reports, requests, and attention cues.
- Evaluation: Full-vocabulary evaluation reports top-10 accuracy across word occurrences and groups predictions by part of speech.The figure also presents a word cloud of the best-predicted words.
- Ethics and release: Participants consented to sharing pseudoanonymised data, with identifying information removed or replaced under ethical oversight procedures.The dataset release applies anonymisation procedures to safeguard participant privacy.
- Ethics and release: Open-source tooling and publicly released data are documented to support reproducibility, with distribution through public platforms and Hugging Face.Long-term availability is supported by redundant hosting and code release.
- Ethics and release: Brain-decoding methods have dual-use privacy risks if applied without consent, although current non-invasive approaches remain far from enabling such uses.The paper stresses informed consent and respect for participant autonomy.