Source-linked AI summary
Cephalonauts One: A deep fMRI dataset for decoding naturalistic speech in the human brain
Antoine Collas, Louis Jalouzot, Géraud Ilinca, Corentin Caris, Romain Valabrègue, Ahmed Hassayoune, David Goncalves, Madeleine Hueber, Thaddée Delebarre, Julien Savatovsky, Clara Fonteneau, Charles Maussion, Bertrand Thirion, Alexis Thual
TL;DR
Deep subject-level speech fMRI and benchmark-ready resources remain limited. Cephalonauts One pairs extensive podcast fMRI with aligned stimulus representations and evaluates held-out audio-segment retrieval; decoding performance improves as within-subject training data increases.
Problem
Method
Cephalonauts One releases paired fMRI and stimulus representations and benchmarks decoding as retrieval of time-aligned stimulus segments from held-out brain activity.
Source: §2 · p. 4, S1.4 Formal definition of the brain… · p. 18Results
Across subjects, decoding improves with additional training sessions; Cosine Similarity increases from approximately 0.075–0.090 to approximately 0.144–0.155.
Source: §4.2 · p. 8Takeaways & Limitations
The dataset supports subject-specific decoding and scaling analyses, with performance improving and no clear saturation over the range studied.
Source: §5.2 · p. 9, §6 · p. 9With only three subjects, the dataset limits conclusions about population-level variability and cross-subject generalization.
Source: §5.2 · p. 9
Abstract
from arXiv · showhide
Cephalonauts One is a whole-brain 3 Tesla (3T) functional magnetic resonance imaging (fMRI) dataset recorded while subjects listened to audio podcasts. Three healthy subjects underwent multiple scanning sessions, each consisting of five 15-minute runs, while listening to podcasts in their native language. With 30 hours of fMRI data per subject, the current release is the deepest available fMRI dataset using naturalistic speech stimuli. The dataset pairs brain activity with the corresponding podcast audio, transcript annotations, and derived stimulus embeddings. Furthermore, we introduce a brain decoding benchmark formulated as audio segment retrieval: given fMRI activity from a held-out session, the decoder must identify the corresponding time-aligned podcast audio segment among candidate segments. We provide standardized splits, evaluation metrics, and baseline decoders for this task. Finally, a scaling analysis shows that decoding performance improves continuously with the amount of training data per subject.
1 Introduction
Brain research links properties of the brain to properties of the external world, while encoding and decoding approaches connect brain activity with stimulus features. Natural speech offers rich, alignable representations for studying language, but speech fMRI remains limited by per-participant data depth.
Brain-behavior links
- Brain research links anatomical, functional, or biological properties to external-world properties such as task ability or stimulus category.Historical lesion studies linked damage in frontal and temporal regions with impaired speech and comprehension [Broca, 1861, Wernicke, 1874].Source: §1 · p. 1
Brain encoding and decoding
- Encoding predicts recorded brain activity from stimulus features, whereas decoding infers stimulus properties from recorded brain activity.Recent language decoding has moved beyond coarse stimulus labels toward richer speech representations.Source: §1 · p. 2
Natural speech
- Spoken narratives engage distributed cortical systems across multiple timescales and align readily with transcripts and computational language representations.fMRI during speech listening can help study biological and computational language representations.Source: §1 · p. 2
Dataset depth
- Speech fMRI resources remain limited in per-participant depth, constraining subject-specific decoders that improve with additional within-subject data.Existing resources range from roughly 2 hours of unique stimuli per participant to up to 16 hours for three participants.Source: §1 · p. 3
2 Related work
Speech fMRI datasets have often emphasized participant breadth over within-subject depth, and ready-to-use benchmarks with deep subject-level data remain lacking. Cephalonauts One addresses this gap with paired stimulus representations and a standardized naturalistic-speech decoding benchmark.
Early scaling of naturalistic fMRI datasets
- Prior visual fMRI datasets scaled to roughly 40 hours per subject in the Natural Scenes Dataset and roughly 80 hours in Courtois NeuroMod.Such scale has supported advances in visual encoding and decoding, including natural-image reconstruction.Source: §2 · p. 3
Speech stimuli in fMRI datasets: breadth and depth
- Speech fMRI datasets have often scaled across participants rather than within them, limiting per-subject depth for subject-specific decoding.Narratives aggregates 345 participants, while Le Petit Prince provides approximately 1.7 hours per subject; LeBel et al. [2023] include up to 16 hours for three participants.Source: §2 · p. 3
Remaining gap
- Existing brain-decoding resources lack fMRI datasets that combine deep subject-level data with aligned representations, held-out sessions, standardized metrics, and reproducible baselines.Standardized tasks, code, and baselines in other releases support model comparison and progress tracking.Source: §2 · p. 3
Contributions
- Cephalonauts One pairs 30 hours per subject of whole-brain 3T fMRI with podcast audio, transcripts, and embeddings, and provides held-out retrieval tasks, metrics, and baselines.Its scaling analysis reports approximately linear improvement on a log–log scale with per-subject data, without saturation over the observed range.Source: §2 · p. 4
3 The Cephalonauts One dataset
Cephalonauts One releases deep per-subject French podcast fMRI alongside aligned audio, transcript, and stimulus-embedding data. Its acquisition, preprocessing, and stimulus pipeline yields multiple brain representations and time-aligned brain–stimulus pairs.
3 The Cephalonauts One dataset
- The release includes preprocessed fMRI in multiple spatial representations, run-level time series, masks, transcripts, audio, and text- and audio-derived embeddings.The dataset and baseline code are hosted on Hugging Face, with access requiring authentication and approval.Source: §3 · p. 4
3.1 Dataset overview
- Three healthy native French-speaking adults listened to podcasts across sessions containing five 15-minute runs, totaling 30 functional hours per subject.Runs used three distinct podcasts per session, and within each subject stimuli were never repeated; audio-onset timing was recorded for alignment.Source: §3.1 · p. 4
3.2 Brain activity recordings
- Functional scans used a 3T protocol with 520 ms repetition time and 3 mm isotropic resolution, then DeepPrep preprocessing produced T1w, MNI, and fsaverage5 representations.These spaces preserve individual anatomy, support cross-subject comparisons, or focus on cortical gray matter while reducing dimensionality, respectively.Source: §3.2 · p. 5
3.3 Stimulus
- Text segments use the preceding 20 completed words and audio segments use the preceding 10 seconds, sampled every four TRs; embeddings represent each modality.Each stimulus event is paired with the brain frame acquired after a fixed 2.5-second hemodynamic lag; text and audio embeddings have 4096 and 3584 dimensions, respectively.Source: §3.3 · p. 6
4 Decoding experiments
The experiments evaluate held-out-session podcast-segment retrieval from fMRI using standardized session splits and complementary retrieval and embedding metrics; performance improves with more subject-specific training data.
4 Decoding experiments
- The benchmark asks a decoder to retrieve the corresponding podcast audio segment from held-out-session fMRI activity, while leaving the decoding approach open.The baseline predicts concatenated audio and transcript embeddings, then ranks candidate segments by cosine similarity.Source: §4 · p. 6, §4.1 · p. 8
4.1 Experimental setup
- Each fold tests on one of sessions 19–23, validates on the next session, and trains on the remaining 22 sessions; test retrieval uses about 2,000 candidates.The held-out candidates come from podcast segments not used in training.Source: §4.1 · pp. 6–7
- Top-10 Retrieval Accuracy, Median Relative Rank, and Cosine Similarity capture complementary aspects of retrieval and predicted-embedding quality.The experiments align stimulus embeddings with preprocessed fMRI volumes using a 2.5-second lag and 10-second stimulus context.Source: §4.1 · p. 7, Figure 2 · p. 7
4.2 Results
- Cosine Similarity rose from approximately 0.075–0.090 to 0.144–0.155, Median Relative Rank fell from 0.11–0.19 to 0.022–0.034, and Top-10 Retrieval Accuracy rose from 3.8–6.4% to 22.3–27.3%.All three metrics improved across subjects, with approximately linear log-log trajectories and no clear saturation over the training sizes studied.Source: §4.2 · p. 8, Figure 3 · p. 8
5 Discussion
Cephalonauts One is presented as both a neuroscience resource and a benchmark for naturalistic language decoding. Its depth and aligned stimuli support speech research, while its scope and evaluation design constrain the conclusions that can be drawn.
5 Discussion
- Cephalonauts One pairs deep speech fMRI with aligned audio and text as a neuroscience resource and naturalistic-language decoding benchmark.The dataset is also intended to support multimodal model development.Source: §5 · p. 8
5.1 Broader impact
- Pairing aligned brain, audio, and transcript data supports speech-comprehension studies and multimodal representation learning.The data enable study across acoustic-to-semantic content; the benchmark addresses modalities with unequal pretrained representation quality and data availability.Source: §5.1 · p. 9
- Brain recordings are sensitive biomedical data, so access is gated and terms prohibit re-identification attempts and contact with study volunteers.Participants gave informed consent for data sharing.Source: §5.1 · p. 9
5.2 Limitations
- Only three subjects are included, limiting conclusions about population-level variability and cross-subject generalization.The depth-over-breadth design supports subject-specific decoding and scaling analyses.Source: §5.2 · p. 9
- Single-site acquisition with one scanner and MRI sequence limits conclusions about robustness across sites, scanners, field strengths, and acquisition settings.Using the same setup improves consistency across sessions and subjects.Source: §5.2 · p. 9
- The 520 ms TR favors temporal sampling over spatial resolution, making the dataset less suited to precise localization of small cortical structures.Functional resolution is 3 mm isotropic.Source: §5.2 · p. 9
- French podcasts from one program constrain the stimulus domain, and models may not generalize to other languages or broader auditory environments.The benchmark retrieves a matching segment from a finite candidate pool in a held-out session, rather than covering the full range of decoding tasks.Source: §5.2 · p. 9
6 Conclusion
Cephalonauts One releases deep, paired fMRI and stimulus data alongside standardized decoding resources. Its scaling analysis reports continued performance improvement with more within-subject training data over the range studied.
6 Conclusion
- Decoding performance improves with additional within-subject data, with no clear saturation over the range studied.The conclusion describes this scaling analysis as supporting deep recordings for naturalistic language decoding.Source: §6 · p. 9
- The release provides 30 hours of whole-brain 3T fMRI per subject for three subjects listening to French podcasts, paired with aligned audio and transcript representations.It also includes preprocessed data in multiple spaces, text and audio embeddings, standardized tasks, baselines, and scaling analyses.Source: §6 · p. 9
S1 A visual glimpse at the data
The supplement illustrates a preprocessed brain volume, provides an interactive notebook and documents acquisition and participant procedures. It also formalizes decoding as mapping brain activity to stimulus features under held-out-session evaluation.
S1 A visual glimpse at the data
- Figure S1 presents one preprocessed brain volume before masking in axial, coronal, and sagittal views in a participant’s native anatomical space.The accompanying text notes that recordings can be noisy and contain reconstruction artifacts, yet retain information useful for decoding.Source: Figure S1 · p. 15, S1 A visual glimpse at the data · p. 15
S1.1 Interactive exploration notebook
- The interactive Colab notebook lets users inspect a validation file’s fields and visualize aligned brain and stimulus data in a browser.It is intended for quick exploration; full preprocessing, training, and evaluation use the released code repository.Source: S1.1 Interactive exploration notebook · p. 15
S1.2 Participant instructions and comprehension questions
- Participants were instructed in French to listen attentively without moving, and answered three multiple-choice comprehension questions after each run.An example question concerned how large language models generate text.Source: S1.2 Participant instructions and… · pp. 15–16
S1.3 MRI scanner parameters
- Functional MRI used a 2D gradient-echo EPI sequence with 48 slices and 1,736 multiband volumes per run; Table S1 summarizes core functional and anatomical sequence parameters.The functional acquisition used simultaneous multi-slice factor 6 and a 66 × 66 matrix.Source: S1.3 MRI scanner parameters · p. 16, Table S1 · p. 16
S1.4 Formal definition of the brain decoding problem
- The decoding setup maps preprocessed, temporally smoothed brain activity to stimulus-feature embeddings and evaluates the decoder on held-out test data.Training, validation, and test sessions are disjoint; temporal lag can account for hemodynamic delay, while combining multiple brain volumes or stimulus segments was left for future work.Source: S1.4 Formal definition of the brain… · pp. 17–18
S2 Preprocessing and decoder training details
The supplementary scaling analysis finds improving retrieval performance with more training data across all released brain spaces and stimulus targets. The appendix also reports the compute used and the baseline decoder’s settings.
Computational resources
- Preprocessing and decoder training used H100-80G GPU workers, with approximately 660 GPU-hours across preprocessing and experiments, including preliminary runs.About 400 GPU-hours used cloud resources and 260 used GENCI–IDRIS HPC; most compute went to DeepPrep preprocessing.Source: S2 Preprocessing and decoder training… · p. 18
Decoder settings
- The baseline settings specify the fMRI input window, hemodynamic lag, stimulus context windows, embedding models, optimization, and retrieval evaluation protocol.The baseline decoder can run on consumer hardware with less than 8 GB of GPU memory and less than 8 GB of system RAM.Source: S2 Preprocessing and decoder training… · p. 18
Scaling analysis
- Median Relative Rank improves with additional training data for all three brain representations and all three stimulus embedding targets.These are fsaverage5, T1w, and MNI representations, and text, audio, and concatenated targets.Source: S2 Preprocessing and decoder training… · p. 18
S3 Licenses and terms of use for existing assets
The appendix lists software dependencies, model and service terms, preprocessing software, and supplementary figure and hyperparameter information.
Code dependencies
- The code dependency list includes scientific Python, machine-learning, neuroimaging, embedding, data, and visualization packages.The listed tools include NumPy, SciPy, PyTorch, transformers, nilearn, and pandas.Source: S3 Licenses and terms of use for… · p. 18
Embedding models and hosted services
- Transcript and audio embeddings use Qwen3-Embedding-8B and LCO-Embedding-Omni-7B, both released under Apache License 2.0.AssemblyAI Universal 3 Pro generated transcripts, and Gemini 3 Pro generated comprehension questions; their respective terms of service govern those uses.Source: S3 Licenses and terms of use for… · p. 18
Decoder hyperparameters
- Table S2 identifies the baseline decoder hyperparameters used across decoding experiments unless otherwise specified.The caption does not provide individual parameter values.Source: Table S2 · p. 19
Preprocessing software
- The preprocessing pipeline is DeepPrep, released under Apache License 2.0; it uses TemplateFlow templates and FreeSurfer components, which require a free license file.FreeSurfer is distributed under its own software license.Source: S3 Licenses and terms of use for… · p. 19
Supplementary scaling figure
- Figure S3 plots Median Relative Rank against subject-specific training hours across released brain spaces and stimulus targets; lower values indicate better retrieval.The spaces are fsaverage5, T1w, and MNI; targets are text, audio, and concatenated embeddings.Source: Figure S3 · p. 20
NeurIPS Paper Checklist
The checklist affirms that the abstract and introduction accurately represent the paper’s contributions and scope. Its guidance emphasizes matching claims to results and making assumptions and generalization boundaries clear.
Claims and scope
- The checklist response says the abstract and introduction accurately reflect the paper’s contributions and scope, including the dataset, paired representations, retrieval benchmark, baselines, and scaling analysis.The justification notes that the Discussion and Limitations describe restrictions on scope and generalization.Source: NeurIPS Paper Checklist · p. 21
Checklist guidance
- The checklist guidance says claims should match the results and indicate important assumptions and limitations that affect expected generalization.Aspirational goals are acceptable when clearly distinguished from contributions achieved.Source: NeurIPS Paper Checklist · p. 21
2. Limitations
The checklist material identifies limits in participant number, acquisition site, language and stimulus domain, and retrieval-based evaluation. It also records information about release, experiments, resources, documentation, and study procedures.
Scope limitations
- The stated scope restrictions are the small number of subjects, single-site acquisition, French podcast stimuli, and retrieval-based evaluation.These are identified as the main limitations described in the Discussion and Limitations sections.Source: §2 · p. 21
Data and code access
- The release is intended to provide preprocessed fMRI data, stimulus representations, benchmark splits, metrics, and baseline code for reproducing main results.The checklist justification says a public release link for the dataset and code appears in the abstract.Source: §2 · p. 23
Experimental details
- The experimental description specifies held-out-session retrieval, candidate segments, metrics, baseline decoder, optimizer, and loss; Appendix S2 gives baseline hyperparameters.The reported evaluation uses five folds, with session indices 19–23 each held out once for testing.Source: §2 · pp. 23–24
Statistical reporting
- Scaling-plot error bars are empirical standard deviations across fold results, capturing held-out-session split variability rather than model initialization.No formal significance test is used, and the bars are not standard errors or confidence intervals.Source: §2 · p. 24
Compute resources
- The appendix reports H100-80G GPU workers for preprocessing and decoder training.The cited checklist justification does not give per-experiment resource estimates.Source: §2 · p. 24
Broader impact and safeguards
- The paper’s broader-impact discussion covers positive uses in neuroscience and multimodal representation learning, potential privacy and misuse risks, and gated dataset access.Access requires accepting terms of use and complying with restrictions on identifying or contacting volunteers, institutional and ethics-review rules, and redistribution conditions.Source: §2 · p. 25
Documentation and human subjects
- The checklist describes documentation of the dataset, released representations, benchmark, metrics, baseline, and hyperparameters, with accompanying access and usage documentation intended for the release.The supplied checklist also notes that the paper reports participant instructions and compensation, ethics approval, consent, and monetary compensation.Source: §2 · pp. 26–27
LLM usage disclosure
- The checklist reports that Gemini 3 Pro generated comprehension questions and that Qwen3-Embedding-8B and LCO-Embedding-Omni-7B generated transcript and audio embeddings.These uses are described in the paper’s Dataset and Stimulus sections.Source: §2 · p. 27