PaperlayerView exact revision on arXiv

Source-linked AI summary

Decoding Naturalistic Speech from fMRI with Temporal Modeling and Multimodal Alignment

Corentin Caris, Louis Jalouzot, Nona Rajabi, Géraud Ilinca, Ahmed Hassayoune, Asilbek Muminov, Thaddée Delebarre, Clara Fonteneau, Charles Maussion, Bertrand Thirion, Alexis Thual, Antoine Collas

arXiv:2610.22049v1cs.LGcs.AI

TL;DR

Naturalistic speech unfolds over time, but fMRI’s temporally smeared measurements make its structure difficult to recover. NOEMA maps consecutive fMRI measurements to multimodal speech embeddings, and outperforms baselines on held-out sessions and stimuli across two listening datasets, with its advantage persisting under added distractors.

  • Problem

    Naturalistic speech has temporal structure, but fMRI’s limited temporal precision makes that structure difficult to recover.

    Source: §1 · pp. 1–2

  • Method

    NOEMA uses a temporal convolutional decoder to map fMRI windows to corresponding multimodal speech-segment embeddings.

    Source: §1 · p. 2, §3.1 · p. 3, §3.2 · p. 4

  • Results

    NOEMA outperforms all baselines across two listening datasets on held-out sessions and unseen stimuli, and retains its advantage with external distractors.

    Source: §6 · p. 9

  • Takeaways & Limitations

    Ablations show that temporal context, frame order, and complementary stimulus representations contribute to retrieval performance.

    Source: §2 · p. 3, §6 · p. 9

    Evaluation covers six participants across two datasets; generalization to new individuals and other tasks remains untested.

    Source: §6 · p. 9

Abstract

from arXiv · show

Reconstructing what a person perceives from their brain activity is a stringent test of whether neural recordings carry meaningful information. Naturalistic speech offers a rich testbed for this challenge: it unfolds over multiple timescales, but fMRI's noisy, temporally smeared measurements make this structure hard to recover. We introduce NOEMA, a temporal convolutional decoder that aligns windows of consecutive fMRI measurements with multimodal stimulus embeddings. These embeddings combine complementary text, audio, and speech representations of the same stimulus segment, enabling its retrieval directly from brain activity. On two naturalistic speech datasets, Cephalonauts One and LeBel, NOEMA retrieves the correct segment among 2,000 candidates within the top 10 for more than 42% of held-out queries. This substantially outperforms baseline decoders, an advantage that persists when the candidate pool is extended with thousands of distractors absent from training. Controlled ablations establish the contributions of temporal modeling and complementary stimulus representations.

1 INTRODUCTION

Naturalistic speech provides temporal structure for studying brain decoding, but fMRI’s limited temporal precision makes that structure difficult to exploit. NOEMA maps consecutive fMRI measurements to multimodal speech representations and outperforms existing baselines on two listening datasets.

1 INTRODUCTION

  • Decoding cognitive states from brain activity is linked to representation learning and can provide insight into the structures present in neural signals.Source: §1 · p. 1
  • Naturalistic speech unfolds over time, giving decoders temporal structure to exploit, unlike static images.Source: §1 · pp. 1–2
  • NOEMA maps windows of consecutive fMRI measurements to multimodal representations of speech segments, combining spatial and temporal information for decoding.The approach is evaluated on two naturalistic speech datasets and outperforms existing baselines on held-out sessions and extended retrieval sets.Source: §1 · p. 2
  • fMRI offers spatial resolution but limited temporal precision, whereas M/EEG offers finer temporal dynamics, suggesting complementary routes to speech decoding.Source: §1 · p. 2

2 RELATED WORK

Prior work spans natural-language decoding, temporal integration of fMRI, and cross-modal alignment. NOEMA combines these lines by applying learned temporal convolutions to multimodal speech representations and evaluates their contribution to retrieval.

2 RELATED WORK

  • Prior natural-language fMRI work decoded concepts, vocabulary words, semantic features, or continuous text, while BrainLLM feeds fMRI-derived embeddings into a language model.Source: §2 · p. 2
  • Earlier fMRI methods integrate temporal context through averaging, sequence pretraining, language-model prompts, or spatiotemporal encoding; their generation results are not directly comparable to segment retrieval.NOEMA compares learned temporal convolutions with frame-wise decoders and an MLP receiving the same context.Source: §2 · p. 2
  • Cross-modal alignment has supported retrieval and neural decoding across image, text, audio, and brain-signal representations.Prior work includes CLIP, MEG/EEG alignment with heard audio, and fMRI alignment with image representations.Source: §2 · p. 2
  • Distinct cortical contributions of acoustic, articulatory, and semantic features motivate evaluating complementary targets alongside temporal decoding.Source: §2 · p. 3
  • NOEMA combines temporal convolutional decoding with multimodal speech targets, outperforming evaluated frame-wise baselines and a same-context MLP on two datasets.The reported contributions also include benefits from temporal context, frame order, added distractors, and complementary text, audio, and speech representations.Source: §2 · p. 3

3 METHOD

NOEMA frames naturalistic speech decoding as retrieval: a participant-specific decoder maps fMRI windows to embeddings of corresponding speech segments. It uses temporal convolutions and fixed text, audio, and speech encoders to form the targets.

3.1 PROBLEM FORMULATION AND RETRIEVAL OBJECTIVE

  • Given an fMRI window, a participant-specific decoder predicts the corresponding segment embedding, which is ranked against held-out candidates by cosine similarity.At inference, the decoder receives only fMRI; candidate audio and transcripts are used only to precompute fixed embeddings.Source: §3.1 · p. 3
  • The decoder is trained to align its predictions with multimodal stimulus targets using a batch-based CLIP contrastive loss.The loss scores matched brain–stimulus pairs, and its temperature is optimized per model.Source: §3.1 · pp. 3–4

3.2 NOEMA ARCHITECTURE

  • Two residual temporal convolutional blocks integrate consecutive frames, giving the central representation a seven-frame receptive field spanning earlier and later measurements.The convolutions use kernel size 3 and dilations 1 and 2; the final linear layer projects only the central representation into the multimodal target space.Source: §3.2 · p. 4

3.3 MULTIMODAL STIMULUS REPRESENTATIONS

  • Three frozen encoders produce text, audio, and speech targets; the shorter speech window complements the longer text and audio contexts.The encoder parameters remain fixed while the participant-specific fMRI decoder is trained.Source: §3.3 · p. 4

4 EXPERIMENTAL SETUP

The experiments use session-held-out evaluation on two listening datasets, with separate participant-specific decoders. Models are compared using retrieval accuracy and rank metrics, with matched targets and retrieval settings.

4.1 DATASETS, PREPROCESSING, AND SPLITS

  • The study evaluates three participants in Cephalonauts One and three in LeBel, training a separate decoder for each participant and dataset.Cephalonauts One comprises French podcast listening; the selected LeBel participants completed additional listening hours for sufficient training data.Source: §4.1 · pp. 4–5
  • Session-based splits hold out test and validation sessions, preventing stimulus overlap across training, validation, and test sets.Each model is evaluated on five splits, each with one session for testing and one for validation.Source: §4.1 · p. 5

4.2 BASELINES, TRAINING, AND EVALUATION

  • Baselines include Ridge, two frame-wise MLPs, and a Context MLP; all predict the same concatenated target embedding using identical splits.Unless stated otherwise, NOEMA and Context MLP receive seven-frame windows, while the other baselines receive only the central frame.Source: §4.2 · p. 5
  • Retrieval ranks candidate embeddings by cosine similarity, with Top-10 accuracy as the primary metric and median rank measures for retrieval beyond a fixed cutoff.For the extension experiment, Cephalonauts One test sets are augmented with up to 8,494 embeddings from 50 additional podcast episodes.Source: §4.2 · pp. 5–6
  • The reported Figure 3 comparison summarizes Top-10@2000 accuracy across decoder configurations and subjects.Subject columns show five-fold means and standard deviations; the mean column gives a confidence interval across three subject means.Source: Figure 3 · p. 6

5 RESULTS

NOEMA outperforms the evaluated baselines on naturalistic speech retrieval, and its performance benefits from multimodal targets and temporal context. Its advantage persists with unseen-episode distractors, while feature-importance maps describe model sensitivity rather than causal brain regions.

5.1 RETRIEVAL PERFORMANCE

  • 42.7% of Cephalonauts One queries and 43.6% of LeBel queries retrieve the correct segment within the top 10, exceeding all baselines and 0.5% chance.NOEMA also surpasses the same-input Context MLP despite that baseline having more parameters.Source: §5.1 · p. 6

5.2 STIMULUS REPRESENTATIONS AND TEMPORAL CONTEXT

  • On Cephalonauts One, adding audio raises NOEMA’s Top-10@2000 accuracy from 27.6% to 35.5%, and adding speech raises it to 42.7%.Both additions improve retrieval for all three models in every subject and fold; NOEMA’s mean gains exceed Context MLP’s by 1.58 and 2.34 percentage points, respectively.Source: §5.2 · p. 7, Figure 4 · p. 7
  • On Cephalonauts One, NOEMA’s Top-10@2000 accuracy rises from 28.7% with one frame to 42.7% with seven, while Context MLP peaks at 34.5% with five frames.NOEMA has 253.8M trainable parameters versus Context MLP’s 623.0M; shuffling frames reduces NOEMA accuracy while leaving Context MLP essentially unchanged.Source: §5.2 · p. 7, Figure 4 · p. 7

5.3 EXTENDED RETRIEVAL SETS

  • With up to 8,494 unseen-episode segments added, NOEMA’s median rank increases by 66 positions, versus 112 for Context MLP and 191 for MLP.All models’ ranks grow by 4.5–4.6× for a 5.1× larger set, and NOEMA maintains the lowest median relative rank throughout.Source: §5.3 · p. 8, Figure 5 · p. 8

5.4 MODEL INTERPRETATION

  • Across the models, feature importance is higher in lateral temporal cortex, with smaller frontal patches and no distinct high-importance regions on medial views.Among seven-frame models, only NOEMA shows a central-frame bias, consistent with its temporal convolutions and central-frame readout.Source: §5.4 · p. 8, Figure 6 · p. 9
  • The attribution maps do not establish a causal relationship between the highlighted regions and decoding, and they show no clear left-right asymmetry.The authors note that larger-scale language models might be necessary for the asymmetry to re-emerge in this setup.Source: §5.4 · p. 9

6 CONCLUSION

NOEMA combines temporal fMRI modeling with alignment to text, audio, and speech representations, outperforming baselines across two naturalistic listening datasets and retaining its advantage with external distractors. Ablations support contributions from temporal context and target representations, while the evaluation leaves several scope and mechanism questions unresolved.

Conclusion

  • NOEMA outperforms baselines across two naturalistic listening datasets and retains its advantage when retrieval sets include external distractors.The evaluation uses held-out sessions and speech stimuli absent from training.Source: §6 · p. 9
  • Ablations indicate that temporal context and target-representation choice both contribute to retrieval performance.NOEMA benefits more from additional fMRI frames than an MLP with the same context despite having fewer parameters, and its performance depends on chronological frame order.Source: §6 · p. 9
  • Evaluation covers six participants across two datasets with separate participant-specific decoders; transfer to new people or tasks remains untested.The temporal-context benefit’s underlying source and the optimal context length are also unresolved.Source: §6 · p. 9
  • NOEMA retrieves segments from a predefined candidate set, so finite-set retrieval does not establish unrestricted speech reconstruction.The authors identify using predicted embeddings to guide text generation as a future direction.Source: §6 · p. 9

A DATASET SIZE

The table reports participant-level counts of sessions, runs, and acquisition hours across training, validation, and test sessions.

Dataset size

  • Counts cover training, validation, and test sessions, with each run counted once.Hours are computed from native fMRI frames multiplied by TR, before temporal downsampling and window selection.Source: Table 1 · p. 15

B.1 MODEL ARCHITECTURE

NOEMA uses a temporal convolutional architecture to map fMRI windows to stimulus embeddings, with training hyperparameters selected through validation-based search. The reported experiments examine alignment lag and ASR context-window sensitivity, then compare retrieval accuracy and rank against baselines.

B.1 MODEL ARCHITECTURE

  • The decoder applies a spatial projection per frame, two TCN blocks, and a linear projection of the central frame; the TCN blocks provide a seven-frame receptive field.Table 2 specifies N = 7 frames, V = 20,484 cortical vertices, H = 4096, and D = 8704.Source: Table 2 · p. 15, §B.1 · p. 16, Figure 7 · p. 16

B.2 HYPERPARAMETER OPTIMIZATION

  • Bayesian hyperparameter search uses five session-based training/validation splits on Cephalonauts One sub-01, excluding test sessions entirely.The selected configurations are reported with validation median relative rank averaged over five folds.Source: §B.2 · p. 17, Table 3 · p. 17, Table 4 · p. 17

C SENSITIVITY TO LAG

  • Across tested alignment offsets, NOEMA is more accurate than the MLP and remains within 10% of its default Top-10@2000 accuracy from −1.25 to 4 seconds.NOEMA peaks at 47.75% at δ = 1 second, compared with 46.71% at the default δ = 2.5 seconds; the test-session curves are from one participant and were not used to select δ.Source: C SENSITIVITY TO LAG · p. 18, Figure 8 · p. 18

D ASR CONTEXT WINDOW

  • NOEMA’s Top-10@2000 accuracy peaks with a 5-second Parakeet window on Cephalonauts One (48.19%) and a 3-second window on LeBel (39.84%).The default window equals marker spacing to give each target a distinct, non-overlapping speech segment; the test varies window length for one subject per dataset.Source: D ASR CONTEXT WINDOW · p. 19, Figure 9 · p. 19

E.1 TOP-1 ACCURACY AND MEDIAN RELATIVE RANK

  • NOEMA ranks the correct segment first for 12.0% of queries on Cephalonauts One and 13.6% on LeBel, versus 7.7% and 10.0% for the seven-frame MLP.Mean median relative rank falls from 0.0146 to 0.0087 on Cephalonauts One and from 0.0145 to 0.0093 on LeBel.Source: §E.1 · p. 20, Figure 10 · p. 20

E.2 RANK DISTRIBUTIONS

Across nearly the entire retrieval-rank range, NOEMA’s cumulative retrieval curve is at or above every baseline for every subject. Its rank density concentrates at low ranks, while baseline density peaks farther down the rank range.

E.2 RANK DISTRIBUTIONS

  • NOEMA places half of queries within rank 16 on both datasets, compared with ranks 28 and 25 for the seven-frame MLP on Cephalonauts One and LeBel.It places 90% of queries within rank 328 on Cephalonauts One and 263 on LeBel, compared with 445 and 390 for the MLP.Source: §E.2 · p. 21, Figure 11 · p. 21
  • For every subject, NOEMA’s cumulative rank curve lies on or above every baseline across nearly the entire range, with small crossings near 100%.The cumulative distribution reports the percentage of test queries whose correct segment ranks at or above each position.Source: §E.2 · p. 21, Figure 11 · p. 21
  • NOEMA’s rank density is highest at the lowest ranks and declines beyond rank 100, whereas MLPs peak around rank 100 and Ridge concentrates near the range’s top.The density is binned on a log scale and normalized per decade of rank.Source: §E.2 · p. 22, Figure 12 · p. 22

F.1 RESULTS ON LeBel DATASET

NOEMA has the highest mean accuracy across target combinations on both datasets, with the full text, audio, and speech target most accurate for every model. On LeBel, adding modalities and temporal context improves retrieval, and NOEMA’s accuracy advantage over Context MLP grows with longer windows without relying on a larger parameter count.

F.1 RESULTS ON LeBel DATASET

  • On LeBel, adding audio raises NOEMA’s Top-10@2000 accuracy by 8.52 percentage points and adding speech by 12.06 points, exceeding Context MLP’s gains by 2.01 and 3.90 points, respectively.The speech addition has a larger effect on LeBel than on Cephalonauts One, where it adds 7.25 points.Source: §F.1 · p. 23, Figure 13 · p. 23

F.2 ABSOLUTE RETRIEVAL ACCURACIES ACROSS STIMULUS REPRESENTATIONS

  • NOEMA has the highest mean accuracy for every target combination on both datasets, and every model is most accurate with the full T+A+S target.Figure 14 reports the absolute accuracies underlying the representation gains.Source: §F.2 · p. 23, Figure 14 · p. 24

F.3 ABSOLUTE RETRIEVAL ACCURACIES WITH INCREASING TEMPORAL CONTEXT

  • As windows grow from one to seven frames, NOEMA’s mean-accuracy lead over Context MLP widens from 3.3 to 9.1 percentage points on Cephalonauts One and from 1.8 to 7.6 points on LeBel.Figure 15 shows the absolute accuracies; its wide LeBel bands reflect differences between subjects, while within-subject gains provide the more direct comparison.Source: §F.3 · p. 24, Figure 15 · p. 24

F.4 RETRIEVAL ACCURACY AND MODEL SIZE

  • With 253.8M parameters, NOEMA’s seven-frame configuration achieves higher mean accuracy than all four Context MLP configurations on both datasets, while Context MLP grows from 119.6M to 623.0M parameters.The comparison indicates that NOEMA’s advantage does not come from a larger parameter count.Source: §F.4 · p. 25, Figure 16 · p. 25

G SENSITIVITY TO FMRI FRAME ORDER

Frame-order shuffling distinguishes models that can absorb a fixed permutation from those that benefit from chronological temporal neighborhoods. Both models lose accuracy when each sample receives a different permutation, with a larger observed dependence for NOEMA.

G SENSITIVITY TO FMRI FRAME ORDER

  • With a shared frame permutation, Context MLP is nearly unchanged (+0.33 percentage points), while NOEMA loses 4.26 points despite retraining.The experiment applies the same shuffling rule during training and evaluation, measuring the benefit of original temporal organization after adaptation rather than test-time robustness.Source: G SENSITIVITY TO FMRI FRAME ORDER · p. 26, Figure 17 · p. 26
  • With per-sample permutations, accuracy falls by 10.90 points for Context MLP and 18.46 points for NOEMA, indicating a larger observed dependence on consistent temporal positions for NOEMA.Each sample receives its own permutation, breaking temporal ordering across samples.Source: G SENSITIVITY TO FMRI FRAME ORDER · p. 26, Figure 17 · p. 26

H TRAINING-DATA SCALING

The appendix describes the training-session budgets used to compare Context MLP and NOEMA, the construction of extended retrieval sets, and the feature-attribution procedure and maps. The supplied passages specify these analyses but do not report their performance outcomes.

H TRAINING-DATA SCALING

  • Training-data scaling compares seven-frame Context MLP and NOEMA across nested whole-session subsets, with validation and test sessions fixed within each fold.Budgets are 1, 2, 3, 6, 12, and 22 sessions for Cephalonauts One and 1, 2, 3, 5, 8, and 13 for LeBel; Figure 18 plots Top-1@2000, Top-10@2000, and median relative rank against training duration.Source: H TRAINING-DATA SCALING · p. 27, Figure 18 · p. 27

I EXTENDED RETRIEVAL SETS

  • The extended retrieval sets add up to 8,494 candidates from 50 additional podcast episodes absent from Cephalonauts One, evaluating models without retraining.For each test session, random subsets of 500, 1,000, 2,000, or 5,000 candidates, or all 8,494, are added to the original set of approximately 2,000 segments.Source: I EXTENDED RETRIEVAL SETS · p. 28

J FEATURE-IMPORTANCE MAPS

  • Feature-importance analysis uses 50-step integrated gradients from a zero baseline to attribute each model’s CLIP loss to input vertices and frames.Attributions are averaged over held-out samples; Figures 19–21 show NOEMA, seven-frame MLP, and one-frame MLP maps, respectively.Source: J FEATURE-IMPORTANCE MAPS · p. 28, Figure 19 · p. 28, Figure 20 · p. 29

Related papers

Loading 2610.22049v1…