PaperlayerView exact revision on arXiv

Source-linked AI summary

Latent Prediction Yields Cross-Subject Alignable Representations in Sensory-Evoked fMRI

Serwan Jassim, Victoria Bosch, Daniel Anthes, Elia Bruni, Tim Kietzmann

arXiv:2610.40280v1cs.LGcs.AIq-bio.NC

TL;DR

Sensory-evoked fMRI decoding faces limited per-participant data and barriers to pooling across participants. This paper trains brain-only JEPA encoders on repeated responses, then finds their frozen embeddings support competitive generative decoding and can be aligned for cross-participant transfer.

  • Problem

    Per-participant fMRI datasets provide few training examples, while brain differences hinder pooling and functional alignment commonly relies on shared-stimulus responses.

    Source: §1 · pp. 1–2

  • Method

    Participant-specific JEPA encoders predict masked-parcel representations across repetitions of the same stimulus, using brain data without stimulus images or derived representations.

    Source: §1 · p. 2, §3.2 · p. 5

  • Results

    Frozen embeddings support image reconstruction and captioning competitively with end-to-end supervised pipelines, and aligned representations let a participant-trained decoder retain most within-participant quality on another participant.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    Because pretraining and alignment use neither stimulus content nor stimuli shared across subjects, the approach can in principle combine data from participants who saw different stimuli.

    Source: §5 · p. 10

    The evaluation uses repeated presentations in NSD; pooling gains, alignment across datasets, and adaptation to unseen participants remain future tests.

    Source: §3.2 · p. 5, §5 · p. 10

Abstract

from arXiv · show

Decoding sensory-evoked functional magnetic resonance imaging (fMRI) data is a central analytical tool in cognitive neuroscience. Here we ask whether self-supervised learning (SSL), which has been scarcely explored in this domain, can yield representations that support strong downstream performance. To do so, we adapt a joint embedding predictive architecture (JEPA) to train participant-specific encoders on brain data alone. The encoders never see the stimulus images or any representation derived from them. Applied to the Natural Scenes Dataset (NSD), we evaluate our learned brain embeddings on the downstream tasks of brain captioning and image reconstruction, as well as cross-participant generalization. Our self-supervised approach achieves competitive performance with parameter-rich supervised pipelines that are trained end-to-end with access to image and language data. We further find that encoders trained independently for different participants can be aligned fully unsupervised, using neither images nor stimuli shared between them. After alignment, a neural decoding head trained on one participant can be applied to another without fine-tuning and retains most of its within-subject performance on image reconstruction and brain captioning. By removing the need for images and shared stimuli, our approach is a step towards pooling fMRI data across participants and, eventually, across datasets.

1 INTRODUCTION

The introduction motivates learning stimulus-evoked fMRI representations from brain data alone to address data demands and obstacles to cross-participant pooling. It presents a brain-only JEPA whose frozen embeddings support image reconstruction and captioning, and whose independently trained participant representations can be aligned for cross-participant decoding.

1 INTRODUCTION

  • Limited per-participant fMRI examples make pooling attractive, but brain differences impede decoder transfer and existing functional alignment typically requires responses to shared stimuli.The passage notes that fMRI acquisition is time-consuming and expensive.Source: §1 · pp. 1–2
  • Learning from repeated brain responses avoids tying representations to a particular image or language feature space and supports multiple readouts.The objective applies to experiments that repeat stimuli, regardless of stimulus modality.Source: §1 · p. 2
  • The brain-only JEPA predicts masked cortical-parcel latents from a different repetition of the same stimulus, without seeing images or stimulus-derived representations.The objective exploits repeated responses without requiring external image or language representations.Source: §1 · p. 2
  • Frozen brain embeddings support image reconstruction and captioning competitively with corresponding end-to-end supervised pipelines, despite requiring less task-specific parameter learning.The downstream heads use MindEye2 and CorText, respectively.Source: §1 · p. 2
  • Aligned independently trained participant embeddings let a decoder trained on one participant transfer unchanged to another, retaining most within-participant quality.The embeddings remain related by orthogonal maps recoverable without paired data.Source: §1 · pp. 2–3

2 RELATED WORK

Related work spans self-supervised learning in neuroimaging, sensory-evoked image and language decoding, and cross-subject functional alignment. This paper differs by evaluating learned brain representations on generative readouts and removing shared-stimulus and image-derived anchors from pretraining and alignment.

Self-Supervised Representation Learning

  • JEPAs predict latent representations of masked regions from visible context rather than reconstructing raw observations, a design suited to fMRI variation unrelated to stimuli.SSL commonly pretrains representations for use across downstream tasks.Source: §2 · p. 3
  • Neuroimaging SSL often targets resting-state phenotype or clinical readouts; Dong et al. (2024) report JEPA outperforming reconstruction- and contrastive-based SSL on demographic and disease prediction.Marcos-Manch´on et al. (2026) use a largely linear NSD pipeline and evaluate cross-trial retrieval, whereas this work learns end-to-end representations for generative readouts.Source: §2 · p. 3

Decoding from fMRI: vision and language

  • Sensory-evoked decoding maps brain signals to generative image-model or language-model spaces, while MindEye2 and CorText train brain encoders against stimulus-derived targets.Here those pipelines serve as downstream readouts of a frozen encoder pretrained without stimulus images or derived representations.Source: §2 · p. 3

Cross-Subject Functional Alignment

  • Functional alignment has used shared-stimulus responses or image-encoder anchors; this setting removes both, following an alignment framework based on internal similarity structure.Marcos-Manch´on et al. (2026) applied that framework to sensory-evoked fMRI with unpaired participant embeddings.Source: §2 · pp. 3–4

3 METHOD

The method trains participant-specific JEPA encoders on parcellated NSD responses, then evaluates frozen embeddings with image-reconstruction and captioning heads and aligns them across participants without shared stimuli.

3.1 DATASET

  • NSD experiments use four participants’ 9,000 participant-specific training stimuli and hold out 1,000 stimuli shared across participants for evaluation.Responses are parcellated into 500 parcels using the Schaefer atlas.Source: §3.1 · p. 4

3.2 SELF-SUPERVISED BRAIN ENCODER

  • JEPA training predicts masked parcel representations from one repetition using another repetition of the same stimulus as the target.A context encoder uses per-parcel MLPs and transformer layers; an exponential-moving-average target encoder supplies representations for prediction.Source: Figure 1 · p. 5, §3.2 · p. 5

3.3 GENERATIVE NEURAL DECODING

  • Frozen pre-trained embeddings feed generative heads for image reconstruction and brain captioning, without task-specific updates to the encoder.The reconstruction head uses MindEye2, while the captioning head is integrated into CorText.Source: §3.3 · p. 6

3.4 CROSS-SUBJECT ALIGNMENT

  • Cross-participant alignment whitens each participant’s representations and fits a separate orthogonal map for each token, using only training representations.The aligned representations are evaluated by cross-participant retrieval and by applying one participant’s decoding head to another’s mapped representations without retraining.Source: §3.4 · pp. 6–7
  • The evaluation tables cover caption generation, image reconstruction, and within-subject cross-trial retrieval, with the retrieval ablation adding one component per row.The captioning and reconstruction tables report within-subject and cross-subject groups; cross-subject captioning averages over 12 ordered participant pairs.Source: Table 1 · p. 7, Table 2 · p. 7, Table 3 · p. 8

4 RESULTS

Frozen brain embeddings support captioning and image reconstruction near supervised baselines, while cross-trial retrieval and cross-subject alignment reveal how the learned representations preserve stimulus information and transfer across participants.

Within-subject decoding

  • Captioning matches the fully supervised CorText baseline, while reconstruction trails MindEye2 slightly; both use frozen encoders trained without stimulus supervision.The encoders outperform trial-permutation and zeroed-betas controls on every metric except SSIM, which does not distinguish the zeroed-betas control.Source: §4 · p. 8

Within-subject retrieval

  • Visual-cortex-only retrieval raises Recall@1 from 0.39 to 0.63 and reduces mean rank from 31.48 to 16.53 versus the full sequence.Cross-trial pairing and context loss are necessary for stimulus-discriminative representations, with context loss producing the largest gain; reliability weighting adds a small improvement mainly in mean rank.Source: §4 · p. 8

Cross-subject alignment

  • Whitened representations reach cross-subject retrieval Recall@1 of 0.59 and mean rank of 7.45; the complete 500-parcel pipeline reaches 0.38 and 17.45, respectively.The Table 4 ablation averages over 12 ordered participant pairs and uses the 66-parcel visual subset; each successive row adds a component, except the final whitening condition.Source: §4 · p. 8, Table 4 · p. 9
  • Without a learned transformation, cross-subject retrieval is near chance, but orthogonal maps recover correspondence between independently trained participant spaces.The full sequence retrieves less accurately than the visual subset, consistent with the lower stimulus specificity of mostly non-visual parcels.Source: §4 · p. 9

Cross-subject decoding

  • Transferring a decoding head without alignment collapses performance, whereas mapping another participant’s representations into the head’s space substantially recovers decoding.The transferred head is applied without refinement to both captioning and image reconstruction.Source: §4 · p. 9
  • Across participants, the visual subset transfers better for captioning (0.453 vs. 0.394 CIDEr), while the full sequence transfers slightly better for reconstruction (+1.2 CLIP, +1.1 Inception).The authors attribute the difference to attentive pooling in reconstruction versus fixed weights on every token in captioning.Source: §4 · p. 9

5 CONCLUSION

The paper presents latent prediction as a route to stimulus-independent brain representations that support decoding within and across participants. Its conclusion also identifies several settings that remain to be tested.

Conclusion

  • A JEPA encoder trained per NSD participant predicts masked cortical-parcel representations from another repetition, without seeing images or image-derived representations.Frozen encoders support captioning and reconstruction close to end-to-end pipelines trained on text or image targets.Source: §5 · p. 9
  • Independently trained subject encoders can be aligned with orthogonal maps estimated without paired data, allowing an unchanged decoding head to retain most within-subject quality on both tasks.Without alignment, decoding collapses; the results extend prior compact retrieval alignment to representations that drive generative decoders.Source: §5 · p. 9
  • Pooling across subjects who saw different stimuli is possible in principle, but pooling benefits, cross-dataset alignment, use without repeated presentations, and adaptation to unseen subjects remain future questions.Neither pre-training nor alignment uses stimulus content, and alignment requires no stimuli shared between subjects.Source: §5 · p. 10

Disclosure

  • The authors report using generative AI for writing refinement, related-work sourcing, and code debugging and review, with manual review of sourced references.They state that AI was not used for method implementation or interpretation of results.Source: AI USE STATEMENT · p. 10

Ethics

  • The work uses the publicly released NSD, collected with participant consent, and reports no new human data collection or specific ethical concerns.The authors also state that no new data are released.Source: ETHICS STATEMENT · p. 10

A TRAINING PARAMETERS

The appendix identifies shared pre-training settings for all participants and lists the associated optimization-parameter tables for downstream pipelines and alignment.

Pre-training

  • All participants use the same settings across pre-training, captioning, reconstruction, and alignment; pre-training uses paired trials with context and target encoders receiving complementary single trials.The appendix identifies Table 5 as the pre-training hyperparameters for all four participants.Source: A TRAINING PARAMETERS · p. 15, Table 5 · p. 15
  • Pre-training uses AdamW, warmup plus cosine scheduling, an initial learning rate of 5 × 10^-5, batch size 512, and 300 epochs.Weight decay increases from 0.04 to 0.4 and EMA momentum from 0.996 to 1.0 on cosine schedules; warmup lasts 40 epochs.Source: A TRAINING PARAMETERS · p. 15

Downstream and alignment parameters

  • The appendix identifies separate optimization-parameter tables for MindEye2 image reconstruction, CorText brain captioning, and cross-subject alignment.The alignment map is fitted per ordered participant pair from each participant’s own training embeddings, without shared stimuli, paired samples, or labels.Source: Table 6 · p. 15, Table 7 · p. 16, Table 8 · p. 16

B PER-PARTICIPANT RESULTS

The appendix provides per-subject and per-pair result tables for captioning, image reconstruction, and retrieval, including within-subject and cross-subject evaluations.

Captioning

  • Table 9 reports within-subject caption-generation scores per subject, with higher values better and the final block averaging across subjects.The table corresponds to the within-subject rows of Table 1.Source: Table 9 · p. 17
  • Table 10 reports cross-subject caption generation per participant pair, where a head trained on one subject is applied to another’s aligned representations.Higher is better, and the final block averages over the 12 ordered pairs and corresponds to Table 1’s cross-subject rows.Source: Table 10 · p. 18

Image reconstruction and retrieval

  • Table 11 reports within-subject image reconstruction and retrieval per subject; Table 12 reports both tasks per participant pair across subjects.The final blocks average across subjects in Table 11 and across 12 ordered pairs in Table 12; Table 11 omits undefined brain retrieval for the zeroed-betas control.Source: Table 11 · p. 19, Table 12 · p. 20

C IMAGE RECONSTRUCTIONS

The section presents within-subject and cross-subject image reconstructions from brain activity. Reconstructions use the refined MindEye2 pipeline, while Table 2 reports metrics on unrefined outputs.

C IMAGE RECONSTRUCTIONS

  • Within-subject reconstructions use the refined MindEye2 pipeline, while Table 2 reports metrics on unrefined outputs.Stimuli were randomly drawn from the 1,000-image test set with a fixed seed.Source: Figure 2 · p. 21
  • Cross-subject reconstructions use the refined MindEye2 pipeline; the w/o alignment control transfers the head without the alignment transformation.The stimuli are the same as those shown for within-subject reconstruction.Source: Figure 3 · p. 22
Loading 2610.40280v1…