PaperlayerView exact revision on arXiv

Source-linked AI summary

Many Brains, One Geometry: A Shared Visual-Semantic Space for Cross-Dataset fMRI Decoding

Moein Khajehnejad, Michelangelo Tronti, Forough Habibollahi, Tommaso Boccato, Matteo Ferrante, Nicola Toschi

arXiv:2610.09352v1cs.HCq-bio.NCq-bio.QM

TL;DR

Visual fMRI decoders are often confined to the participants and experiments used for training, leaving uncertain whether heterogeneous measurements can share a transferable geometry. BRAID-fMRI jointly aligns regional brain activity with CLIP using a shared decoder, and reports retrieval across datasets, matched-participant baseline comparisons, and target-dataset holdout gains without participant conditioning.

Abstract

from arXiv · show

Visual decoding from fMRI is typically siloed by participant and experiment, obscuring whether heterogeneous neural measurements can be organized within a common computational geometry. Here we introduce BRAID-fMRI (Brain Representation Alignment across Individuals and Datasets), a shared CLIP-supervised decoding framework. BRAID-fMRI uses a single ROI-wise Transformer with optional participant conditioning across eight visual-fMRI datasets comprising 93 dataset-specific participant entries, 430,007 single-trial responses and 162,839 unique stimuli. Regional brain activity is aligned with 512-dimensional CLIP ViT-B/32 representations using a multi-positive contrastive objective that treats repeated stimuli across participants and datasets as positives. BRAID-fMRI supports retrieval across seven evaluation datasets. On eight matched participant entries, it achieves 35.0 +/- 11.1% Top-10 accuracy, exceeding the observed mean accuracy of the two evaluated baselines - the MindEye-style pooled-CLIP decoder (27.1 +/- 5.1%) and ridge regression (21.1 +/- 9.1%) - and attaining the highest observed accuracy for seven of eight entries. In separately trained participant-agnostic models, expanding the source pool increased target-dataset holdout accuracy by up to 92.1% relative to the initial source-training condition. The learned space preserves graded semantic structure, while ablations and saliency highlight ventral and early visual cortex and category-specific motion and attentional systems. These results support scalable cross-dataset decoding into a common CLIP-aligned space, with model sensitivity concentrated in ventral and early visual inputs.

Introduction

The introduction asks whether heterogeneous fMRI measurements can share a visual geometry that transfers across participants and experiments. BRAID-fMRI tests this with a joint CLIP-supervised decoder and reports retrieval and transfer across datasets.

Introduction

  • The unresolved question is whether neural measurements across independently acquired datasets retain a common relational geometry that supports direct transfer to unseen participants and experiments.Prior shared-geometry findings were established under coordinated experimental conditions, while cross-dataset learning adds scanner, resolution, design, stimulus, and data-volume differences.Source: Introduction · pp. 1–2
  • MOSAIC harmonizes eight visual-fMRI datasets, with 93 dataset-specific participant entries, 430,007 single-trial pairs, and 162,839 unique stimuli.The resource uses uniform preprocessing and curated splits that prevent duplicate or perceptually similar train–test stimuli.Source: Introduction · p. 2
  • A shared ROI-wise Transformer aligns regional activity to frozen CLIP embeddings, with optional participant embeddings and a multi-positive objective for repeated stimuli.Participant conditioning can be disabled to evaluate transfer without fitted participant embeddings.Source: Introduction · p. 2
  • One model retrieved stimuli across seven evaluation datasets, performed best for seven of eight matched participants against Ridge and a MindEye-style baseline, and participant-agnostic source expansion yielded up to 92.1% relative target-holdout gains.The source-pool comparison jointly changes dataset diversity and training-data amount.Source: Introduction · p. 2
  • The analyses characterize graded semantic structure and identify common early and ventral visual dependence alongside category-dependent motion and attentional contributions.The contribution combines matched-participant retrieval, target-dataset holdout transfer without fitted participant embeddings, and analyses of semantic geometry and anatomical sensitivity.Source: Introduction · pp. 2–3

Results

BRAID-fMRI aligns ROI-wise brain responses with frozen CLIP embeddings using a Transformer shared across datasets and participants, optional participant conditioning, and a multi-positive contrastive objective. Its results span cross-dataset retrieval, participant-agnostic transfer, semantic organization, and regional contributions to decoding.

One model retrieves visual stimuli across datasets and participants

  • 35.0% mean Top-10 accuracy exceeded MindEye's 27.1% and Ridge regression's 21.1%; BRAID-fMRI led seven of eight matched participants.With all validation stimuli across seven evaluation datasets as candidates, accuracy exceeded global random-ranking chance; restricting candidates to the query participant’s dataset generally increased accuracy.Source: One model retrieves visual stimuli… · p. 3, Table 1 · p. 6

The participant-agnostic encoder supports transfer in target-dataset holdout evaluations

  • 92.1% was the largest first-to-final relative gain as the participant-agnostic model improved across all seven held-out target datasets.The curriculum's individual steps were not uniformly beneficial, indicating that source datasets differed in their relationship to each target.Source: The participant-agnostic encoder… · p. 6, Figure 3 · p. 7

Retrieval performance follows visual-semantic organization

  • People and portraits ranked highest in Top-10 retrieval, while home appliances and some architectural categories were among the most difficult.The ten categories used for subsequent analyses were selected by ranking all 40 clusters by retrieval accuracy and retaining every fourth cluster.Source: Retrieval performance follows… · p. 6

A shared visual backbone supports retrieval across datasets

  • Masking ventral visual/object regions reduced pooled Top-10 accuracy by 4.8 percentage points, and masking early visual cortex reduced it by 3.5 points.Ventral and early visual ablations were especially disruptive for BMD and NSD, while contributions from other systems varied more across datasets.Source: A shared visual backbone supports… · pp. 7–8, Figure 3 · p. 7

The learned space preserves graded semantics and category-specific cortical structure

  • Matched brain–CLIP similarity exceeded same-cluster non-match similarity by 0.094, while same-cluster similarity exceeded different-cluster similarity by 0.043.Both margins were visible across the seven evaluation datasets and representative semantic clusters, though their magnitudes varied by dataset and category.Source: The learned space preserves graded… · p. 8, Figure 4 · p. 9
  • Early and ventral visual systems were enriched across several clusters, while tennis retrieval was impaired by ventral, parietal-attention, and early-visual ablations.People and portraits were most impaired by ventral visual/object ablation (−8.0 percentage points); category and dataset composition were not disentangled.Source: The learned space preserves graded… · p. 8

Shared semantic geometry persists across datasets and retrieval difficulty

  • Same-stimulus, same-cluster, and cross-cluster brain embeddings formed a graded hierarchy across datasets; five evaluable dataset pairs had positive Spearman correlations of 0.61–0.88.BMD, GOD, and Deeprecon lacked sufficient shared clusters, and the observed correspondence did not separate target-imposed structure from neural support.Source: Shared semantic geometry persists… · pp. 8–9
  • Retrieval stayed above chance with larger pools and all same-cluster distractors; brain–CLIP nearest-neighbor overlap exceeded random expectations but remained modest.Together, these findings indicate partial preservation of local CLIP organization alongside informative retrieval under large and semantically challenging candidate pools.Source: Shared semantic geometry persists… · pp. 9–10

Discussion

Joint modeling of eight visual-fMRI datasets organizes heterogeneous measurements in a common visual-semantic representation. The authors argue that independently acquired experiments can contribute transferable structure to computational models of brain activity.

Discussion

  • One jointly trained model retrieved held-out stimuli across seven evaluation datasets and outperformed matched participant-specific baselines for seven of eight participants.Participant-agnostic models also supported target-dataset holdout retrieval, with final source-pool accuracy improving by up to 92.1% relative to the initial condition.Source: Discussion · p. 10
  • With participant embeddings disabled, the final, more diverse source pool improved retrieval for all seven unseen target datasets, although individual curriculum steps were not uniformly beneficial.The source pool expanded across participants, trials, stimuli, and datasets, so the analyses do not isolate which forms of diversity contributed most.Source: Discussion · p. 10
  • Brain embeddings were most similar for the same stimulus, intermediate for different stimuli in the same semantic cluster, and least similar across clusters, including across datasets.Semantic representational geometries also corresponded across five evaluable dataset pairs, with Spearman correlations from 0.61 to 0.88; BMD, GOD, and Deeprecon lacked sufficient shared clusters.Source: Discussion · pp. 10–11, Figure 5 · p. 11
  • Retrieval remained above chance as candidate pools expanded and same-cluster distractors increased, while brain-space neighborhoods only partially overlapped CLIP neighborhoods.The partial overlap leaves room for prediction error, regularization, measurement noise, and distinct representational structure.Source: Discussion · pp. 11–12, Figure 5 · p. 11
  • Early and ventral visual systems formed the most consistent backbone, while motion and parietal-attention systems contributed more selectively across stimulus categories.Saliency and ablation convergence indicates category-associated input dependence in the fitted mapping, but does not independently establish biological category selectivity.Source: Discussion · p. 12
  • The conclusions are bounded by CLIP-supervised geometry, imbalanced source datasets, and attribution analyses that do not establish causal necessity of the implicated brain systems.The authors also identify unquantified few-shot participant adaptation and call for comparisons with behavior-derived and alternative visual representations.Source: Discussion · p. 12
  • The authors propose that jointly modeling anatomical correspondence, cross-participant structure, and stimulus semantics can make independently acquired experiments computationally compatible.They suggest that future work could extend the framework to few-shot adaptation and other sensory or cognitive domains.Source: Discussion · p. 12

Methods

MOSAIC harmonizes heterogeneous visual-fMRI studies, and BRAID-fMRI maps anatomically corresponding ROI inputs into a shared encoder. Participant conditioning is optional, allowing the model to operate on identities not represented during training.

MOSAIC dataset and harmonization

  • MOSAIC combines eight independently acquired datasets into 93 dataset-specific participant entries, 430,007 single-trial responses, and 162,839 unique stimuli.Stimulus-level split reconciliation keeps repeated observations together, and DreamSim filtering excludes near-duplicates across the train–test boundary; common preprocessing and training-derived normalization harmonize measurements.Source: MOSAIC dataset and harmonization · pp. 12–13

ROI representation and model input

  • ROI beta vectors vary in raw dimension, and per-trial within-ROI normalization preserves spatial patterns while removing regional mean and scale information.Non-finite values are replaced with zero, and normalized values are clipped to [−5, 5].Source: ROI representation and model input · p. 13

Shared brain encoder with optional participant conditioning

  • ROI-specific tokenizers map variable-sized inputs into a common token space before a Transformer shared across all participant entries and datasets.Learned participant embeddings provide a lightweight participant-dependent shift while the remaining model components stay shared; disabling these embeddings permits evaluation on unseen identities.Source: Shared brain encoder with optional… · pp. 13–14

Attention pooling and alignment with CLIP

The model pools contextualized ROI representations into one brain vector and projects it into a frozen CLIP image-embedding space. For video datasets, the target represents a central frame rather than temporal dynamics.

Attention pooling and alignment with CLIP

  • Learned attention pooling combines contextualized ROI representations into a trial-level brain vector, which a projection head maps into visual-semantic embedding space.The projection head uses layer normalization, two 512-dimensional linear transformations, and GELU.Source: Attention pooling and alignment with… · pp. 14–15
  • Static-image targets are 512-dimensional embeddings from frozen CLIP ViT-B/32, while video targets use the clip's central frame.This video target evaluates correspondence to a representative frame and does not separately establish decoding of temporal dynamics or actions; gradients update the brain encoder, pooling module, and projection head, not CLIP.Source: Attention pooling and alignment with… · p. 15

Multi-positive contrastive learning

The contrastive objective treats repeated observations of the same stimulus as positives rather than false negatives. It uses a symmetric brain-to-CLIP and CLIP-to-brain training objective across the distributed batch.

Multi-positive contrastive learning

  • A uniform target over all same-stimulus observations in the distributed batch prevents repeated presentations from being treated as negatives.Repeated observations can occur within or across participants and datasets.Source: Multi-positive contrastive learning · p. 15
  • The objective symmetrizes brain-to-CLIP and CLIP-to-brain cross-entropy using normalized embeddings and a learned, capped logit scale.Embeddings and stimulus identifiers are gathered across GPUs before computing the loss, which has weight 0.5; the masked-autoencoding branch is disabled in reported experiments.Source: Multi-positive contrastive learning · p. 15

Model optimization

This section describes the retrieval evaluation and analysis procedures, including participant-matched baselines, semantic clustering, and functional-network ablations.

Model optimization

  • Training used AdamW with cosine-annealed learning rates, global batches of 1,024, and early stopping based on held-out minibatch Top-5 retrieval.The stopping proxy was distinct from full-gallery retrieval metrics and required five consecutive epochs without improvement of at least 10^-4.Source: Model optimization · pp. 15–16

Stimulus retrieval

  • Retrieval ranked unique candidate stimuli by cosine similarity to each predicted brain embedding, using global or dataset-restricted candidate pools.The global pool combined validation stimuli across seven datasets; dataset-restricted pools removed cross-dataset distractors and had dataset-dependent chance levels.Source: Stimulus retrieval · p. 16

Participant-matched baseline comparison

  • The participant-matched comparison used unique-stimulus Top-10 accuracy for four BOLD5000 and four NSD participants with results for all three methods.The MindEye-style baseline predicted a pooled CLIP embedding for retrieval, without the full framework’s diffusion prior or image reconstruction.Source: Participant-matched baseline comparison · p. 16

Semantic organization of the stimulus space

  • Semantic clusters were defined by k-means on CLIP embeddings, with t-SNE used only to visualize the clusters.Ten clusters spanning high-to-low retrieval accuracy were selected by taking every fourth cluster after ranking by accuracy, then reused in geometry, ablation, and saliency analyses.Source: Semantic organization of the stimulus… · p. 16

Functional-network ablation

  • Functional-network ablations independently removed each of nine ROI groups from attention and pooling while keeping model parameters, retained-token embeddings, and candidate pools unchanged.Ablated tokens were deleted rather than zeroed; pooled effects weighted queries equally, while participant-level summaries weighted participants equally.Source: Functional-network ablation · p. 17
  • Ablation effects were tested with one-sided Wilcoxon signed-rank tests, corrected across nine networks, and network differences with a Friedman repeated-measures test.The analysis also used bootstrap confidence intervals and rank-biserial effect sizes; positive retrieval drops indicate impairment after ablation.Source: Functional-network ablation · p. 17

Zero-shot cross-dataset transfer

The zero-shot curriculum tests whether expanding source training improves retrieval on entirely unseen target datasets without participant-specific conditioning.

Zero-shot cross-dataset transfer

  • Target datasets, participants, and stimuli were excluded from training, and participant embeddings were disabled throughout each curriculum experiment.The target evaluation set remained fixed while source datasets were added cumulatively, one per step.Source: Zero-shot cross-dataset transfer · p. 17
  • Performance was expressed relative to the initial source-training condition to compare target datasets with different absolute retrieval accuracies.The step-0 model is represented at 0%, and the final annotation reports the total change from the first to last curriculum conditions.Source: Zero-shot cross-dataset transfer · p. 18

Semantic geometry of predicted brain representations

This section examines whether the learned representation preserves semantic neighborhood structure beyond identifying the matched stimulus.

Semantic geometry of predicted brain representations

  • Candidate stimuli were grouped as the matched item, non-matching items in its semantic cluster, or items from different clusters, and mean cosine similarities were compared.Identity and semantic margins summarize stimulus-specific information beyond category membership and preservation of broader category structure, respectively.Source: Semantic geometry of predicted brain… · p. 18

ROI saliency and semantic selectivity

This section quantifies ROI attribution and tests whether cluster-selective ROIs are overrepresented in functional groups.

ROI saliency and semantic selectivity

  • ROI attribution sums the absolute products of z-scored, clipped beta values and their effect on matched-stimulus similarity, then normalizes each query’s values across ROIs.The resulting attribution reflects within-ROI spatial activity patterns rather than mean amplitude.Source: ROI saliency and semantic selectivity · p. 18
  • Cluster selectivity is measured by standardized differences in normalized ROI attribution inside versus outside each semantic cluster, retaining the 20 positively selective ROIs per cluster.Network enrichment compares selected-ROI counts with expected counts proportional to each functional group’s share of the 379-ROI atlas; positive scores indicate overrepresentation.Source: ROI saliency and semantic selectivity · pp. 18–19

Cross-dataset semantic geometry and retrieval robustness

The analyses test whether brain embeddings preserve stimulus identity, semantic structure, and local CLIP neighbourhoods across participants, and how retrieval behaves as candidate pools become more challenging. Cross-participant margins and cross-dataset representational dissimilarity matrices provide complementary tests of shared geometry.

Cross-participant geometry

  • Same-stimulus, same-cluster, and different-cluster embedding similarities define identity and semantic margins for testing hierarchical organization across participant pairs.At most ten participant pairs were sampled per stimulus; same-cluster controls used different stimuli, with comparisons averaged in both directions.Source: Cross-dataset semantic geometry and… · p. 19
  • Pair-level summaries do not account for dependence among participants or shared stimuli, so population-level inference requires a design respecting both sampling factors.Margins were tested with one-sided Wilcoxon signed-rank tests, and group means and 95% confidence intervals were bootstrapped over participant pairs.Source: Cross-dataset semantic geometry and… · p. 19

Cross-dataset correspondence

  • Cross-dataset semantic geometry is measured by correlating dataset-specific cosine-distance matrices for shared cluster centroids, excluding clusters with fewer than five stimuli or fewer than five shared clusters.Significance uses a 2,000-permutation Mantel-style test with Benjamini–Hochberg correction across evaluable dataset pairs.Source: Cross-dataset semantic geometry and… · pp. 19–20

Retrieval robustness

  • Candidate-pool scaling estimates expected Top-10 retrieval analytically from each query’s full-pool rank under uniform sampling without replacement.The analysis evaluates multiple candidate-pool sizes and preserves the full-pool ranking structure rather than repeatedly drawing candidate sets.Source: Cross-dataset semantic geometry and… · p. 20
  • Hard-negative retrieval increases the fraction of same-cluster distractors at fixed pool size, testing discrimination among stimuli within a semantic neighbourhood.Expected performance is calculated from same- and different-cluster distractors’ full-ranking positions rather than repeated candidate resampling.Source: Cross-dataset semantic geometry and… · pp. 20–21

Local neighbourhood preservation

  • Local-neighbourhood preservation compares each stimulus’s k-nearest neighbours in brain and CLIP spaces against an analytical random-overlap expectation.Stimulus-level brain embeddings are averaged and normalized before cosine-similarity neighbourhoods are compared across the specified neighbourhood sizes.Source: Cross-dataset semantic geometry and… · p. 21

Funding Statements

The research received support from a company and named research projects.

Funding

  • Tether Operations, S.A. de C.V. provided salary support for participating employees and resources required to conduct the research.Source: Funding Statements · p. 21
  • The BRAINFEDERATION FISA-2022-00899 supported the work.Source: Funding Statements · p. 21
  • The CROSSBRAIN project and BRAINSTORM project also supported the work through European Union grants.CROSSBRAIN grant agreement No. 101070908; BRAINSTORM grant agreement number 101099355.Source: Funding Statements · p. 21

Supplementary information

The supplementary material documents MOSAIC’s composition, extends semantic and cortical analyses to all 40 stimulus clusters, broadens the Ridge comparison, and checks robustness to checkpoint selection.

Composition of the aggregated MOSAIC dataset

  • MOSAIC contains 93 dataset-specific participant entries corresponding to 77 individuals; cross-dataset stimulus filtering reduces duplicate or near-duplicate stimuli and associated trials while preserving the participant collection.Dataset-specific entries, rather than unique individuals, are the modeling units.Source: Composition of the aggregated MOSAIC… · p. 24, Table S1 · p. 24

Complete semantic organization of the validation stimulus space

  • Across all 40 semantic clusters, retrieval varied substantially and did not simply track cluster stimulus counts; many categories included stimuli from multiple evaluation datasets.The clusters span visual content including people, animals, actions, landscapes, vehicles, architecture, and manufactured objects.Source: Complete semantic organization of the… · p. 24, Figure S1 · p. 25

Semantic geometry and cortical contributions across all stimulus clusters

  • Across all 40 clusters, ventral visual/object and early visual regions produced some of the largest ablation-related retrieval impairments, while network effects and semantic margins varied by category.The positively cluster-selective ROI composition also varied markedly across clusters.Source: Semantic geometry and cortical… · p. 24, Figure S2 · p. 26

Expanded participant-level comparison with Ridge regression

Robustness to checkpoint selection

  • Checkpoint selection increased overall Top-10 retrieval by 1.6 percentage points, while participant-specific retrieval was essentially unchanged and semantic-category ranking remained highly preserved.Semantic-category retrieval profiles correlated at Spearman ρ = 0.98; the candidate-pool scaling pattern was also preserved.Source: Robustness to checkpoint selection · p. 28, Figure S3 · p. 29
Loading 2610.09352v1…