PaperlayerView exact revision on arXiv

Source-linked AI summary

The Wisdom of a Crowd of Brains: A Universal Brain Encoder

Roman Beliy, Navve Wasserman, Amit Zalcher, Michal Irani

arXiv:2406.12179v4cs.CV

TL;DR

Existing brain encoders are largely subject- and machine-specific, while limited scanning time constrains available training data. This paper introduces a voxel-centric encoder trained jointly across diverse datasets; evaluations report improved encoding from pooled subjects, gains for older 3-T data using 7-T data, and adaptation with few new examples.

  • Problem

    Existing brain encoders are primarily subject- and machine-specific and require extensive individual data, while limited fMRI scanning time and between-person variability constrain datasets and cross-subject data combination.

    Source: Introduction · p. 1

  • Method

    A universal image-to-fMRI encoder jointly trains across diverse datasets using a learned embedding for each voxel and cross-attention between voxel embeddings and multi-level image features.

    Source: Introduction · pp. 1–2

  • Results

    Joint training on subjects with no shared stimuli exceeds any cohort member’s individual performance; the encoder also improves older 3-T datasets using 7-T data and adapts to new subjects or datasets with few examples.

    Source: Introduction · p. 2, Experiments & Evaluations · p. 5

  • Takeaways & Limitations

    Voxel embeddings capture voxel sensitivity to image features, allowing voxel functionality to emerge through voxel-image cross-attention.

    Source: Introduction · pp. 1–2

    For subjects with poor signal quality, voxel functionality is harder to estimate, and meaningful brain-region segmentation may be less successful.

    Source: C Assumptions & Limitations · p. 13

Abstract

from arXiv · show

Image-to-fMRI encoding is important for both neuroscience research and practical applications. However, such "Brain-Encoders" have been typically trained per-subject and per fMRI-dataset, thus restricted to very limited training data. In this paper we propose a Universal Brain-Encoder, which can be trained jointly on data from many different subjects/datasets/machines. What makes this possible is our new voxel-centric Encoder architecture, which learns a unique "voxel-embedding" per brain-voxel. Our Encoder trains to predict the response of each brain-voxel on every image, by directly computing the cross-attention between the brain-voxel embedding and multi-level deep image features. This voxel-centric architecture allows the functional role of each brain-voxel to naturally emerge from the voxel-image cross-attention. We show the power of this approach to (i) combine data from multiple different subjects (a "Crowd of Brains") to improve each individual brain-encoding, (ii) quick & effective Transfer-Learning across subjects, datasets, and machines (e.g., 3-Tesla, 7-Tesla), with few training examples, and (iii) use the learned voxel-embeddings as a powerful tool to explore brain functionality (e.g., what is encoded where in the brain).

Introduction

Existing image-to-fMRI encoders are typically subject- and dataset-specific, while small datasets and between-person variability constrain brain-function analysis. The paper proposes a voxel-centric encoder that combines diverse data, with reported benefits for encoding, transfer learning, and exploring brain functionality.

  • Small fMRI datasets and variability in brain responses across people make it difficult to analyze brain functionality and combine data across individuals.Source: Introduction · p. 1
  • The Universal Brain-Encoder jointly trains across diverse subjects, datasets, image stimuli, and scanners using a voxel-centric architecture.Each voxel has a learned embedding, while other network components are shared across voxels and subjects.Source: Introduction · p. 1, Figure 1 · p. 2
  • Cross-attention between each voxel embedding and multi-level image features predicts that voxel’s response and learns its visual functionality.Source: Introduction · pp. 1–2
  • Combining data from crowds of brains and datasets provides more training data and leads to significant encoding improvement.Source: Introduction · p. 2
  • Transfer learning to new subjects or datasets requires few training examples.Source: Introduction · p. 2
  • Learned voxel embeddings capture known visual functionality and may help explore what is encoded where in the brain.Source: Introduction · p. 2

Related Work

Earlier multi-subject image-to-fMRI approaches have faced constraints on shared stimuli, joint training, or the number of voxels they predict. The paper contrasts those approaches with voxel-level sharing across brains.

  • Most prior brain models were subject-specific, and anatomical alignment did not provide accurate functional correspondences.Source: Related Work · p. 2
  • Earlier multi-subject methods required shared images (Van Uden et al. (2018)), finetuned across subjects without joint training (Wen et al. (2018b); Gu et al. (2022)), or predicted only a small subset of highly correlated voxels (Khosla et al. (2020)).Source: Related Work · pp. 2–3
  • Unlike fMRI-centric methods that share representations of whole scans, the proposed voxel-centric model shares network weights across voxels within and across brains.The paper motivates this design by noting that similarities between individual voxels are more frequent than similarities between entire scans.Source: Related Work · p. 3, Figure 2 · p. 3

The Universal-Encoder

The Universal-Encoder jointly learns from diverse fMRI datasets by predicting individual voxel responses with shared network components and voxel-specific embeddings. Its cross-attention connects voxel functionality to relevant image features.

The Universal-Encoder

Overview of the Approach

  • Each voxel has a learned embedding that captures its visual functionality, while shared components enable functional similarities to surface across voxels and brains.Functionally similar voxels can map to nearby embeddings, supporting exploration of brain regions.Source: Overview of the Approach · pp. 3–4

Architecture and Training

  • The architecture extracts multi-scale image features, assigns a voxel-specific embedding, and uses spatial attention, MLPs, and functional attention to predict a scalar voxel response.Spatial attention selects image locations; functional attention weights the attended features to produce the activation prediction.Source: Architecture and Training · pp. 4–5

Experiments & Evaluations

Experiments compare the Universal-Encoder across datasets and subjects, test whether joint training improves encoding, and evaluate transfer to new subjects and prior-work baselines.

The Wisdom of a “Crowd of Brains”

  • ∼7% improvement for Subject1 and ∼15% for Subject8 in median Pearson correlation; multi-subject training also leads in Top-1 and Top-5 retrieval.The jointly trained model uses the union of eight subjects’ non-shared training images and is evaluated on each subject individually.Source: Figure 4 · p. 6, The Wisdom of a “Crowd of Brains” · pp. 6–7

Experiments & Evaluations

  • Evaluation spans Vim1, fMRI-on-ImageNet, and NSD, using voxelwise Pearson correlation, encoding accuracy, MSE, image retrieval, and RSA.The NSD test set contains approximately 1,000 images shared across subjects.Source: Experiments & Evaluations · pp. 5–6

Quantitative Comparison to Prior Work

The Wisdom of a “Crowd of Datasets”

Transfer-Learning to New Subjects

  • ≥77% improvement for Subject1 in NSD at any training-example count; 100 subject-specific examples also outperform a scratch-trained model using 9,000.Transfer from NSD also significantly outperforms single-subject models on the fMRI-on-ImageNet and VIM1 datasets.Source: Transfer-Learning to New Subjects · p. 7, Figure 6 · p. 7

Exploring the Brain

Voxel-embedding clusters offer a way to examine shared and finer-grained functional organization, while the paper distinguishes encoding from multi-subject fMRI decoding.

Exploring the Brain

  • Clustering voxel-embeddings across subjects reveals shared clusters whose top-activating images suggest distinct functions, including food, faces, text, and outdoor scenes.The shared regions emerged even though the two subjects viewed different images, and their voxel indices were not anatomically aligned.Source: Exploring the Brain · p. 8, Figure 7 · p. 8
  • Within the PPA, clustering separates indoor-scene and outdoor-scene subregions, illustrating potential to identify functional organization beyond predefined anatomical boundaries.The PPA is described as an area corresponding to places and scenes.Source: Exploring the Brain · p. 8

A Additional Related Work: fMRI Decoding

  • fMRI-to-image decoding reconstructs an image from its corresponding fMRI recording; unlike encoding, it maps subject-specific inputs toward a shared image output space.The authors argue that shared-space approaches used for decoding do not transfer directly to encoding, whose fMRI outputs differ across subjects.Source: A Additional Related Work: fMRI Decoding · p. 12

B.1 fmri datasets

The study uses three natural-image fMRI datasets with dataset-specific voxel selection and normalization procedures.

B.1 fmri datasets

  • The datasets contain BOLD responses to natural images across scanning sessions, with peak responses represented as scalar values per voxel and image.Voxel responses are Z-scored within each continuous scanning run to standardize responses across runs.Source: §B.1 · p. 12
  • NSD uses a selected set of approximately 40,000 vision-related voxels, while the dataset provides per-voxel normalization.The selection is from the post-processed dataset of Gifford et al. (2023).Source: §B.1 · p. 12
  • For fMRI-on-ImageNet, the authors use the provided set of around 5,000 voxels and implement Z-scoring normalization.Source: §B.1 · p. 12
  • For VIM1, the authors select the 7,000 highest-SNR voxels and implement Z-scoring normalization.SNR is calculated from variance across averaged repeated measurements relative to average variance within repeated measurements for the same stimuli.Source: §B.1 · p. 12

B.2 Training and Lora Modification

The Lora modification changes which self-attention projection weights are adapted.

B.2 Training and Lora Modification

  • The implementation modifies the output projection weights (Wo), rather than the value and query projection weights (Wv, Wq) modified in the original Lora paper.Source: §B.2 · p. 12

B.3 Training

The model is trained end-to-end to predict voxel activations while aligning predicted activation patterns with their targets. Joint training on eight NSD subjects takes about one day on a single Quadro RTX 8000 GPU.

B.3 Training

  • The objective predicts each voxel’s activation while encouraging the predicted voxel-wise pattern to align with the target pattern’s relative structure.The loss combines mean squared error and cosine similarity with α=0.1.Source: §B.3 · pp. 12–13
  • Each batch samples 32 images and 5,000 voxel indices per image, with each subject using unique voxel indices.Training uses Adam with a learning rate of 1e-3.Source: §B.3 · p. 12
  • Joint training on eight NSD subjects takes about one day on one Quadro RTX 8000 GPU; inference takes about 50 ms per image.Source: §B.3 · p. 13

B.4 The Wisdom of a “Crowd of Datasets”

On fMRI-on-ImageNet and VIM1, adding data from other subjects and datasets improves encoding performance over the single-subject baseline or same-dataset training.

B.4 The Wisdom of a “Crowd of Datasets”

  • Adding NSD training data to the Universal Encoder significantly improves performance on VIM1.Source: §B.4 · p. 13
  • On fMRI-on-ImageNet, the multi-subject same-dataset encoder outperforms the single-subject Gaziv et al. (2022) baseline.Source: §B.4 · p. 13
  • On fMRI-on-ImageNet, adding NSD subjects to the Universal Encoder trained on that dataset yields further improvement.The figure reports median correlation and the 25th and 75th percentiles.Source: §B.4 · p. 13

C Assumptions & Limitations

The work assumes fMRI responses are memory-less and replicable, while its demonstrated brain exploration and regional performance depend on signal quality and visual-feature coverage. Generalization also depends on training data, though the shared embedding can combine datasets.

Assumptions

  • The analysis assumes responses are memory-less and replicable; the latter assumption neglects representational drift and its effect on fMRI signals.Memory-less means prior images do not affect the current response; replicable means an image elicits the same response whenever it is seen.Source: C Assumptions & Limitations · p. 13

Subject Variability

  • Brain exploration is demonstrated for relatively high-SNR subjects; poor signal quality makes voxel-function estimates harder and meaningful region segmentation less likely.Source: C Assumptions & Limitations · p. 13

Performance across regions

  • Performance is highest in early visual areas, while word-related areas show substantially lower correlations; DINO’s training biases may limit word-feature capture.The authors also note higher signal-to-noise and easier-to-predict low-level features as possible reasons for stronger early-visual performance.Source: C Assumptions & Limitations · p. 13

Generalization abilities

  • Generalization depends on training data, although the shared embedding can combine datasets and visual backbones trained on larger image corpora support generalization beyond available image-fMRI pairs.The broader discoveries this could enable are presented as a possibility.Source: C Assumptions & Limitations · p. 13

D Statistical Tests

Statistical tests support performance differences between models, significant voxel predictions for most voxels, retrieval improvements, and transfer-learning gains across datasets.

Prediction significance

  • For eight NSD subjects, repeated-measures ANOVA found a significant prediction-correlation gap between models (F=72.74, p=0; Num DF=2, Den DF=14).Source: D Statistical Tests · p. 13

Voxel significance

  • An average of 98% of voxel predictions across subjects were statistically significant against the null distribution at P < 0.05 with FDR correction.The null distribution used correlations between independent Gaussian random vectors of size N=982.Source: D Statistical Tests · p. 14

Retrieval significance

  • The multi-subject Universal Encoder significantly improves retrieval accuracy over both subject-specific models, with FDR-corrected tests across 10 random-distractor repeats.The highest observed p-value was approximately 9e-13.Source: Table S4 · p. 14, D Statistical Tests · p. 14

Transfer-learning significance

  • Across all datasets, the transferred Universal Encoder significantly outperforms single-subject models in median Pearson correlation.The transfer comparisons use five runs with different random initializations and apply FDR correction.Source: D Statistical Tests · p. 14, Table S5 · p. 15

E Additional evaluation metrics and ablations

Additional evaluations report performance comparisons and ablations, test reconstruction from encoded fMRI, and explore the functional structure captured by voxel embeddings. They also examine brain-to-brain transformations using the Universal Encoder.

E Additional evaluation metrics and ablations

  • The Pearson-correlation evaluation compares the Universal Encoder variants with three published encoders and reports scores across coarse brain regions.The figure summarizes median values and interquartile ranges.Source: Figure S8 · p. 16
  • The full model performs best; shared latent space performs significantly worse, while removing spatial or functional attention reduces correlation and retrieval performance.Encoding performance is minimally sensitive to voxel-embedding dimension on Subject 1.Source: Table S6 · p. 16, Figure S9 · p. 17
  • Multi-subject training improves retrieval accuracy over both subject-specific models and improves individual-subject encoding MSE.Retrieval accuracy is reported as significantly higher; MSE is summarized by median and quartiles.Source: Figure S11 · p. 18, Figure S12 · p. 18

F decoding from predicted fMRI

  • Reconstructions from encoded fMRI are comparable to reconstructions from ground-truth fMRI in qualitative and quantitative evaluations.The quantitative comparison averages metrics over approximately 1,000 Subject 1 test images.Source: Table S7 · p. 21, Figure S15 · p. 21

G.1 Brain exploration: functional brain regions

  • Voxel-embedding clusters are largely consistent across subjects and divide EBA into three functionally distinct subregions associated with sports, crowds, and food images.Clusters are explored through the images that most strongly activate them.Source: Figure S16 · p. 22, Figure S17 · p. 23, Figure S18 · p. 24

G.2 Brain exploration: voxel embedding

  • Voxel embeddings separate early, lateral, and parietal brain streams while intermingling across subjects, and resemble predefined category-selective regions for faces, bodies, places, and words.The visualizations randomly sample 50,000 voxels.Source: Figure S19 · p. 25, Figure S20 · p. 26
  • Voxel-embedding similarity across subject 1’s ROIs strongly correlates with representational similarity measured from real fMRI.Embedding similarity is measured by average nearest-neighbor cosine similarity; RSA uses real fMRI data.Source: Figure S21 · p. 27

H Brain-to-brain transformation

  • With no shared stimuli between subject pairs, transformations using the Universal Encoder show improved mean Pearson correlation over the comparison encoder.The evaluation uses 40k voxels per subject.Source: Figure S22 · p. 28
Loading 2406.12179v4…