PaperlayerView exact revision on arXiv

Source-linked AI summary

MindBridge: A Cross-Subject Brain Decoding Framework

Shizun Wang, Songhua Liu, Zhenxiong Tan, Xinchao Wang

arXiv:2404.07850v1cs.CVcs.AI

TL;DR

Brain decoding models are typically limited to the subject whose fMRI data trained them, amid differences in signal dimensions and neural responses and limited data for new subjects. MindBridge uses one cross-subject model with cyclic reconstruction and reset-tuning; experiments on NSD report competitive multi-subject decoding and better new-subject adaptation than training from scratch. The authors note that evaluation is limited to the small NSD dataset.

  • Problem

    Brain decoding is generally subject-specific, while cross-subject use must address variation in fMRI dimensions and neural responses and limited data for new subjects.

    Source: §1 · p. 2, §3.2 · p. 3

  • Method

    MindBridge uses adaptive signal aggregation and cyclic fMRI reconstruction for subject-invariant representations, with reset-tuning to adapt a pretrained model to new subjects.

    Source: §1 · p. 2

  • Results

    On NSD, one MindBridge model performs comparably to subject-specific methods, and new-subject adaptation surpasses methods trained from scratch.

    Source: §1 · p. 2

  • Takeaways & Limitations

    MindBridge supports cross-subject image reconstruction and can synthesize brain signals from other subjects while preserving semantic meaning.

    Source: §6 · p. 8

    Evaluation is restricted to the small NSD dataset, so generalizability remains to be validated on more diverse and larger-scale data.

    Source: §5 · p. 8

Abstract

from arXiv · show

Brain decoding, a pivotal field in neuroscience, aims to reconstruct stimuli from acquired brain signals, primarily utilizing functional magnetic resonance imaging (fMRI). Currently, brain decoding is confined to a per-subject-per-model paradigm, limiting its applicability to the same individual for whom the decoding model is trained. This constraint stems from three key challenges: 1) the inherent variability in input dimensions across subjects due to differences in brain size; 2) the unique intrinsic neural patterns, influencing how different individuals perceive and process sensory information; 3) limited data availability for new subjects in real-world scenarios hampers the performance of decoding models. In this paper, we present a novel approach, MindBridge, that achieves cross-subject brain decoding by employing only one model. Our proposed framework establishes a generic paradigm capable of addressing these challenges by introducing biological-inspired aggregation function and novel cyclic fMRI reconstruction mechanism for subject-invariant representation learning. Notably, by cycle reconstruction of fMRI, MindBridge can enable novel fMRI synthesis, which also can serve as pseudo data augmentation. Within the framework, we also devise a novel reset-tuning method for adapting a pretrained model to a new subject. Experimental results demonstrate MindBridge's ability to reconstruct images for multiple subjects, which is competitive with dedicated subject-specific models. Furthermore, with limited data for a new subject, we achieve a high level of decoding accuracy, surpassing that of subject-specific models. This advancement in cross-subject brain decoding suggests promising directions for wider applications in neuroscience and indicates potential for more efficient utilization of limited fMRI data in real-world scenarios. Project page: https://littlepure2333.github.io/MindBridge

1. Introduction

Brain decoding is largely subject-specific, while cross-subject use must contend with differences in fMRI input size, neural responses, and data scarcity for new subjects. MindBridge addresses these challenges with a single framework and reports competitive multi-subject decoding on NSD, plus improved adaptation over methods trained from scratch.

1. Introduction

  • Cross-subject decoding must handle subject-specific fMRI dimensions, differing neural responses, and scarce data for adapting to new subjects.The paper frames cross-subject decoding as a way to use one model across people and reduce the expense of storing and training separate models.Source: §1 · p. 2
  • MindBridge combines adaptive max pooling for common input dimensions with cyclic fMRI reconstruction to learn subject-invariant representations aligned in CLIP space.The aggregation is motivated by sparse brain activation; the shared embeddings support interpretation across varying neural responses.Source: §1 · p. 2
  • Reset-tuning adapts to a new subject by resetting shallow layers while reusing deep layers from cross-subject pretraining.The paper associates transferable knowledge with deep layers and subject-specific projection into invariant embeddings with shallow layers.Source: §1 · p. 2
  • MindBridge uses image and text training data with a versatile diffusion model and can synthesize brain signals from other subjects while preserving semantic meaning.The authors describe this synthesis as a potential way to augment available data.Source: §1 · p. 2
  • MindBridge uses one model to reconstruct images across multiple subjects, matching subject-specific methods, and its new-subject adaptation surpasses methods trained from scratch.The NSD training set has no common images across subjects, adding a challenge to cross-subject decoding.Source: §1 · p. 2
  • The paper presents MindBridge as a cross-subject framework combining adaptive signal aggregation and cyclic reconstruction, with experiments demonstrating efficacy and adaptability.The authors describe these as contributions and characterize the framework as, to their knowledge, the first effective approach to cross-subject brain decoding.Source: §1 · p. 2

2. Related Work

Prior brain decoding methods map fMRI signals toward visual features or generative-model spaces, while MindBridge targets decoding across subjects with one model. The related work also motivates versatile diffusion for its image-and-text guidance capability.

2.1. Brain Decoding

  • Brain-decoding work progressed from sparse regression predicting pretrained CNN features to GAN latent-space methods for reconstructing digits, faces, and natural scenes.The cited examples include [15] for sparse regression and, [42], [12] [25] for GAN-based reconstruction.Source: §2.1 · p. 3
  • MindBridge aims to decode across subjects with one model, unlike almost all brain decoding research described as requiring multiple subject-specific models.Earlier work maps fMRI to CLIP embeddings that guide latent diffusion, with improvements explored through self-supervision, masked modeling, contrastive learning, advanced diffusion, and conditional control.Source: §2.1 · p. 3

2.2. Diffusion Models

  • Versatile Diffusion is an advanced latent diffusion model that can generate images guided by both image and text inputs.The paper adopts it for this dual-input capacity and its image-generation potential.Source: §2.2 · p. 3
  • Diffusion models use iterative denoising from Gaussian noise, while latent diffusion reduces computational demands by denoising in an autoencoder-produced latent space.The passage notes their use in text-to-image generation and the development of latent diffusion models such as Stable Diffusion.Source: §2.2 · p. 3

3. MindBridge

MindBridge uses a shared framework to decode fMRI from multiple subjects, addressing differing signal sizes and the challenge of learning subject-invariant representations. It also adapts to new subjects using reset-tuning and pseudo data augmentation.

3.1. Data Elaboration

  • 982 commonly viewed images form the test set, while each of four subjects contributes 8,859 distinct training images and about 13,000–16,000 ROI voxels.The study uses NSDGeneral ROI data from NSD subjects 01, 02, 05, and 07.Source: §3.1 · p. 3

3.2. Cross-Subject Brain Decoding

  • MindBridge adaptively aggregates variable-sized fMRI into fixed-size inputs, maps them through subject-wise embedders to semantic embeddings, then predicts image and text embeddings for diffusion-based reconstruction.The framework also uses subject-wise brain builders to reconstruct unified fMRI inputs.Source: §3.2 · pp. 3–4, Figure 2 · p. 4
  • MindBridge trains image and text CLIP embeddings with SoftCLIP and MSE losses, with MSE added to improve prediction accuracy when SoftCLIP alone leaves image artifacts.The SoftCLIP objective aligns fMRI and CLIP embeddings through contrastive learning.Source: §3.2 · pp. 4–5
  • Cyclic fMRI reconstruction synthesizes one subject’s signal from another’s semantic embedding, supporting shared representations even though training subjects lack commonly viewed images.The method reconstructs fMRI with a subject-wise brain builder and uses a cycle loss to enforce consistency.Source: §3.2 · p. 5

3.3. New-Subject Adaptation

  • For new-subject adaptation, reset-tuning retrains reset brain embedders and builders while freezing the pretrained brain translator, and cycle reconstruction converts prior subjects’ fMRI into pseudo data.The framework proposes these techniques to address limited data for a new subject.Source: §3.3 · p. 5

4. Experiments and Analysis

MindBridge uses one model to decode fMRI from multiple subjects, achieving performance comparable to subject-specific methods. Experiments also examine limited-data adaptation, novel fMRI synthesis, and the contributions of aggregation and training choices.

4. Experiments and Analysis

  • Evaluation uses eight image-quality metrics, with PixCorr, SSIM, and AlexNet assessing low-level properties and four other metrics assessing higher-level properties.The higher-level metrics are Inception, CLIP, EffNet-B, and SwAV.Source: §4 · p. 5

4.1. Cross-Subject Brain Decoding

  • MindBridge achieves brain-decoding performance comparable to state-of-the-art methods while using one model across subjects, unlike their per-subject models.The comparison averages metrics across four subjects.Source: §4.1 · pp. 5–6, Table 1 · p. 6

4.2. New Subject Adaptation

  • With 500, 1500, and 4000 training points for subject 7, reset-tuning from cross-subject pretraining outperforms training MindBridge from scratch.The tested subsets come from 8859 training points; the authors note potential to reduce scan times in practical applications.Source: §4.2 · pp. 6–7, Table 2 · p. 7

4.3. Novel fMRI Synthesis

  • Cycle reconstruction converts subjects 2 and 5 fMRI into subject 1 fMRI, whose signals decode stimuli subject 1 had never viewed.The reconstructed images are reported as faithful, demonstrating novel fMRI synthesis within the tested setup.Source: §4.3 · p. 7, Figure 4 · p. 7

4.4. Ablation Study

  • Adaptive max pooling outperforms adaptive average pooling and interpolation while retaining biological interpretability.The ablation evaluates models trained and tested on subject 1.Source: §4.4 · p. 7, Table 3 · p. 7
  • Adding reconstruction and cycle losses improves subject-invariant representation learning and cross-subject decoding performance.The pretraining-loss ablation trains on subjects 1, 2, and 5 and evaluates on subject 1.Source: §4.4 · p. 8, Table 4 · p. 8
  • Reset-tuning with pseudo data augmentation yields the most satisfactory fine-tuning results, and augmentation further improves performance.The fine-tuning ablation evaluates adaptation to subject 7 after pretraining on subjects 1, 2, and 5.Source: §4.4 · p. 8, Table 5 · p. 8

5. Discussion

The evaluation is limited to the small NSD dataset, and broader generalizability remains to be tested. The discussion also notes a potential loss of spatial relationships from 1D fMRI serialization and ethical requirements for responsible use.

5. Discussion

  • Evaluation is restricted to the small NSD dataset, so generalizability still needs validation on more diverse and larger-scale data.The paper attributes this scope boundary to limited availability of high-quality fMRI data.Source: §5 · p. 8
  • Serializing fMRI signals as 1D vectors may ruin their original spatial relationships.The authors identify this as another limitation of the framework.Source: §5 · p. 8
  • Potential misuse for malicious or immoral purposes makes consented data privacy protocols and responsible research conduct necessary.Source: §5 · p. 8

6. Conclusion

MindBridge is a cross-subject brain-decoding framework that addresses variability and data scarcity through adaptive aggregation, cyclic fMRI reconstruction, and reset-tuning. Experiments on NSD demonstrate its effectiveness across multiple subjects and point to fMRI synthesis and practical applications in neuroscience.

6. Conclusion

  • MindBridge combines adaptive signal aggregation, cyclic fMRI reconstruction, and reset-tuning to address cross-subject decoding challenges in NSD experiments.The conclusion characterizes the framework as addressing size variability, diverse neural responses, and data scarcity for new subjects.Source: §6 · p. 8
  • Experiments on NSD show improved decoding accuracy across multiple subjects.Source: §6 · p. 8
  • The authors identify fMRI synthesis and practical neuroscience applications as avenues opened by MindBridge.Source: §6 · p. 8

A. More Dataset Information

The experiments use NSD fMRI data from four participants who completed all scan sessions, with preprocessed voxels from the NSDGeneral posterior-cortex ROI. Training uses one trial per image, while testing averages repeated responses.

A. More Dataset Information

  • NSD contains 7-Tesla fMRI scans from 8 healthy adults who viewed thousands of MS-COCO images across 30–40 sessions.Each participant saw 9,000–10,000 distinct images, with three presentations per image, yielding 22,000–30,000 response trials.Source: A. More Dataset Information · p. 11
  • The experiments mainly use four participants who completed all scan sessions and preprocessed voxels from the NSDGeneral posterior-cortex ROI.The ROI is defined as a subset of posterior-cortex voxels that respond most strongly to presented visual stimuli.Source: A. More Dataset Information · p. 11
  • Training uses one randomly selected single-trial fMRI response per image, while testing averages responses across the three repeated presentations.Source: A. More Dataset Information · p. 11

B. Implementation Details

Implementation uses a versatile diffusion model for image reconstruction, adaptive pooling for uniform fMRI inputs, and image and text embeddings. The appendix also specifies training settings, augmentation, and additional cross-subject reconstruction examples.

B. Implementation Details

  • Image reconstruction uses versatile diffusion with 20 UniPCMultistep steps, guidance scale 5, and a text-image ratio of 0.5.For each test sample, eight images are reconstructed and the one with the highest CLIP similarity to the stimulus is selected.Source: B. Implementation Details · p. 11
  • AdaptiveMaxPool1D maps fMRI signals to 8,192 values, while CLIP ViT-L/14 supplies image and text embeddings of shapes 257 × 768 and 77 × 768.The pooling output is the largest power-of-two integer smaller than all fMRI signal sizes and is applied in the dataloader for uniform batch dimensions.Source: B. Implementation Details · p. 11
  • Cross-subject models train for 600 epochs and new-subject adaptation models for 200 epochs, with batch size 50 per GPU.The loss weights for Limage, Ltext, Lrec, and Lcyc are 1, 1e4, 1, and 1, respectively; single-subject models omit Lrec and Lcyc.Source: B. Implementation Details · p. 11
  • Figure 5 presents additional cross-subject reconstructions for subjects 1, 2, 5, and 7.Source: Figure 5 · p. 11
  • Image augmentation includes random cropping, resizing, and randomized adjustments to appearance and sharpness.For new-subject pseudo data augmentation, the number of data points from previously trained subjects matches the new subject's data count, and all previously trained subjects are included.Source: B. Implementation Details · p. 11

C. More Reconstruction Results

MindBridge uses one model for cross-subject reconstruction, with additional results reported for a model trained on subjects 1, 2, 5, and 7. Its architecture combines adaptive fMRI aggregation with embedding, reconstruction, and translation modules.

C. More Reconstruction Results

  • One MindBridge model trained on subjects 1, 2, 5, and 7 produces cross-subject brain-signal reconstructions.The section presents these additional visual results in Figure 5.Source: C. More Reconstruction Results · p. 11

D. Architecture of MindBridge

  • Adaptive max pooling unifies fMRI input size, alongside a brain embedder, brain builder, and brain translator.The embedder uses a LoRA-like adapter and linear layer; the translator is a four-layer residual MLP with two linear heads and hidden size 2048.Source: D. Architecture of MindBridge · pp. 11–12
  • The brain builder is basically the reverse of the brain embedder, and architectural code for both modules and the translator appears in Figure 6.Source: D. Architecture of MindBridge · p. 12

E. More Results for New Subject Adaptation

MindBridge consistently outperforms the Vanilla method in the reported limited-data adaptation results. The text-image ratio trades semantic correctness against image consistency and diversity, while the paper also flags ethical concerns around brain decoding.

E. More Results for New Subject Adaptation

F. Effect of Text-Image Ratio

  • A 0.5 image-text ratio balances semantic accuracy and visual consistency; increasing text input improves semantic correctness but adds diversity.Figure 7 illustrates reconstructions across text-image ratios, using eight random seeds on subject 1.Source: F. Effect of Text-Image Ratio · p. 12, Figure 7 · p. 14

G. Ethic and Social Impact

  • The authors identify involuntary mind reading as an ethical concern and call for informed consent, robust data privacy, and attention to societal implications.Source: G. Ethic and Social Impact · p. 12
Loading 2404.07850v1…