PaperlayerView exact revision on arXiv

Source-linked AI summary

Meta-learning In-Context Enables Training-Free Cross Subject Brain Decoding

Mu Nan, Muquan Yu, Weijian Mai, Jacob S. Prince, Hossein Adeli, Rui Zhang, Jiahang Cao, Benjamin Becker, John A. Pyles, Margaret M. Henderson, Chunfeng Song, Nikolaus Kriegeskorte, Michael J. Tarr, Xiaoqing Hu, Andrew F. Luo

arXiv:2604.08537v1cs.LGq-bio.NC

TL;DR

Cross-subject fMRI decoding is hindered by substantial neural variability, which has required bespoke models or subject-specific fine-tuning. BrainCoDec addresses this gap through meta-learned hierarchical in-context inversion of visual encoders, using a small set of image-brain examples to adapt to unseen subjects. It reports generalization across subjects, scanners, and acquisition protocols without fine-tuning, anatomical alignment, or stimulus overlap.

  • Problem

    Substantial cross-subject variability makes mappings learned for one individual ineffective for another, creating a need for generalizable visual decoding models.

  • Method

    BrainCoDec meta-learns hierarchical functional inversion by inferring voxelwise encoder parameters from image-brain contexts and integrating them across voxels.

  • Results

    BrainCoDec generalizes to novel subjects, scanners, and acquisition protocols without retraining or fine-tuning, anatomical alignment, or stimulus overlap.

  • Takeaways & Limitations

    The framework provides a training-free foundation approach for decoding across heterogeneous neural datasets and studying population-level cortical organization.

  • Takeaways & Limitations

    The inversion assumes sufficiently many voxels and accurate encoders, while encoder bias and metric choice can affect the recovered stimulus.

Abstract

from arXiv · show

Visual decoding from brain signals is a key challenge at the intersection of computer vision and neuroscience, requiring methods that bridge neural representations and computational models of vision. A field-wide goal is to achieve generalizable, cross-subject models. A major obstacle towards this goal is the substantial variability in neural representations across individuals, which has so far required training bespoke models or fine-tuning separately for each subject. To address this challenge, we introduce a meta-optimized approach for semantic visual decoding from fMRI that generalizes to novel subjects without any fine-tuning. By simply conditioning on a small set of image-brain activation examples from the new individual, our model rapidly infers their unique neural encoding patterns to facilitate robust and efficient visual decoding. Our approach is explicitly optimized for in-context learning of the new subject's encoding model and performs decoding by hierarchical inference, inverting the encoder. First, for multiple brain regions, we estimate the per-voxel visual response encoder parameters by constructing a context over multiple stimuli and responses. Second, we construct a context consisting of encoder parameters and response values over multiple voxels to perform aggregated functional inversion. We demonstrate strong cross-subject and cross-scanner generalization across diverse visual backbones without retraining or fine-tuning. Moreover, our approach requires neither anatomical alignment nor stimulus overlap. This work is a critical step towards a generalizable foundation model for non-invasive brain decoding.

1. Introduction

The paper targets cross-subject visual decoding, where neural variability has required subject-specific models or fine-tuning. It introduces BrainCoDec, a hierarchical in-context inversion framework designed to generalize without fine-tuning, anatomical alignment, or stimulus overlap.

  • Motivation: Inter-subject neural variability makes mappings learned for one individual ineffective for another, requiring data-intensive retraining or fine-tuning.This limits development of population-wide brain models and applications such as BCIs, cognitive assessment, and personalized diagnostics.
  • Approach: The framework infers forward-model parameters from stimulus-activity pairs for each voxel, then fuses those parameters with observed activations across voxels to reconstruct stimuli.This two-stage design is illustrated as in-context estimation followed by aggregated decoding.
  • Approach: BrainCoDec treats decoding as hierarchical functional inversion constrained by an encoding model that predicts brain activity from stimuli.It first estimates voxelwise visual-response weights in context, then integrates information across multiple brain regions.
  • Contribution: BrainCoDec generalizes to novel subjects without fine-tuning, anatomical alignment, or stimulus overlap, including across scanners and acquisition protocols.The claimed cross-scanner and cross-protocol generalization is presented as a contribution of the framework.
  • Contribution: Selective dropout and a small subset of higher-visual-cortex voxels provide robustness to input variability, while attention maps align with known visual-cortex functional regions.These analyses support robustness and interpretability claims.

2. Related work

Related work frames brain analysis through complementary encoding and decoding models, increasingly supported by learned visual features and generative models. BrainCoDec combines meta-training with in-context learning to adapt functionally to novel subjects without parameter updates.

  • Computational Encoding and Decoding Models: Encoding models predict neural activity from stimuli, whereas decoding models reconstruct stimuli from brain activity.Both approaches commonly use learned visual features mapped to neural activity.
  • In-Context Learning: BrainCoDec builds on meta-optimized encoding models and uses in-context learning as an implicit adaptation mechanism for novel subjects.The framework integrates meta-training with in-context learning to avoid parameter updates during adaptation.
  • BrainCoDec: Its architecture performs voxelwise in-context encoder inference before integrating encoder parameters and activations across multiple voxels.Both context sizes can vary across the two stages.
  • Computational Encoding and Decoding Models: Generative models have enabled stimulus decoding from fMRI, EEG, and MEG across images, dynamic visual stimuli, and speech, audio, or language.Recent cross-subject approaches using flatmaps, pooling, or surfaces require anatomical alignment.

3. Methods

BrainCoDec treats cross-subject visual decoding as hierarchical functional inversion, using in-context examples to infer voxelwise encoding functions and aggregate them across voxels. The method supports training-free decoding for unseen subjects while accommodating variable voxel contexts and requiring neither anatomical alignment nor stimulus overlap.

  • Decoding as the Functional Inversion: BrainCoDec reformulates neural decoding as meta-learned functional inversion rather than a fixed mapping from neural representations to stimuli.The approach learns to perform in-context inversion across a variable number of higher visual cortex voxels.
  • Motivation and Problem Definition: At test time, a small set of image-brain activation pairs provides context for inferring the embedding of an unseen image for a new subject.The goal is to infer the unseen image embedding using only these context examples.
  • Hierarchical Training-Free Stimulus Decoding: Hierarchical inference first estimates voxelwise encoder parameters from stimulus-response contexts, then inverts aggregated encoder representations across multiple voxels.Stage 1 operates independently for each voxel; Stage 2 constructs a voxel context for a single novel stimulus.
  • Hierarchical Training-Free Stimulus Decoding: BrainCoDec enables unseen-subject image retrieval without fine-tuning, anatomical alignment, or stimulus overlap.The reported comparison includes BrainCoDec-200 against MindEye2 with anatomical alignment and TGBD; the method uses 200 in-context images.
  • Hierarchical Training-Free Stimulus Decoding: The voxel-context transformer jointly inverts multiple functional representations while supporting variable-length contexts and order-invariant voxel inputs.The model omits positional embeddings and uses logit scaling to accommodate differing context sizes.
  • Training Objective: The hybrid cosine-contrastive objective combines cosine embedding loss with InfoNCE to optimize reconstruction and instance-level discriminability.The loss is described as supporting both fine-grained alignment and instance-level discrimination.

4. Experiments and Analysis

BrainCoDec is evaluated on unseen subjects, selective regional dropout, attention interpretability, context scaling, and cross-site generalization. It improves retrieval without retraining, reaches near-full-context performance with limited context, and transfers across scanners and datasets.

  • Unseen Subject Brain Decoding: BrainCoDec consistently outperforms MindEye2 and TGBD on unseen-subject image retrieval without retraining.The comparison uses the limited-context BrainCoDec-200 variant and four leave-one-subject-out conditions.
  • Contextual Scaling: Increasing image or voxel context consistently improves decoding across subjects and CLIP, DINO, and SigLIP backbones.With 200 images and 4,000 voxels, performance is similar to inference using approximately 9,000 images and all higher-visual-cortex voxels.
  • Robust Decoding through ROI Dropout: Targeted dropout of category-selective regions causes minimal degradation for most categories, indicating decoding does not rely on any single functional region.The analysis masks regions including PPA, OPA, and RSC for scene-related stimuli.
  • Neural Interpretability via Attention Analysis: Attention weights focus on face- and body-selective regions for faces and place-related regions for places, matching established cortical semantics.The attention maps align with category-selective regions such as FFA, EBA, PPA, OPA, and RSC.
  • Neural Interpretability via Attention Analysis: UMAP embeddings of voxelwise attention weights form semantic clusters that mirror known representational gradients across higher visual cortex.The organization suggests the model learns where semantically relevant neural representations occur, not only how to invert them.
  • New Scanner Adaptation on BOLD5000: BrainCoDec achieves strong BOLD5000 retrieval across held-out subjects and visual backbones despite different scanners, stimuli, voxel sizes, and scanning protocols.BOLD5000 provides a cross-site test using a 3T scanner, 2 mm isotropic voxels, and a different subject pool and image set.

5. Conclusion

The conclusion presents BrainCoDec as a foundation framework for training-free fMRI decoding across heterogeneous subjects and acquisition settings. It combines meta-learned functional inversion with hierarchical in-context inference and points toward broader neural-decoding applications.

  • Conclusion: BrainCoDec generalizes fMRI decoding across subjects, scanners, and acquisition protocols without fine-tuning.The framework uses meta-learning to invert visual encoding functions through hierarchical in-context inference across stimuli and voxels.
  • Conclusion: The framework achieves gains in data efficiency, interpretability, and cross-subject performance over strong baselines.The conclusion also describes learned functional inversion as a computational lens on population-level cortical organization.
  • Conclusion: The strategy is proposed as extensible to EEG, MEG, and other modalities for training-free neural decoding.The stated application areas include cognitive science, machine perception, and real-world BCIs.

A. Technical Appendices and Supplementary Material Sections

The supplementary material expands the paper’s technical, quantitative, retrieval, context-scaling, attention, and ablation analyses.

  • Technical Appendices and Supplementary Material Sections: The appendices cover model architecture, implementation details, quantitative and retrieval comparisons, context scaling, attention UMAP analyses, additional retrieval results, and model ablations.These sections include analyses on unseen NSD and BOLD500 subjects.

A.1. Model Architecture

BrainCoDec combines voxel-context projection, transformer-based aggregated encoder inversion, and an image-embedding prediction head. It is evaluated with CLIP, DINOv2, and SigLIP ViT-B variants of different embedding dimensions and parameter counts.

  • Voxel context token input projection: BrainCoDec forms voxel context tokens by concatenating each voxel’s response-function parameter with its measured neural activation.A residual MLP projects each concatenated token before transformer processing.
  • Contextual decoder transformer: An eight-layer self-attention transformer performs aggregated encoder inversion across voxel and register tokens to infer the stimulus.The blocks use pre-normalization, logarithmic sequence scaling by voxel count, dropout, and SwiGLU feed-forward layers.
  • Image embedding prediction head: After transformer processing, the model retains register tokens for image-embedding prediction.
  • Backbone variants: CLIP, DINOv2, and SigLIP variants use ViT-B backbones with embedding dimensions 512, 768, and 1152, respectively.Their approximate parameter counts are 55.70M, 88.76M, and 157.35M, respectively.

A.2. Implementation Details

Training uses supervised embedding prediction with cosine-similarity and InfoNCE losses, while experiments evaluate held-out-subject variants across completed and incomplete NSD scans. Supplemental results distinguish subjects used for training from subjects unseen during training.

  • Training procedure: Each training step samples in-context voxel tokens and target image embeddings, then optimizes a combined cosine-similarity and InfoNCE objective.Dropout is applied throughout residual and attention blocks for regularization.
  • Datasets and models: The main experiments focus on NSD subjects S1, S2, S5, and S7, which completed scanning.
  • Datasets and models: Across three backbones, the study trains 15 models comprising four single-subject-held-out variants and one all-subject model per backbone.The variants are described as fine-tuned versions of a model trained with synthetic data only.
  • Evaluation organization: Supplemental results first cover NSD subjects S1, S2, S5, and S7, then subjects S3, S4, S6, and S8.Unless otherwise noted, the model has not seen the particular subject during training.

A.3. Quantitative table for S2-8

The supplement includes quantitative comparison tables for NSD subjects 1, 2, 5, 7 and subjects 3, 4, 6, 8, but the supplied passages provide no table entries or metric values.

  • Subjects 1, 2, 5, and 7: Table S.1 presents a quantitative comparison on NSD subjects 1, 2, 5, and 7.
  • Subjects 3, 4, 6, and 8: Table S.2 presents a quantitative comparison on NSD subjects 3, 4, 6, and 8.

A.4. Retrieval visualizations for NSD

Supplementary figures show image-retrieval comparisons on unseen subjects S1 through S8. Each visualization is organized around a specific unseen subject.

  • Unseen-subject retrieval: Figure S.1 shows an image-retrieval comparison on unseen subject S1.
  • Unseen-subject retrieval: Figure S.2 shows an image-retrieval comparison on unseen subject S2.
  • Unseen-subject retrieval: Figure S.3 shows an image-retrieval comparison on unseen subject S5.
  • Unseen-subject retrieval: Figure S.4 shows an image-retrieval comparison on unseen subject S7.
  • Unseen-subject retrieval: Figure S.5 shows an image-retrieval comparison on unseen subject S3.
  • Unseen-subject retrieval: Figure S.6 shows an image-retrieval comparison on unseen subject S4.
  • Unseen-subject retrieval: Figure S.7 shows an image-retrieval comparison on unseen subject S6.
  • Unseen-subject retrieval: Figure S.8 shows an image-retrieval comparison on unseen subject S8.

A.5. Context scaling of other unseen NSD subjects

This section presents context-scaling visualizations across NSD and BOLD5000 unseen subjects, alongside semantic attention and image-retrieval analyses. It also compares model variants and ablations using novel-subject brain activations as context.

  • Figures S.9 and S.10 examine image-context and voxel-context scaling on NSD subjects 3, 4, 6, and 8.
  • Figures S.11 and S.12 examine image-context and voxel-context scaling on BOLD5000 subjects.
  • Figure S.13 visualizes semantic attention patterns in BrainCoDec.
  • Figure S.14 presents image-retrieval results for BOLD5000 unseen subjects using 80 context images and a 500-image retrieval pool.BOLD5000 provides only 20 test images, so the figure visualizes retrieval from a larger pool for evaluation.
  • Model variants and ablations are quantitatively compared using 200 images and brain activation patterns from the novel subject as context.The comparison includes a synthetic-data-only PT model and an Inversion model that uses gradient-based optimization to recover image embeddings.
Loading 2604.08537v1…