PaperlayerView exact revision on arXiv

Source-linked AI summary

FlatClip: Reusing Image Foundation Models for fMRI Representation Learning via Cortical Flatmaps

Mo Wang, Wenhao Ye, Zihan Ning, Jiayu Zuo, Junfeng Xia, Hongkai Wen, Quanying Liu

arXiv:2609.31204v2cs.CEcs.CVq-bio.NC

TL;DR

This paper asks whether cortical spatial organization can help transfer frozen image representations to fMRI prediction. FlatClip encodes anatomy-linked cortical flatmaps with frozen SigLIP2 and trains a lightweight probe; results vary across benchmarks, while task-relevant inputs and cortical arrangement matter in the evaluated settings.

  • Problem

    fMRI foundation models use either coarse ROI summaries that may discard cortical geometry or detailed voxel representations that require specialized architectures and substantial computation.

    Source: §1 · pp. 1–2

  • Method

    FlatClip renders cortical activity as anatomy-linked flatmaps, extracts features with frozen SigLIP2, and trains only a lightweight downstream probe.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    Across resting-state benchmarks, FlatClip exceeds most ROI-level baselines but remains below the strongest voxel-level models overall; task-relevant cortical inputs improve NSD visual decoding over whole-cortex inputs.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The evaluations identify cortical arrangement and task-relevant input selection as useful design choices for prediction with frozen image features.

    Source: §5 · p. 9

    Flatmaps introduce cuts and metric distortion, omit subcortical and brainstem signals, and temporal mean pooling discards frame order.

    Source: §5 · p. 10

Abstract

from arXiv · show

Can the spatial organization of cortical activity help transfer frozen image representations to fMRI prediction? We introduce FlatClip, which renders cortical activity as anatomy-linked flatmaps and extracts features with a frozen SigLIP2 encoder. Only a lightweight downstream probe is trained, without fMRI-specific encoder pretraining or fine-tuning. Across four resting-state benchmarks, FlatClip outperforms the evaluated ROI-level baselines on HCP and ADNI, but is weaker on PPMI and below the strongest voxel-level models overall. On NSD visual-fMRI decoding, restricting inputs to visual or NSD-provided task-active cortex improves recognition over whole-cortex inputs. Spatial perturbations reduce performance with both retrained and fixed readouts, and anatomy-linked arrangements outperform cortical-pixel permutations across three colormaps. Together, these results demonstrate a practical route for reusing image-pretrained features in fMRI and highlight cortical arrangement and input-region selection as important design choices. Code is available at https://github.com/OneMore1/FlatClip.

1 Introduction

FlatClip asks whether frozen image features can use cortical spatial organization, addressing a trade-off between coarse ROI representations and detailed but costly voxel models. It uses anatomy-linked cortical flatmaps and evaluates resting-state prediction, visual-fMRI decoding, and spatial controls.

1 Introduction

  • ROI-level models are efficient but coarse and parcellation-dependent, whereas voxel models preserve more detail at the cost of specialized architectures, computation, and storage.This trade-off motivates testing whether a frozen image encoder can use cortical spatial organization.Source: §1 · pp. 1–2
  • FlatClip renders cortical activity as anatomy-linked flatmap sequences, extracts features with frozen SigLIP2, and trains only a lightweight downstream probe.The approach uses no fMRI-specific encoder pretraining or fine-tuning.Source: §1 · p. 2
  • Across resting-state benchmarks, FlatClip exceeds most ROI-level baselines while remaining below the strongest voxel-level models overall; task-relevant cortical inputs improve NSD visual decoding over whole-cortex inputs.The reported resting-state comparison uses a frozen backbone and lightweight probe.Source: §1 · p. 2
  • Block-shuffling, cortical-pixel permutation, and FC heatmaps reduce performance relative to real flatmaps, while spatial-perturbation controls test the role of cortical arrangement.The broader analysis also compares input-region selection and spatial granularity.Source: §1 · p. 2

2 Related Work

Prior fMRI representation learning spans coarse ROI and connectome methods, volumetric models, and surface-based approaches. FlatClip uses cortical flatmaps as an interface for transferring frozen image features, complementing models pretrained specifically on fMRI.

fMRI foundation models across representation scales

  • ROI- or atlas-based methods reduce voxel-level signals into region-wise time series or functional-connectivity graphs, whereas voxel- or volume-based models operate directly on 3D or 4D fMRI and preserve more fine-grained spatial structure.Source: §2 · pp. 2–3

Geometry-aware cortical surface and flatmap representations

  • FlatClip adopts cortical surface rendering as an interface for frozen image-feature transfer, alongside prior flatmap-based fMRI decoding and masked-autoencoding work.Pycortex projects volumetric data onto cortical surfaces and generates flattened visualizations.Source: §2 · p. 3

Mesh- and sphere-based cortical models

  • Cortical models use graph, manifold, spherical-mesh, and patch-based representations; CortexMAE-F instead pretrains a spatiotemporal encoder on fMRI flatmaps.FlatClip provides a complementary reference using an image-pretrained encoder without fMRI-specific backbone training.Source: §2 · p. 3

Image-pretrained encoders as out-of-domain frozen baselines

  • Image-pretrained encoders include masked image modeling, contrastive image-text pretraining, self-distillation, and sigmoid-loss image-text pretraining, including SigLIP2.The pipeline applies frozen SigLIP2 features to resting-state sequences or NSD stimulus-level GLM maps before downstream prediction.Source: §2 · p. 3, Figure 2 · p. 3

3 Method

FlatClip converts cortical activity into anatomy-linked flatmap inputs, extracts features with a frozen SigLIP2 encoder, and trains only lightweight downstream probes. Resting-state sequences and stimulus-level visual-fMRI maps use different feature-pooling and prediction setups.

3.1 Overview

  • FlatClip’s pipeline constructs cortical flatmaps, extracts frozen image features, and uses a lightweight downstream probe for prediction.Only probe parameters are optimized.Source: §3.1 · p. 3, §3.4 · p. 4

3.2 Geometry-aware Flatmap Sequence Adapter

  • Resting-state inputs are time-point flatmaps, whereas visual-task inputs are stimulus-level GLM response maps associated with viewed images.Activity is projected onto cortical vertices, rasterized to a deterministic flatmap, normalized and color-mapped, masked, cropped, and composited as RGB.Source: §3.2 · p. 4

3.3 Frozen SigLIP2 Encoder and Feature Construction

  • A frozen SigLIP2 encoder supplies global frame representations for resting-state data and pooled patch features for NSD visual-fMRI.Resting-state representations are temporally averaged; NSD patch features are pooled to 16 × 16, with their mean forming the main feature and token-rich variants retaining all 256 tokens.Source: §3.3 · p. 4

3.4 Downstream Prediction

  • Downstream tasks train lightweight probes, with a two-hidden-layer MLP as the main resting-state probe and class-weighted losses for resting-state and NSD prediction.Only probe parameters are optimized; the resting-state MLP uses 768 → 256 → 256 → C dimensions, while NSD uses a task-specific multilabel MLP and class-weighted binary cross-entropy.Source: §3.4 · pp. 4–5

4 Evaluating Geometry-Aware Surface Representations

FlatClip evaluates whether anatomy-linked cortical flatmaps paired with frozen image features can support fMRI prediction across resting-state and visual-decoding tasks. Results vary by task and input region, while spatial controls underscore the predictive value of cortical arrangement.

4.1 Resting-state fMRI Benchmarks

  • FlatClip outperforms most ROI-level baselines on HCP and ADNI, but remains below the strongest voxel-level models and is relatively weaker on PPMI.The authors suggest PPMI weakness may reflect the cortical-only input, given Parkinson’s involvement of subcortical and brainstem systems.Source: §4.1 · p. 5, Table 1 · p. 6

4.2 Visual-fMRI COCO80 Multi-label Recognition

  • NSDgeneral achieves the best mean mAP, and visual-cortex and NSDgeneral inputs both improve weighted F1 over whole-cortex input.FlatClip variants also outperform the evaluated general-purpose fMRI foundation-model baselines.Source: Figure 3 · p. 6, §4.2 · p. 7

4.3 Geometry Controls and Representation Scale

  • Real flatmaps perform best on HCP sex classification; voxel models also outperform the 450-ROI model, while adaptive patching helps HCP but has mixed effects on AD diagnosis.The voxel-model comparison uses native volumetric signals and differs in input format and architecture from the ROI model.Source: §4.3 · p. 7, Table 2 · p. 8, Table 3 · p. 8

4.4 Implementation Ablations

  • Retaining additional frame or patch information improves some tasks, while the main implementation uses one compact subject-level feature.Backbone comparisons fix the input and readout, with SigLIP2 as the default encoder.Source: §4.4 · p. 7

4.5 Surface-Model Comparisons

  • CortexMAE-F is slightly stronger on HCP, while FlatClip achieves higher accuracy and weighted F1 on both ADNI tasks.The SiT-style model is competitive on ADNI AD/CN.Source: §4.5 · p. 8, Table 5 · p. 9

4.6 Spatial Arrangement Across Renderers and Readouts

  • Across three colormaps, real arrangements outperform cortical-pixel permutations in weighted F1; retrained probes recover only part of the loss from spatial perturbations.Fixed readouts also degrade substantially under block shuffle, cortical-pixel permutation, and phase randomization.Source: §4.6 · p. 8

4.7 Measured Covariates, Temporal Sampling, and Probe Choice

  • Covariate adjustment changes results only slightly in the reported examples, alternative ADNI windows retain predictive performance, and readout strength varies by task.HCP weighted F1 is 80.88 after age adjustment versus 82.04 raw; AD/CN weighted F1 is 75.34 after motion adjustment versus 75.41 before. MLP is stronger on ADNI and PPMI, while linear probing reaches 84.77 accuracy and weighted F1 on HCP.Source: §4.7 · pp. 8–9

4.8 Direct-Volume Rendering Comparisons

  • On HCP, encoding non-empty volume slices and averaging embeddings yields 89.32 weighted F1, versus 82.04 for FlatClip and 72.81 for axial mean projection.For NSD subj01, direct-volume slice encoding yields 4.342 mAP and 15.922 weighted F1, versus 15.311 and 26.922 for FlatClip-NSDgeneral; Omni-fMRI’s continuous and event-centered inputs perform similarly.Source: §4.8 · p. 9

5 Discussion

FlatClip uses frozen image features to examine how cortical arrangement and task-relevant input selection relate to prediction. The discussion presents it as a practical reference complementary to fMRI-specific encoders, while noting limitations of flatmaps and temporal pooling.

5 Discussion

  • Spatial arrangement and cortical input selection are identified as useful design choices for prediction with frozen image features.The analyses compare spatial perturbations and NSD input masks; anatomy-linked flatmaps are also compared with cortical-pixel permutations across renderers.Source: §5 · p. 9, Table 6 · p. 10
  • FlatClip separates reusable feature extraction from task-specific readout, allowing different cortical inputs to be evaluated with the same frozen encoder.This makes it a practical reference without fMRI-specific backbone training, complementing domain-specific encoders.Source: §5 · pp. 9–10
  • Flatmaps introduce cuts and metric distortion, omit subcortical and brainstem signals, and temporal mean pooling discards frame order.Future work is proposed on hybrid cortical–subcortical inputs and explicit temporal modeling, including naturalistic movie paradigms.Source: §5 · p. 10

A.1 Details of Datasets and Task

The dataset descriptions specify the cohort sizes and train/validation/test partitions used for the HCP, PPMI, and ADNI analyses.

A.1 Details of Datasets and Task

  • HCP includes 1,000 subjects, partitioned into 602 training, 201 validation, and 197 test subjects.The partition is also used for supplementary HCP feature evaluations.Source: §A.1 · p. 13
  • PPMI’s three-way classification cohort contains 474 participants: 29 controls, 142 with PD, and 303 prodromal participants.The training, validation, and test partitions contain 331, 47, and 96 participants, respectively.Source: §A.1 · p. 13
  • ADNI’s MCI/CN task uses 381 subjects split 266/37/78, while its AD/CN task uses 350 subjects split 244/34/72 across training, validation, and test.Source: §A.1 · p. 13

A.2 Evaluation Protocol

The supplementary section describes subject-level evaluation, comparison baselines, frozen-feature configurations, and the effect of temporal sampling. More sampled frames generally improve or stabilize performance, with task-specific exceptions.

A.2 Evaluation Protocol

  • Evaluation assigns all frames from each subject to one partition, selects checkpoints on validation data, and evaluates on held-out test subjects.Supplementary probe experiments use the same partitions and report means and standard deviations over seeds 42, 43, and 44.Source: §A.2 · p. 13

A.3 Details of Baseline Models

  • The baselines include atlas/ROI foundation models and models operating directly on 4D fMRI volumes; Omni-fMRI is used as a strong voxel-level comparator.BrainLM, BrainMASS, Brain-Harmony, and LCM use ROI, connectome, or multimodal subject representations, while SwiFT and NeuroSTORM model raw volumes.Source: §A.3 · pp. 13–14

A.4 Ablation of Architectures

  • With a shared global representation and MLP probe, SigLIP2 performs strongest on HCP among the tested frozen image backbones and is used in later geometry controls.The frame-token and patch-mean variants test retaining additional feature information; Table 7 reports Accuracy and weighted F1.Source: §A.4 · p. 14, Table 7 · p. 14

A.5 Configuration

  • FlatClip freezes the SigLIP2-NaFlex encoder, renders scalar cortical activity as RGB flatmaps, and trains only the downstream MLP probe.The resting-state setting uses T = 40, following prior works; resting-state inputs are globally z-scored over the selected vertex-by-time matrix.Source: §A.5 · p. 14
  • Feature extraction is performed offline with the frozen encoder, after which cached features allow lightweight downstream MLP training without encoder backpropagation.HCP extraction took 3.96 hours on four A100 GPUs for 40,440 flatmap images, or 2.84 images/s in wall-clock throughput.Source: §A.5 · p. 15

A.6 The influence of frame number

  • 40 frames give the best or tied-best performance on HCP and most ADNI metrics, while ADNI AD/CN saturates around 20 and PPMI shows no consistent gain.Across tasks, additional frames generally improve or stabilize results, but gains are modest under one-token pooling; sparse samples still yield useful subject-level information.Source: §A.6 · p. 16, Table 10 · p. 16

B Details of NSD Visual-fMRI Experiments

The NSD experiments use stimulus-level fMRI responses to predict COCO labels, with image identities separated across training, validation, and test partitions. The baseline feature pipelines process each stimulus beta volume independently.

B Details of NSD Visual-fMRI Experiments

  • The NSD decoding task predicts an 80-dimensional COCO multi-hot label vector from stimulus-level GLM beta estimates, using disjoint image identities across partitions.For each subject, 8,550 non-shared images are used for training, 450 for validation, and 1,000 shared images for testing.Source: B Details of NSD Visual-fMRI Experiments · p. 16
  • The alignment analysis compares matched-image cosine similarity and paired-random cosine margin in SigLIP2 space across shared stimuli.Scores are averaged across four subjects; significance brackets compare SigLIP2-NSDgeneral with other methods using paired Wilcoxon signed-rank tests.Source: Figure 4 · p. 17

B.1 GLM for Baselines models

  • SwiFT and NeuroSTORM independently encode each NSD beta volume, while Omni-fMRI repeats a beta volume across its 40 input channels before encoding.Source: §B.1 · p. 17

B.2 Supplementary Analysis of Image–fMRI Feature Alignment

This analysis measures how closely fMRI-derived features align with SigLIP2 image-feature geometry on NSD shared images. SigLIP2-NSDgeneral leads across the three metrics, and task-active cortex aligns more closely than a broader visual mask.

B.2 Supplementary Analysis of Image–fMRI Feature Alignment

  • The alignment analysis fits subject- and backbone-specific ridge maps for paired cosine and margin, while Local-RDM compares distance profiles directly.Evaluation uses 100 shared images sampled with seed 42; preprocessing and PCA are fitted on mapping-training data, with ridge α = 100.Source: §B.2 · p. 17
  • On stimulus-level image–fMRI feature alignment in SigLIP2 space, SigLIP2-NSDgeneral was strongest across all three metrics, improving over NeuroSTORM by 0.083 Local-RDM, 0.088 paired cosine, and 0.079 paired-random margin.The analysis used a randomly sampled subset of NSD shared1000, averaging values over four subjects to leave one value per image.Source: §B.2 · p. 17, Table 11 · p. 18
  • NSD task-active cortex yields tighter image-feature geometry alignment than a broader visual mask, complementing COCO80 classification.Paired cosine compares aligned fMRI-derived features with matched image features; the paired-random margin subtracts mismatched-image cosine.Source: §B.2 · p. 18

B.3 Ablation and Detailed Results

  • Table 9 distinguishes pooled representations from starred rows retaining all 256 adaptively pooled feature tokens to test representation sensitivity.The passage identifies the table’s comparison but provides no cell values.Source: §B.3 · p. 18

C Details of Geometry perturbation Experiments

The geometry controls vary cortical layout while holding the frozen encoder and probe protocol fixed, then compare regional time-series and voxel-volume reference inputs. The controls distinguish preserved local patterns from disrupted cortical organization.

C.1 Geometry-control perturbations.

  • All HCP perturbation conditions use the same subjects, splits, 40-frame inputs, frozen encoder, and probe protocol, with an independently trained probe per condition.The protocol is designed to make input-layout changes the primary difference within each control family.Source: §C.1 · p. 18
  • Hemisphere swapping disrupts laterality, while within-hemisphere shuffling preserves hemisphere identity but rearranges cortical regions; phase randomization preserves silhouette and intensity distribution.The phase control’s histogram matching can change Fourier amplitudes, so frequency matching is approximate.Source: §C.1 · pp. 18–19
  • Block-shuffling preserves local 16 × 16 activation patterns but scrambles global topology; cortical-pixel permutation also disrupts local neighborhoods and location-signal correspondence.The same block or pixel permutation is applied consistently across subjects and frames.Source: §C.1 · p. 19
  • FC heatmaps retain subject-level connectivity but remove surface geometry and frame-wise dynamics; random images retain only the image interface, and random weights remove pretrained features.The random-initialized encoder condition keeps real flatmaps and the downstream MLP.Source: §C.1 · p. 19
  • The frozen DINOv2 geometry-control ablation shows the same qualitative trend as the main SigLIP2 experiment.Table 12 reports HCP sex-classification results using one global token per frame and independently trained probes.Source: §C.1 · p. 19, Table 12 · p. 20

C.2 Details of ROI and voxel reference models

  • The reference comparison includes a 450-ROI time-series model and native voxel models using standard or adaptive patching with a ViT-Small backbone.Voxel inputs retain voxel-wise signals, with outside-mask voxels zeroed and voxel-wise temporal z-scoring; adaptive patch allocation follows Omni-fMRI.Source: §C.2 · pp. 19–20
  • ROI processing averages voxel signals within each region, temporally z-scores them, and samples 40 frames, removing within-ROI spatial geometry while retaining regional dynamics.The ROI and voxel reference models use subject-level train/validation/test partitions, so frames from one subject are not split across sets.Source: §C.2 · p. 20

D Additional Evaluation Protocols and Results

The additional evaluations test FlatClip against matched NSD cortical-region baselines and document protocols for confound adjustment, alternative windows, readouts, and other input formats. Under both tested masks, FlatClip has higher weighted F1 than the matched ROI-linear and small-MLP baselines.

D.1 Native Image Dimensions; D.2 Surface-Model Implementations

  • The supplementary implementations include CortexMAE-F surface features and SiT-style parcel-token and vertex-level spherical-patching controls; Table 13 lists native flatmap dimensions.CortexMAE-F pretraining includes HCP-YA, while ADNI evaluates transfer to a different cohort.Source: §D.2 · pp. 20–21, §D.1 · p. 20, Table 13 · p. 20

D.3 Measured-Confound Sensitivity

  • Covariate encodings and regression are fit on training data only, and adjusted features use those coefficients before training the classification probe.For MCI/CN with age, sex, and site adjustment, all three seeds produce the same number of correct test predictions, yielding zero accuracy standard deviation but nonzero weighted-F1 standard deviation.Source: §D.3 · p. 21, Table 14 · p. 21

D.4 Alternative Forty-Frame Windows

  • Alternative-window tests compare TR0–39 with TR98–137, while the AD/CN later-window split uses 243/34/72 training, validation, and test subjects.The later window is constrained by one training acquisition containing only 97 TRs; validation and test membership remain unchanged.Source: §D.4 · p. 21, Table 15 · p. 22

D.5 Class Distribution and Probe Sensitivity; D.10 Labeled-Subject Scaling

  • HCP probe evaluations use a 1,000-subject cohort; the linear-probe experiment changes the readout, while labeled-subject scaling uses nested 10%, 25%, 50%, and 100% training subsets.The linear-probe table reports accuracy and weighted F1; the scaling protocol keeps validation and test subjects fixed.Source: §D.5 · p. 22, Table 17 · p. 22, Table 16 · p. 22

D.6 Retrained and Fixed Readouts

  • Retrained controls fit a new probe per input condition, whereas fixed-readout controls apply the real-flatmap-trained classifier and standardizer unchanged to perturbed features.Table 18 reports both protocols for the same three perturbations.Source: §D.6 · p. 22, Table 18 · p. 23

D.7 Matched Cortical-Region Baselines on NSD

  • FlatClip has higher weighted F1 than ROI-linear and small-MLP under both NSD masks, and higher mAP on Visual; on NSDgeneral, ROI-linear and FlatClip have comparable mAP.The comparison averages results across four subjects.Source: §D.7 · p. 22, Table 19 · p. 23

D.8 Direct Acquired-Volume Renderings

  • Direct acquired-volume protocols compare HCP mean projections and slice-wise inputs, and NSD cortical flatmaps with volume projections and independent axial slices.The HCP slice-wise procedure requires one encoder pass per non-empty slice; retained volumes have a median of 77 such slices per TR.Source: §D.8 · p. 23, Table 20 · p. 23, Table 21 · p. 24

D.9 Omni-fMRI with Native Temporal Inputs

  • NSD temporal adaptations encode either continuous 40-TR BOLD windows or concatenated response-centered segments, using 10,000 images and a 7,000/1,000/2,000 split.The temporal-input rows are reported separately from the original beta-map results; labeled-subject scaling is summarized above.Source: §D.9 · pp. 23–24, Table 22 · p. 24

Related papers

Loading 2609.31204v2…