PaperlayerView exact revision on arXiv

Source-linked AI summary

Natural Image Autoencoder-Based fMRI Representations for Trait and State Prediction

Juhyeon Park, Yeonwoo Kim, Peter Yongho Kim, Yansen Wang, Mingqing Xiao, Dongqi Han, Dongsheng Li, Taesup Moon

arXiv:2609.34167v2cs.CV

TL;DR

Large-scale fMRI foundation models perform well but require costly data and computation, and it remains unclear how far frozen representations learned outside fMRI can go. FReD uses a frozen natural-image-pretrained DCAE with readouts tailored to trait and state prediction. Its results show competitive performance across both task types without fMRI-specific representation pre-training, supporting frozen natural-image features as a baseline for measuring its added value.

  • Problem

    Although fMRI foundation models perform well, their data and computation costs are substantial, and the performance of frozen natural-image representations against them remains unclear.

    Source: §1 · p. 1

  • Method

    FReD uses a frozen DCAE pre-trained exclusively on natural images and adapts its readout to the task, using temporal summaries for traits and a Transformer for state prediction.

    Source: §1 · p. 2, §3 · p. 3, §3.2 · p. 5

  • Results

    Frozen DCAE features with task-matched readouts generally outperform foundation-model representations under linear probing for trait prediction and perform comparably to the strongest evaluated models for state prediction.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    Strong performance on current benchmarks can be achieved without fMRI-specific representation pre-training, making frozen natural-image representations a baseline for evaluating its added value.

    Source: §6 · p. 10

    The Gaussian injection analysis is synthetic and spatially localized, so it assesses linear accessibility rather than downstream ranking.

    Source: §6 · p. 10

Abstract

from arXiv · show

Foundation models pre-trained on large-scale fMRI datasets have shown strong downstream performance, but at substantial data and computation cost. To investigate how much fMRI-specific pre-training is actually needed for such performance, we introduce FReD, which derives fMRI representations from a frozen Deep Compression AutoEncoder (DCAE) pre-trained exclusively on natural images and pairs them with a task specific readout. For trait prediction, FReD summarizes frame-wise representations by their temporal mean and log-standard deviation and applies linear probing, with late fusion across two normalization schemes. For state prediction, it represents each frame as a single token and models temporal dependencies with a shallow Transformer. Across four resting-state datasets spanning six trait-prediction targets, linear probes on frozen DCAE features generally outperform those on fMRI foundation model representations and remain competitive with fully fine-tuned fMRI foundation models. On three task-fMRI state-prediction tasks, a temporal readout on DCAE features performs comparably to the strongest foundation models evaluated. A Gaussian injection analysis further shows that localized signal changes are recovered more accurately from the frozen DCAE features than from the evaluated foundation-model representations. Together, these results show that strong performance on current fMRI benchmarks is possible without fMRI-specific representation pre-training, making frozen natural-image features as a useful baseline for assessing its added value.

1 INTRODUCTION

Large-scale fMRI foundation models perform well but require costly data collection, preprocessing, and computation, while the value of frozen natural-image representations for fMRI prediction remains unclear. FReD uses a frozen natural-image-pretrained encoder with readouts matched to trait and state prediction, and reports competitive performance across both task types.

1 INTRODUCTION

  • FReD asks whether competitive fMRI prediction is possible without fMRI-specific pre-training, addressing a gap left by prior comparisons focused mainly on models trained from scratch.The prior approach also used volumetric data and a uniform tokenization and temporal-modeling recipe across tasks.Source: §1 · pp. 1–2
  • For trait prediction, temporal mean and log-standard-deviation summaries with linear probes generally outperform foundation-model representations, and late fusion remains competitive with fully fine-tuned models on most tasks.These results use resting-state fMRI and two complementary normalization schemes.Source: §1 · p. 2
  • For task-fMRI state prediction, a shallow Transformer over one token per frame performs comparably to the strongest evaluated foundation models.It also outperforms TABLeT with a 27-fold shorter attention sequence and 40% fewer trainable parameters.Source: §1 · p. 2
  • Localized injected perturbations are recovered more accurately from frozen DCAE features than from the evaluated foundation-model representations.The analysis also finds that a substantial fraction of fMRI variation is captured by principal directions of the DCAE latent space.Source: §1 · p. 2

2 PRELIMINARIES

FReD uses volumetric and cortical flat-map fMRI, which retain spatial structure, and evaluates two normalization schemes. The preliminaries also distinguish model families by their input data types and prediction tasks by whether labels vary within a subject.

2.1 FMRI DATA TYPES & NORMALIZATION SCHEMES

  • FReD operates on volume and cortical flat-map data, which retain spatial structure, while parcellation averages voxel signals within predefined regions of interest.The DCAE-based approach uses the first two data types; parcellation appears only among the baselines.Source: §2.1 · p. 2
  • The experiments use global normalization or coordinate–frame normalization, which sequentially applies coordinate-wise standardization and then frame-wise standardization.Global normalization uses shared statistics across the sequence and applies one transformation across frames and spatial coordinates.Source: §2.1 · p. 2

2.2 FMRI FOUNDATION MODELS & DOWNSTREAM TASKS

  • Existing foundation models are grouped into volume-, cortical flat-map-, and parcellation-based approaches, and are generally pre-trained on large-scale neuroimaging datasets with self-supervised objectives.Examples include SwiFT, NeuroSTORM, and Omni-fMRI for volumes; CortexMAE for flat maps; and BrainMASS, Brain-JEPA, BrainHarmonix-F, and Brain-DiT for parcellations.Source: §2.2 · p. 3
  • Trait prediction targets subject-level attributes fixed over time, whereas state prediction targets cognitive or stimulus-driven labels that change within a subject.The text expects stable between-subject representations to inform trait tasks and temporal variation to matter for state tasks.Source: §2.2 · p. 3

3 FRED

FReD combines frozen DCAE frame representations from volumetric or cortical flat-map fMRI with readouts tailored to the downstream task. Trait prediction uses temporal statistics and linear probing, whereas state prediction models temporal dependencies with a Transformer.

3 FRED

  • FReD has two stages: a frozen natural-image-pretrained DCAE constructs frame representations, followed by a task-dependent readout.The framework operates on volumetric and cortical flat-map fMRI.Source: §3 · p. 3

3.2 TASK-DEPENDENT READOUT

  • For state prediction, FReD projects coordinate–frame-normalized frame representations into embeddings and uses a four-layer Transformer to model temporal dependencies.Each frame becomes one token, and the [CLS] output is used for prediction; the Transformer models temporal rather than joint spatial and temporal relationships.Source: §3 · p. 3, §3.2 · p. 5

3.1 DCAE-BASED FRAME REPRESENTATION

  • For volumes, FReD encodes slices from three anatomical axes and groups slice latents before flattening and concatenating them into one frame representation.Grouping reduces dimensionality while retaining spatial granularity; the number of groups is 3 for trait prediction and 24 for state prediction.Source: §3.1 · p. 4
  • For cortical flat maps, FReD encodes the two hemisphere maps separately and flattens and concatenates their latents into a frame representation.The resulting representation has 4,032 dimensions and is used for both trait and state prediction.Source: §3.1 · p. 4

3.2 TASK-DEPENDENT READOUT

  • For trait prediction, FReD summarizes frame representations with element-wise temporal mean and log-standard deviation, fits a linear predictor, and late-fuses predictions from two normalization schemes.The fusion weight is selected using the validation set.Source: §3.2 · pp. 4–5

4 EXPERIMENTAL RESULTS

Across trait and state prediction, the experiments test frozen DCAE representations against fMRI foundation models using task-specific readouts. FReD performs strongly across the evaluated tasks, while the state results highlight the importance of normalization and temporal modeling.

4.1 EXPERIMENTAL SETTINGS

  • The evaluation covers six trait targets across four resting-state datasets and three state-prediction tasks, using subject-level splits and specified cross-validation or train/validation/test protocols.Trait evaluations use 500 sampled 85/15 train–test splits for linear probing; other experiments use 70/15/15 splits or fixed splits with three seeds.Source: §4.1 · p. 5, §4.2 · p. 5

4.2 TRAIT PREDICTION

  • FReD-Trait outperforms evaluated foundation-model baselines on five of six trait tasks under linear evaluation, with comparable performance on ADNI.The comparison is shown over 500 random splits and reports AUC/ρ.Source: §4.2 · p. 5, Figure 2 · p. 6
  • For trait prediction, late fusion of predictions from two normalization schemes generally improves performance, with selected fusion weights varying across tasks.Dimension reduction leaves FReD-Trait performance robust.Source: §4.2 · p. 6
  • Fully fine-tuned foundation models remain broadly competitive with FReD-Trait rather than consistently surpassing it, and fine-tuning does not consistently improve on linear probing.Table 1 compares fully fine-tuned foundation models with linearly probed FReD-Trait.Source: §4.2 · p. 6, Table 1 · p. 6

4.3 STATE PREDICTION

  • On HBN-Movie, coordinate-frame-normalized inputs improve the DCAE-based models, motivating their exclusive use in FReD-State.Volume-based foundation models perform only slightly above chance on globally normalized inputs; the normalization effect is more pronounced for TABLeT and FReD-Trait.Source: §4.3 · pp. 6–7
  • FReD-State substantially outperforms FReD-Trait on HBN-Movie and becomes competitive with the strongest foundation models by modeling temporal interactions among frame representations.FReD-Trait’s temporal mean and log-standard-deviation aggregation discards frame order.Source: §4.3 · p. 7
  • FReD-State outperforms Omni-fMRI and TABLeT on HCP-Task and NSD, performs comparably to CortexMAE, and matches the strongest flat-map baseline with XF.XF performs better on HCP-Task and comparably on NSD.Source: §4.3 · p. 7, Table 2 · p. 7

5 ANALYSES

Analyses show that natural-image DCAE latents capture substantial fMRI variation and retain linearly accessible localized changes. Readout and representation choices matter differently across trait and state prediction.

5.1 DOES THE DCAE LATENT SPACE ACCOMMODATE FMRI VARIATION?

  • At 60% of the ImageNet PCA basis, retained directions explain roughly 0.72–0.85 of variance across fMRI representations, with flat maps closer to their ImageNet reference than volume slices.At small basis fractions, each fMRI representation falls below its ImageNet reference.Source: §5.1 · p. 8

5.2 ARE LOCALIZED SIGNAL CHANGES ACCESSIBLE FROM DCAE REPRESENTATIONS?

  • Both FReD variants outperform every evaluated foundation-model representation at each injected-signal amplitude, though all remain near chance for the weakest perturbation.The probe measures linear accessibility of localized perturbations, not general representation quality.Source: §5.2 · p. 8

5.3 TASK-DEPENDENT NORMALIZATION AND READOUT

  • HBN-Movie is far more sensitive to normalization choice than HBN-Sex, which the authors attribute to prediction task rather than dataset.The comparison uses the same HBN data distribution for movie-state and sex-trait prediction.Source: §5.3 · p. 8, Table 3 · p. 8
  • FReD-State does not improve over FReD-Trait on trait prediction, suggesting explicit temporal modeling matters for state targets while simple temporal aggregation suffices for traits.Table 4 reports FReD-State performance on trait prediction tasks.Source: §5.3 · p. 9, Table 4 · p. 9

5.4 COMPONENT-WISE ABLATIONS

  • Replacing log-standard-deviation features with zeros generally reduces trait-prediction performance, although the effect varies across tasks.The ablation preserves input dimensionality and parameter count; Figure 5(a) reports AUC or ρ differences against the zero-vector replacement.Source: §5.4 · p. 9, Figure 5 · p. 9
  • Trait prediction stays stable even at one group per axis, whereas state prediction degrades with too few groups but remains stable across a moderate range.The number of groups G controls retained spatial granularity in frame representations.Source: §5.4 · p. 9, Figure 5 · p. 9
  • Replacing TABLeT's tokenization with one group-averaged DCAE token per frame improves performance; further gains and fewer parameters follow jointly from remaining architectural and regularization changes.The ablation uses XV; the authors do not attribute the second-stage gain to any single change.Source: §5.4 · p. 9

6 CONCLUSION AND LIMITATIONS

The conclusion argues that frozen natural-image DCAE features with task-matched readouts achieve strong performance on current fMRI benchmarks without fMRI-specific representation pre-training. The authors also identify boundaries in their synthetic perturbation analysis and leave encoder adaptation unexplored.

6 CONCLUSION AND LIMITATIONS

  • Frozen natural-image DCAE features with task-matched readouts achieve competitive trait prediction and state prediction comparable to the strongest evaluated fMRI foundation models.For state prediction, the shallow Transformer uses a smaller readout and shorter attention sequence than TABLeT.Source: §6 · pp. 9–10
  • The Gaussian injection analysis is synthetic and spatially localized, so it supports conclusions about linear accessibility rather than downstream ranking.The authors also leave unresolved when each data type should be preferred and whether adapting the fixed encoder could improve performance.Source: §6 · p. 10

A.1 UTILIZING NATURAL IMAGE PRIOR FOR MEDICAL IMAGING

  • Prior work has transferred natural-image representations to medical imaging, including frozen slice-based features and lightweight adaptation, and TABLeT uses a natural-image autoencoder to tokenize fMRI volumes.The cited examples are Raptor, AnyMC3D, and TABLeT.Source: §A.1 · p. 13

B.1 DATASETS

The appendix identifies datasets used for trait and state prediction, describes preprocessing, and specifies split construction and evaluation protocols.

B.1 DATASETS

  • Table 6 summarizes trait- and state-prediction datasets, including post-preprocessing subject and sample counts, sample duration, task labels, and majority-class proportions where applicable.Trait datasets have one sample per subject, while state datasets have multiple samples per subject.Source: Table 6 · p. 13, §B.1 · p. 13
  • Preprocessing uses fMRIPrep for ADHD-200, ABIDE-II, ADNI, and HBN, with additional nuisance regression for the first three; other datasets use minimally processed or BrainMarks-provided data resampled to model-specific TRs.Splits are subject-level; linear-probing experiments use 500 sampled 85/15 train–test splits, while other experiments use specified 70/15/15 splits or fixed splits and seeds.Source: §B.1 · p. 14

B.2 TRAINING AND EVALUATION

The appendix documents fixed encoders, evaluation and fusion procedures, shared training settings, model-specific hyperparameters, and details of the PCA analysis.

B.2 TRAINING AND EVALUATION

  • All experiments use the unmodified, frozen dc-ae-f32c32-in-1.0 checkpoint; evaluation averages outputs over windows for XV- and XF-based models and pads short inputs with per-coordinate means.For linear probing, the protocol uses window-averaged embeddings.Source: §B.2 · p. 14
  • Late-fusion weight α is selected without test data, using out-of-fold training predictions for 85/15 splits or validation data for 70/15/15 splits.Selection uses a uniform grid, with SLSQP refinement for smooth criteria and the best grid point for rank-based criteria.Source: §B.2 · p. 14
  • Shared neural-network settings include AdamW, cosine learning-rate decay, weight decay 10^-2, fp16, task-appropriate losses and hyperparameter selection, and early stopping except for SwiFT.Hyperparameters are chosen on validation data using task-specific criteria.Source: §B.2 · pp. 14–15
  • Brain-JEPA, BrainHarmonix-F, Brain-DiT, SwiFT, and NeuroSTORM use model-specific learning-rate ranges, batch sizes, and epoch counts; SwiFT and NeuroSTORM also specify window size and TR.The listed schedules and batch sizes differ across the five models.Source: Brain-JEPA · p. 15, BrainHarmonix-F · p. 15, Brain-DiT · p. 15
  • Reported configurations vary by task and dataset: one uses dataset-specific learning-rate ranges, batch sizes 8/16, 30 epochs, window size 40, and TR 0.8 s; another uses window 16 and TR 1 s.A configuration uses scikit-learn’s default search space and TR 0.8 s for trait prediction and HBN.Source: Omni-fMRI · p. 15, CortexMAE · p. 15, FReD-Trait · p. 16

B.3 PCA ANALYSIS DETAILS

  • The PCA analysis extracts latents from 100,000 ImageNet test images and HCP-A, uses the first fMRI frame, and evaluates ImageNet PCA on held-out samples after fitting on 80,000 latents.HCP-A volume slices are evaluated by axis with median and range reported, and flat-map hemispheres are treated as independent samples.Source: §B.3 · p. 16

B.4 GAUSSIAN INJECTION EXPERIMENTS

The appendix examines whether FReD-Trait’s results persist at matched feature dimensionality, how normalization branches contribute across tasks, and how partial fine-tuning compares. It also describes the Gaussian-injection protocol for testing recovery of localized signal changes.

B.4 GAUSSIAN INJECTION EXPERIMENTS

  • The injected perturbation is a cortical Gaussian profile scaled by percent signal change and modulated by a canonical hemodynamic response.The Gaussian tails are restricted to cortex, and the response is normalized to unit peak.Source: §B.4 · pp. 16–17
  • After injection, global mean and standard deviation are matched before coordinate–frame normalization; frozen foundation-model features include repeat and timeseries conditions.FReD uses the peak frame, and its features are reduced to 768 dimensions by PCA fitted on training subjects.Source: §B.4 · p. 17
  • The Gaussian-injection experiment uses four shared centers and subject-disjoint splits to probe recovery of injected locations.It uses all HCP-A subjects, a 16-second window sampled at TR = 0.8 seconds, and 4,276 balanced samples scored by accuracy against a 25% chance level.Source: §B.4 · pp. 16–17

C.1 DIMENSION-REDUCED LINEAR PROBING

  • After reduction to 768 dimensions, FReD-Trait remains competitive with fully fine-tuned foundation models, though performance falls to baseline levels on ABIDE-II and HCP-A intelligence.It remains significantly better than the strongest baseline on ADHD-200, HCP-A sex, and HCP-A age; the clinical datasets use random projection, while HCP-A uses PCA.Source: §C.1 · pp. 17–18, Figure 6 · p. 17

C.2 LATE FUSION WEIGHT DISTRIBUTION

  • Late-fusion weights vary by task, while their means rarely approach either extreme, indicating contributions from both normalization schemes.ASD classification and intelligence prediction favor coordinate–frame normalization; ADHD, ADNI diagnosis, HCP-A sex, and HCP-A age give at least as much weight to global normalization.Source: §C.2 · p. 18, Figure 7 · p. 18

C.3 PARTIAL FINE-TUNING

  • Partial fine-tuning does not consistently outperform FReD-Trait’s linear probes on frozen features.Foundation models are partially fine-tuned while FReD-Trait uses frozen-feature linear probing; results span 12 clinical-task splits and four HCP-A splits.Source: Table 7 · p. 18, §C.3 · p. 19

C.4 LINEAR PROBING RESULTS WITH MATCHED SPLITS FOR FOUNDATION MODELS

Using identical data partitions to compare linear probing and full fine-tuning, the authors find that neither adaptation strategy reliably surpasses FReD-Trait across tasks.

C.4 LINEAR PROBING RESULTS WITH MATCHED SPLITS FOR FOUNDATION MODELS

  • Fine-tuning is not uniformly better than linear probing, and neither protocol brings foundation models past FReD-Trait on most tasks.Linear probing leads on several specified model-task pairs, while fine-tuning helps most consistently on HCP-A sex and age.Source: §C.4 · p. 19, Table 8 · p. 19

C.5 FULL RESULTS OF FIG. 2 AND FIG. 3

This section points readers to full-result tables for Figures 2 and 3; the supplied passages describe table scope and notation but do not report their values.

C.5 FULL RESULTS OF FIG. 2 AND FIG. 3

  • Table 9 contains the full results of Figure 2, with bold and underlined values marking best and second-best performance for each metric.The supplied caption does not include the table’s result values.Source: Table 9 · p. 19, §C.5 · p. 19
  • Table 10 contains the full results of Figure 3 and distinguishes linear probing, fine-tuning, and training from scratch.Its caption says highlighted rows are linear-probing results and bold or underlined values mark best or second-best performance for each metric.Source: §C.5 · p. 19, Table 10 · p. 20

D ARCHITECTURAL COMPARISON BETWEEN TABLET AND FRED-STATE

FReD-State changes how DCAE latents are tokenized and modeled relative to TABLeT. The combined changes reduce trainable parameters and improve accuracy on both reported tasks.

D ARCHITECTURAL COMPARISON BETWEEN TABLET AND FRED-STATE

  • FReD-State uses one token per frame and a sequence of T + 1 tokens, rather than TABLeT’s 27 tokens per frame and 27T + 1 sequence.FReD-State performs spatial mixing in the projection, leaving the Transformer to model temporal structure.Source: D ARCHITECTURAL COMPARISON BETWEEN… · p. 20, Table 11 · p. 21
  • FReD-State uses four Transformer blocks instead of twelve and replaces LayerNorm and a ReLU MLP with RMSNorm and SwiGLU blocks.Model and feed-forward widths are unchanged; the described backbone also uses Q/K normalization and soft-capped attention logits.Source: D ARCHITECTURAL COMPARISON BETWEEN… · pp. 20–21
  • Training drops frame tokens with probability 0.5 and uses stochastic depth whose drop-path probability increases from 0 to 0.1 across four blocks.The [CLS] token is never dropped.Source: D ARCHITECTURAL COMPARISON BETWEEN… · p. 21
  • The combined changes reduce trainable parameters from 129.5M to 78.2M, a 39.6% reduction, while improving accuracy on HCP-Task and NSD.The architectural comparison holds the DCAE latent tensor and downstream loss constant.Source: D ARCHITECTURAL COMPARISON BETWEEN… · p. 21, Table 11 · p. 21

Related papers

Loading 2609.34167v2…