PaperlayerView exact revision on arXiv

Source-linked AI summary

A Scaling Study for fMRI Foundation Models

Wenhao Ye, Xuanye Pan, Junfeng Xia, Junxiang Zhang, Mo Wang, Quanying Liu

arXiv:2609.27232v1cs.LG

TL;DR

The paper addresses uncertainty about how data size, model size, and training progress jointly shape fMRI foundation-model performance. It studies these variables under a fixed pretraining and evaluation protocol, then uses ID performance to select models for fixed compute budgets. Data scaling is more consistent across tasks at matched compute, and the selected larger model has the highest average OOD performance among compared models.

  • Problem

    How data size, model size, and training progress jointly shape fMRI downstream performance remains unclear, since prior studies largely examined these factors separately or only at training endpoints.

    Source: §1 · p. 1

  • Method

    The study holds the pretraining framework and downstream protocol fixed, varies data size, model size, and training progress, and selects models at two compute budgets using ID performance before OOD evaluation.

    Source: §1 · p. 2

  • Results

    At matched compute, pretraining-data increases provide more consistent gains across tasks; the ID-guided 10.42M model achieves the highest average OOD performance among compared models.

    Source: §5.3 · p. 6, §6 · p. 7

  • Takeaways & Limitations

    Compute alone does not characterize performance: the data–model–training combination matters, and the selected allocation is not established as optimal for every OOD task or beyond evaluated candidates.

    Source: §7 · p. 7

    The main limitation is extrapolating the fitted ID response beyond the observed candidates; the selected combination is not shown to be optimal for every OOD task.

    Source: §7 · p. 7

Abstract

from arXiv · show

Scaling laws have guided large-model development in computer vision and natural language processing, but the relationships among data, model size, and compute remain unclear for functional magnetic resonance imaging (fMRI) foundation models. Here, we conduct a controlled empirical study using pretraining data from more than 200 source datasets and over 10,000 GPU-hours of experiments. Holding the pretraining framework and downstream protocol fixed, we vary pretraining data size, model size, and training duration. Downstream performance generally improves with compute, yet models using similar compute can perform substantially differently. Additional pretraining data bring larger gains at larger model sizes, suggesting that data and model size should be scaled together. At matched compute, increasing pretraining data benefits more tasks than increasing model size, although the pattern varies across tasks. We then use in-distribution (ID) downstream performance to select the combination of pretraining data size, model size, and training duration at two fixed compute budgets. The resulting models are locked before out-of-distribution (OOD) evaluation. They achieve the highest average performance across the evaluated OOD tasks among the compared fMRI foundation models while using less pretraining compute. Overall, our results show that compute alone does not characterize fMRI scaling: performance depends on how pretraining data, model size, and training duration are combined.

1 Introduction

The paper asks how fMRI pretraining data size, model size, and training progress jointly shape downstream performance, since similar compute can yield different benefits. It uses a controlled study to compare these choices and reports that data gains vary with model size and are more consistent across tasks than model-size gains at matched compute.

1 Introduction

  • Prior fMRI studies largely examined data size, model size, or training endpoints separately, leaving their joint relationship with training progress and matched-compute allocation unclear.The paper frames this as important because fMRI pretraining depends on finite human recordings and downstream labeled datasets are often small.Source: §1 · p. 1
  • The controlled study holds the pretraining framework and downstream protocol fixed while varying data size, model size, and training progress across configurations.Experiments use data from more than 200 source datasets and compare frozen representations on 12 downstream tasks.Source: §1 · p. 2
  • The paper uses ID downstream performance to select data, model, and training-duration combinations at two fixed compute budgets, then locks the selections before OOD evaluation.The introduction describes comparison with released fMRI foundation models, but does not report the OOD outcome here.Source: §1 · p. 2
  • At matched compute, additional pretraining data provide more consistent gains across tasks, while the benefit of increasing model size is more task dependent.The study also reports that data gains are larger for larger models.Source: §1 · p. 2

2 Related Work

Prior fMRI scaling work studied individual axes or limited downstream trends, but did not jointly characterize data, model size, training progress, and matched-compute allocation. The paper uses NeuroJEPA as a fixed pretraining method for that controlled comparison.

2 Related Work

  • fMRI data expansion commonly adds participants, sites, scanners, and acquisition protocols, so the data axis changes evidence composition as well as dataset size.The paper contrasts acquisition-constrained fMRI pretraining with language and vision scaling work and emphasizes downstream performance across label-limited tasks.Source: §2 · p. 2
  • Earlier fMRI studies reported favorable trends on individual scaling axes, while CortexMAE did not compare alternative data–model allocations at matched total compute.The paper instead relates downstream performance to nested multistudy data sizes, model size, and training progress.Source: §2 · p. 2
  • NeuroJEPA fixes a simple LeJEPA-style objective that aligns global and local views and applies SIGReg, rather than adding several components used by other fMRI objectives.It uses no EMA target network, masked reconstruction head, semantic tokenizer, or training curriculum; the paper does not claim universal optimality.Source: §2 · pp. 2–3, Figure 2 · p. 3

3 Fixed Pretraining Method

The paper fixes NeuroJEPA as the representation learner for all scaling experiments. Its shared encoder and projection head map multiple views of each ROI-time segment into embeddings that are aligned and regularized.

3 Fixed Pretraining Method

  • Each minibatch segment is an ROI-by-time matrix, augmented into eight views: two global views retaining broad context and six local views retaining less context.The same input segment supplies all views; experiments use 100 cortical ROIs.Source: §3 · p. 3
  • A shared Transformer encoder maps each view to a pooled feature vector, and a shared projection head maps that vector to an embedding.Both components are shared across views.Source: §3 · p. 3
  • The objective aligns global and local embeddings to the mean global embedding and applies SIGReg to each view batch to discourage collapsed or concentrated representations.The method uses the LeJEPA SIGReg functional unchanged; alignment preserves information shared across views while SIGReg encourages well-spread representations for linear probes.Source: §3 · p. 3
  • The paper does not claim that this objective preserves every downstream variable or is optimal for every fMRI task.It is held fixed so the scaling variables can be studied separately.Source: §3 · p. 3

4 Controlled Scaling Setup

The study holds the pretraining and evaluation framework fixed while varying data size, model size, and training progress. It evaluates nested recording sets and model sizes on 12 frozen linear-probe tasks under a fixed participant-level protocol.

4 Controlled Scaling Setup

  • The study holds the NeuroJEPA objective, ROI-time input, view construction, architecture and optimizer families, and downstream protocol fixed while varying data size, model size, and training progress.This controlled setup isolates the planned scaling variables.Source: §4 · p. 3

4.1 Data, Models, and Compute

  • The pretraining data come from more than 200 source datasets, and the nested grid includes 2,585, 3,810, 6,533, 11,673, 32,414, and 63,401 recordings; scaling analyses cover model sizes from approximately 2 million encoder parameters.Data levels preserve source proportions up to rounding. Model size counts trainable encoder parameters and excludes the projection head.Source: §4.1 · p. 3
  • Compute is estimated in EFLOP from dominant dense-matrix operations and optimizer steps, while training progress is completed steps divided by planned steps.Equal progress fractions do not imply equal compute across runs; epoch-equivalent exposure instead counts recording presentations relative to available recordings.Source: §4.1 · p. 4

4.2 Evaluation Protocol

  • The evaluation uses 12 demographic and clinical classification or regression tasks, with frozen encoders and linear probes under fixed participant-level splits.Participants remain within one split, pretraining excludes downstream validation and test participants, and classification and regression scores use macro-F1 and Pearson correlation, respectively.Source: §4.2 · p. 4

5 How Should fMRI Pretraining Scale?

Across the evaluated grid, downstream error generally falls with accumulated compute, but compute alone does not determine performance. Larger models show larger fitted gains from additional data, while at matched compute more tasks favor data scaling than model scaling.

5 How Should fMRI Pretraining Scale?

  • The analysis describes the observed scaling grid through training progress, joint data–model relationships, and comparisons at similar estimated compute, rather than fitting or extrapolating a universal scaling law.Source: §5 · p. 4

5.1 Compute Helps, but Scaling Choices Matter

  • Across classification, regression, ID, and OOD task-group summaries, linear-probe error generally decreases as estimated compute accumulates, but nearby-compute configurations can differ or change rank.Data size, model size, training progress, and estimated compute change together, so this pattern does not isolate any factor’s independent effect.Source: §5.1 · p. 4, Figure 3 · p. 5

5.2 Data Gains Depend on Model Size

  • The fitted gains from additional pretraining data are larger for larger models, with positive βD×N throughout measured training stages and a larger value at training’s end than midpoint.The association applies to the evaluated grid and does not establish a causal or strictly monotonic relationship.Source: §5.2 · p. 5, Figure 3 · p. 5

5.3 Most Tasks Favor Data at Matched Compute

  • Eight of twelve tasks favor more pretraining data at matched compute, four are unresolved, and none clearly favors a larger model.This describes the evaluated grid, not a universal allocation rule.Source: Figure 4 · p. 5, §5.3 · p. 6

6 OOD Evaluation of ID-Guided Models

The paper selects observed data-size, model-size, and training-duration combinations using ID performance at two compute budgets, then locks them before OOD evaluation. The 10.42M-parameter model achieves the highest average OOD performance among the compared models, with both selected models showing a favorable performance–compute trade-off.

ID-guided model selection

  • At two prespecified compute budgets, the procedure selects an observed data-size, model-size, and training-duration combination with the best predicted ID performance, then locks it before OOD evaluation.The response is fit to mean linear-probing performance on five ID tasks and ranks configurations only within the measured range.Source: §6 · p. 6

Evaluation protocol

  • OOD evaluation uses six tasks, frozen encoders, the same linear-probe search, and subject-level 60:20:20 splits shared across baselines and selected models.The evaluation uses five runs with seed-specific splits; classification uses logistic regression and regression uses ridge regression.Source: §6 · p. 6

Baseline and compute accounting

  • Pretraining compute for baselines is reconstructed from reported training details, code, configuration files, and checkpoint metadata, so the estimates carry assumptions and available ranges.Source: §6 · p. 6

OOD results

  • The 10.42M model has the highest average OOD performance among compared models, and each selected model beats the closest-compute baseline on five of six task-wise metrics.The 10.42M model has the best mean on at least one reported metric in five of six tasks; the 2.65M model ranks second on at least one metric in four tasks.Source: Figure 5 · p. 6, §6 · p. 7

7 Discussion and Limitations

Compute alone does not determine downstream performance: outcomes depend on the combination of pretraining data, model size, and training progress. ID-guided choices at two compute budgets performed strongly on OOD tasks, but their applicability is bounded by the evaluated configurations and method.

7 Discussion and Limitations

  • At similar estimated compute, configurations can remain separated, and the fitted data-size association strengthens with model size and later training.At matched compute, data-size association was larger than model-size association for eight of twelve tasks; the other four were unresolved.Source: §7 · p. 7
  • The larger ID-selected model achieved the highest average OOD performance among compared fMRI foundation models while using less pretraining compute.The two selected models were locked before OOD evaluation; results do not establish that the chosen combination is optimal for every OOD task or beyond evaluated candidates.Source: §7 · p. 7
  • The fitted ID response is bounded by six nested data sizes spanning approximately 24.5-fold, with no substantially larger configuration reserved for prospective testing.A future corpus with new sites, scanners, or acquisition protocols may require refitting the numerical response; the study also uses one fixed method and ROI-time architecture family with frozen linear probes.Source: §7 · p. 7

Supplementary Material for A Scaling Study for fMRI Foundation Models

The supplement records reproduction details, a qualified geometric rationale for SIGReg, view construction, and settings shared across scaling runs.

A Supplementary Methods and Analyses

  • The appendix organizes supplementary material into fixed pretraining and evaluation details, statistical definitions, and ID-guided selection and OOD audits.It follows the order of the main paper.Source: A Supplementary Methods and Analyses · p. 10

Latent Geometry and SIGReg

  • Equal positive eigenvalues minimize the stated coefficient-variance quantity at fixed total variance, but isotropy is not claimed to create task information or improve every task.The geometry argument motivates well-spread representations for unknown linear readouts and label-limited probes; SIGReg discourages collapse and concentration into a few latent directions.Source: Latent Geometry and SIGReg · p. 10

View Construction

  • Each recording yields two broad-context global views and six lower-context local views, which are cropped, resampled to fixed shapes, perturbed, and z-scored over time within each ROI.The complete sampling recipe is in Table 3.Source: View Construction · p. 10

Architecture and Optimization

NeuroJEPA Compute Accounting

The supplement defines an estimated compute accounting, resource totals, downstream evaluation summaries, probe fitting, and the scope of expanded trajectory figures.

NeuroJEPA Compute Accounting

  • Equal parameter counts need not imply equal compute because view shapes, batch size, and completed optimizer steps also enter the estimate.The accounting counts dominant dense matrix multiplications in the encoder and projection head, treats one multiply–accumulate as two floating-point operations, and excludes normalization, activations, positional encoding, and objective statistics.Source: NeuroJEPA Compute Accounting · p. 10

Experimental Resource Use

  • Approximately 10,347 GPU-hours were used overall; this hardware-resource total is distinct from EFLOP estimates used to compare model computation.Pretraining accounts for most recorded GPU-hours, and most experiments ran on NVIDIA A800-SXM4-80GB GPUs.Source: Experimental Resource Use · p. 11

A.2 Data and Downstream Evaluation

  • Cross-task summaries average each task over five probe seeds, weight tasks equally, and use macro-F1 for classification and Pearson correlation for regression, oriented higher-is-better.Accuracy and mean-squared error are additional task-level metrics but do not enter those summaries; only five ID tasks fit the response and select the final models.Source: §A.2 · p. 12

Frozen Linear Probes

  • Probe regularization is selected on validation data, and the training-fitted probe is evaluated on the test partition without refitting on combined training and validation data.Classification uses L2-regularized logistic regression and regression uses ridge regression.Source: Frozen Linear Probes · p. 12

A.3 Evidence for the Scaling Findings

  • Figures 6 and 7 expand Figure 3(a)'s compute trajectories along two resource axes: Figure 6 groups models by pretraining data size, whereas Figure 7 groups them by model size.Source: §A.3 · p. 12

How Data and Model Size Interact During Training

The fitted data–model relationship becomes more favorable with larger models and later training. At matched compute, a sensitivity analysis finds data-size-favored results for six tasks, with the other six unresolved.

How Data and Model Size Interact During Training

  • A positive interaction coefficient means fitted data-size gain is larger for larger models, equivalently fitted model-size gain is larger at greater data size.The analysis fits this relationship at each task and training fraction, using standardized validation scores and standardized log data and model sizes.Source: How Data and Model Size Interact… · p. 12
  • Figure 6 groups trajectories by pretraining data size and compares model sizes as corrected compute increases; lower linear-probing error is better.Source: Figure 6 · p. 13
  • 0.116 endpoint-minus-midpoint contrast (paired task-bootstrap 95% interval [0.006, 0.216]) accompanies a positive relationship rising from 0.123 at schedule fraction 0.50 to 0.239 at 1.00.Source: How Data and Model Size Interact… · p. 12, Table 10 · p. 13

Data versus Model Size at Matched Compute

  • When the lower averaging bound rises from 0.11 to 0.20 EFLOP, six named tasks remain data-size-favored; the other six intervals cross zero, and none becomes model-size-favored.Source: Data versus Model Size at Matched… · p. 13

A.4 ID-Guided Selection under Fixed Compute

The ID-guided procedure selects models under two compute budgets using a fitted response, then audits selected models against eligible OOD candidates. The response accounts for diminishing value from repeated data passes, while baseline compute estimates and the available comparisons have stated limits.

A.4 ID-Guided Selection under Fixed Compute

  • The response fit uses equal-weight mean validation performance across five ID tasks, excluding OOD scores from fitting, compute budgets, and model selection.Selection uses one minus the mean of the five ID scores as ID error, with a trailing five-model median within each run and equal total weight per run.Source: §A.4 · p. 13, Response, Fitting, and Formula… · p. 14

Response, Fitting, and Formula Comparison

  • The fMRI grid lacks a complete single-pass experiment, so the response is a joint within-grid fit rather than a replication of the original single-pass protocol.The earliest evaluated run states occur after 3–154 epoch-equivalent exposures.Source: Response, Fitting, and Formula… · p. 15
  • 33.4% lower RMSE on complete-run holdout was achieved by Eq. 14 versus treating every repeated pass as new data.The repeated-data correction improves all six grouped folds (p = 0.0156); most improvement comes from modeling diminishing value of repeated passes, while adding capacity decay changes RMSE by only 2.9%.Source: Response, Fitting, and Formula… · p. 15

Selected Models

  • At budgets C ≤0.128 and C ≤1.97 EFLOP, the fitted response and observed smoothed ID performance select the same model at each budget.Table 13 identifies the selected models; ties favor lower compute.Source: Selected Models · p. 15, Table 13 · p. 15

Two-Budget OOD Audit

  • The OOD audit compares eligible run-level candidates at each budget using six-task mean scores, rank, and regret; ADNI-MCI results are reported separately.The archived audit differs from the external benchmark aggregate, and neither audit enters model selection; the ADNI-MCI table reports classification percentages across five evaluation seeds.Source: Two-Budget OOD Audit · p. 15, Table 14 · p. 15, Additional ADNI-MCI Results · p. 15

Baseline Compute Reconstruction

  • Baseline compute is reconstructed from reported training details, code, configurations, and checkpoint metadata, with official cumulative counters taking precedence when available.Estimates vary in precision, and unresolved pretraining-cohort overlap is treated as a limitation rather than evidence of compute equivalence.Source: Baseline Compute Reconstruction · pp. 15–16, Table 16 · p. 16

B Baseline Models

  • Compared baselines span ROI time series, functional connectomes, atlas-derived graphs, and multimodal inputs, using their official weights and required preprocessing.BrainLM predicts masked regional activity; BrainMASS aligns augmented connectomes; BrainGFM uses graph learning and prompts; BrainHarmonix fuses structural and functional MRI; Brain-Semantoks models regional dynamics with semantic tokens and self-distillation.Source: B Baseline Models · p. 16

C Dataset Catalogue

The catalogue documents curated resting-state fMRI sources used to assemble the pretraining subsets. Downstream validation and test participants and OOD datasets are excluded from the two final models' pretraining data.

C Dataset Catalogue

  • The catalogue lists 201 curated sources, retaining resting-state fMRI collections and removing task-only datasets.Counts follow dataset-level curation, and a participant may contribute multiple recordings.Source: C Dataset Catalogue · p. 17, Table 17 · p. 17
  • Downstream validation and test participants, along with OOD datasets, are excluded from the two final models' pretraining data.The continued catalogue tables document the source collections.Source: C Dataset Catalogue · p. 17, Table 17 · pp. 18–20

Related papers

Loading 2609.27232v1…