PaperlayerView exact revision on arXiv

Source-linked AI summary

MnemoDyn: Learning Resting State Dynamics from 40K FMRI sequences

Sourav Pal, Viet Luong, Hoseok Lee, Tingting Dan, Guorong Wu, Richard Davidson, Won Hwa Kim, Vikas Singh

arXiv:2608.23936v1cs.LG

TL;DR

Modeling multiscale rs-fMRI dynamics across subjects, sites, and protocols remains challenging. MnemoDyn learns these dynamics with an efficient wavelet-parameterized evolution operator and consistently outperforms transformer-based baselines across reconstruction, classification, and regression tasks.

  • Problem

    A key challenge is learning rs-fMRI dynamics while capturing spatial and temporal structure and supporting predictions across subjects, sites, and protocols.

  • Method

    MnemoDyn uses a wavelet-parameterized evolution operator to model multiscale temporal dependencies without attention mechanisms, trained on parcellated rs-fMRI sequences.

  • Results

    MnemoDyn consistently improves over state-of-the-art transformer baselines across multiple rs-fMRI datasets and reconstruction, classification, and regression tasks.

  • Takeaways & Limitations

    Large-scale pre-training on 40K rs-fMRI sequences yields an open-source foundation model for fine-tuning on smaller datasets.

  • Takeaways & Limitations

    Experiments are limited to parcellated rs-fMRI data, leaving voxel-level, multimodal, and longitudinal extensions for future work.

Abstract

from arXiv · show

We present a dynamical-systems based model for resting-state functional magnetic resonance imaging (rs-fMRI), trained on a dataset of roughly 40K rs-fMRI sequences covering a wide variety of public and available-by-permission datasets. While most existing proposals use transformer backbones, we utilize multi-resolution temporal modeling of the dynamics across parcellated brain regions. We show that MnemoDyn is compute efficient and generalizes very well across diverse populations and scanning protocols. When benchmarked against current state-of-the-art transformer-based approaches, MnemoDyn consistently delivers superior reconstruction quality. Overall, we find that with such large-scale pre-training on (non-proprietary) rs-fMRI datasets, we get a highly performant model for various downstream tasks. Our results also provide evidence of the efficacy of the model on small sample size studies which has implications for neuroimaging studies at large where resting state fMRI is a commonly acquired imaging modality.

1 INTRODUCTION

MnemoDyn models rs-fMRI signals as trajectories governed by a learnable, multi-resolution dynamical operator rather than relying on attention-based sequence modeling. Trained on 40K rs-fMRI sequences, it consistently improves over transformer-based baselines and supports downstream prediction across heterogeneous populations and smaller datasets.

  • Motivation: MnemoDyn addresses the need to model latent dynamics in resting-state hemodynamic signals for neuroscience and applications including surgery planning and seizure localization.rs-fMRI provides temporal signals encoding rich neural processes.
  • Downstream applications: The model generalizes to prediction of clinical variables across heterogeneous populations, including age, sex, cognitive traits, and neurodegeneration markers.Lightweight adapter layers adapt the pretrained model to diverse downstream cohorts.
  • Method: The model learns a wavelet-parameterized evolution operator that captures multiscale temporal dependencies without attention mechanisms and scales efficiently to long sequences.This design avoids positional embeddings and tokenization schemes that can be domain dependent and sensitive to hyperparameter tuning.
  • Results: MnemoDyn achieves consistent improvements over state-of-the-art transformer-based baselines across multiple rs-fMRI datasets and reconstruction, classification, and regression tasks.The comparison spans diverse datasets and task types.
  • Availability and impact: MnemoDyn is an open-source rs-fMRI foundation model trained on 40K sequences and intended for use and fine-tuning on smaller datasets.The model is positioned as a practical alternative for resource-constrained settings.

2 MODELING BRAIN DYNAMICS

MnemoDyn models rs-fMRI as latent neural dynamics in a low-dimensional, multiscale representation, linking observations to hidden states through state-space, continuous-time, and operator formulations. Its learned-kernel CDE uses wavelet-based multi-resolution controls to capture non-Markovian history, while requiring solutions to substantial matrix and latent-dimensionality costs.

  • Latent dynamical formulation: MnemoDyn decomposes high-dimensional rs-fMRI measurements into a hidden neural process and a measurement process evolving on a low-dimensional multiscale representation.The observed signal x_t ∈ R^n is paired with a latent neural state z_t ∈ R^d.
  • Latent dynamical formulation: The state-space model uses transition dynamics f, observation map h, optional exogenous input u_t, and Gaussian process and observation noise.The latent state may follow potentially nonlinear dynamics, while the observation map returns the measured brain signal.
  • Continuous-time operator formulation: Continuous-time modeling represents latent-state evolution with a parameterized ODE vector field and reframes the trajectory as a nonlinear operator mapping initial states and control functions to functions.This formulation is motivated by the continuous nature of brain signals despite discrete fMRI measurements.
  • Integral operators and CDEs: MnemoDyn expresses dynamics through a learned-kernel integral equation and implements controlled differential equations that capture non-Markovian dependencies from multi-scale control paths.The CDE formulation generalizes ODEs, while the learned kernel avoids numerical solvers used in related integral-equation formulations.
  • Multi-resolution kernel parameterization: Wavelet-based multi-resolution analysis parameterizes the kernel to provide temporal locality and scale adaptivity across different temporal resolutions.Wavelet basis functions and state-modulated matrix-valued functions let the kernel selectively attend to input features at multiple scales.
  • Compute challenge: The formulation faces computational challenges because its multi-scale parameterization requires huge matrices, while high-dimensional rs-fMRI demands a very large latent dimension.Both issues threaten the practicality of the otherwise brain-signal-informed formulation.

3 EXPERIMENTS

The experiments evaluate MnemoDyn through large-scale pretraining, cross-dataset reconstruction, and fine-tuning across diagnostic, demographic, and behavioral prediction tasks. Results show strong generalization, competitive or superior downstream performance, and robustness across pretraining strategies.

  • Datasets and evaluation: MnemoDyn is pretrained self-supervised on UK Biobank and separately on HCP rs-fMRI data to assess generalization across datasets.UK Biobank includes approximately 65K samples from participants aged 44–69, collected across multiple sites.
  • Preprocessing: The preprocessing pipeline converts raw BIDS rs-fMRI volumes into standardized CIFTI time series and parcellated regional signals for model training.The conversion aligns cortical and subcortical signals in 91282 grayordinates before downstream atlas-based processing.
  • Architecture: MnemoDyn uses wavelet-scale layers with residual cross-scale connections to capture fine-scale fluctuations and long-range temporal structure in resting-state dynamics.The model processes sequences of length 1200 for HCP and 490 for UK Biobank with N = 450 brain regions.
  • Pretraining and fine-tuning: Masked pretraining reconstructs randomly masked temporal and spatial blocks from surrounding context, with 70% of the observed signal masked in the presented models.The method uses AdamW with cosine annealing and warm restarts, while fine-tuning freezes the backbone and trains task-specific heads.
  • Representation capability: Cross-evaluation shows that models pretrained on UK Biobank or HCP faithfully reconstruct held-out data from either dataset, demonstrating representation capacity and cross-dataset generalization.The reconstruction evaluation reports validation MSE and R2 scores across foundation models trained with different data.
  • Downstream evaluation: Fine-tuned MnemoDyn variants achieve state-of-the-art ADNI diagnostic and biomarker prediction performance, comparable or improved UK Biobank demographic performance, and strong HCP-Aging demographic and trait results.Reported advantages include higher accuracy and F1 for classification and lower MSE with higher correlation for regression tasks; both pretraining strategies perform equally well.

4 RELATED WORK

Related work spans operator learning for dynamical systems, attention-based models for neuroimaging, and lightweight domain-specific sequence models. The cited literature highlights both advances in learning dynamical operators and limitations of Transformers for noisy, irregular, or low-data sequence modeling.

  • Operator Learning and State Space Models: Operator learning learns mappings between infinite-dimensional function spaces, with DeepONet and Fourier Neural Operator demonstrating solution-operator learning for parametric PDEs.These methods target dynamics governed by differential equations.
  • Attention-Based Models for Brain Imaging Data: Transformers have been proposed for spatiotemporal modeling in fMRI, EEG, and MEG, but evaluations report underperformance on long-range, noisy, or irregularly sampled signals.The passage also notes that attention-based inductive biases can be mismatched to brain data.
  • Lightweight Models for Domain-Specific Sequence Modeling: Benchmark studies show that compact CNN- and RNN-based models can outperform Transformers on time-series tasks, particularly in low-data regimes.Recent work argues against overusing large attention-based or foundation models for structured domains such as time series and brain imaging.

5 CONCLUSIONS

The conclusion argues that compact, domain-specific dynamical models are well suited to noisy, heterogeneous, multiscale neuroimaging data, while identifying important extensions beyond parcellated rs-fMRI and cross-sectional studies.

  • Contributions: Domain-specific inductive biases and compact architectures can surpass large generic foundation models when data are limited or structured, especially for noisy and heterogeneous neuroscience data.Neuroimaging signals also exhibit rich multiscale dynamical structure, favoring temporal locality and sparsity.
  • Contributions: Fine-tuning MnemoDyn demonstrates strong performance across diagnostic, prognostic, and trait prediction tasks.
  • Limitations and future directions: The experiments are limited to parcellated rs-fMRI data, leaving voxel-level and multimodal inputs such as EEG and PET as future directions.
  • Limitations and future directions: Extending MnemoDyn to longitudinal studies is identified as an important next step for establishing its full potential.

7 ETHICS STATEMENT

The study uses publicly available, de-identified rs-fMRI datasets under appropriate data usage agreements and reports compliance with institutional and ICLR ethical guidelines. It collected no new human or animal data and anticipates no direct negative societal impact, while acknowledging potential misuse risks in healthcare applications.

  • 7 ETHICS STATEMENT: The study uses publicly available, de-identified rs-fMRI datasets under appropriate data usage agreements.
  • 7 ETHICS STATEMENT: No new human or animal data were collected, and the study complies with institutional and ICLR ethical guidelines.
  • 7 ETHICS STATEMENT: The authors foresee no direct negative societal impact but acknowledge potential misuse risks for machine learning models applied to healthcare data.

A APPENDIX · A.1 PRE-PROCESSING PIPELINE · A.1.1 NIFTI →CIFTI DTSERIES.

The preprocessing pipeline converted raw BIDS-formatted ADHD-200 fMRI volumes into HCP-style dense time series (dtseries) for standardized grayordinate analysis. Volumetric BOLD runs were mapped to cortical surfaces using anatomically constrained projection, while subcortical structures were processed separately.

  • A.1.1 NIFTI →CIFTI DTSERIES.: Raw BIDS-formatted ADHD-200 fMRI volumes were converted into HCP-style dense time series (dtseries).This conversion enabled analysis in a standardized grayordinate framework.
  • A.1.1 NIFTI →CIFTI DTSERIES.: The conversion targeted a standardized grayordinate framework for subsequent analysis.
  • A.1.1 NIFTI →CIFTI DTSERIES.: For each subject/session, volumetric BOLD runs were projected onto cortical surfaces.
  • A.1.1 NIFTI →CIFTI DTSERIES.: Cortical mapping used ribbon-constrained volume-to-surface projection.The projection used the fs LR 32k left/right midthickness meshes.
  • A.1.1 NIFTI →CIFTI DTSERIES.: Corresponding white and pial surfaces supplied anatomical constraints during cortical mapping.
  • A.1.1 NIFTI →CIFTI DTSERIES.: Subcortical structures were handled separately by resampling the functional data.

A.1.2 CIFTI DTSERIES →PARCELLATION

Dense fMRI time series were converted into parcellated ROI sequences using one of three atlases, with input data oriented as timepoints × grayordinates before applying parcel labels.

  • Atlas selection: Three brain atlases were used for parcellation: Gordon (N =333), Schaefer (N =424), and Tian (N =450).Each atlas defines the number of resulting regions of interest.
  • Input orientation: Each .dtseries.nii was loaded as T ×G, representing timepoints × grayordinates, and transposed when its first dimension equaled G=91,282.The orientation step standardized the dense time-series representation before parcellation.
  • Parcel mapping: The atlas .dlabel.nii provided integer parcel labels used to map the dense time series into regions of interest.The labels were defined over the corresponding grayordinate space.

A.1.3 NORMALIZATION

MnemoDyn estimates normalization constants exclusively from the training split by pooling samples across subjects and time. It applies robust, ROI-wise scaling using fitted median and interquartile-range statistics.

  • Normalization: Normalization constants are estimated solely on the training split, with each sample represented as X(i) ∈ R^T×D and T = 1200 by default.The statistics are computed from training data rather than held-out samples.
  • Normalization: For each ROI or feature, the empirical distribution is formed by concatenating all training samples across subjects and time.This pools observations before estimating per-feature normalization statistics.
  • Normalization: The normalization statistics include Q25,r, Q75,r, IQRr = Q75,r − Q25,r, and Q99,r for each ROI or feature.These quantiles characterize the empirical distribution used for robust normalization.
  • Normalization: The robust scaler retains {medianr, IQRr} for r = 1 through D as its fitted statistics.At transform time, new samples are normalized ROI-wise using these fitted values.

A.2 EXTENDED TRAINING DETAILS … A.9 ABLATIONS

MnemoDyn combines multiresolution operator modeling with denoising, masking, and Brain-JEPA-style self-supervision, then transfers frozen representations across heterogeneous rs-fMRI tasks. Analyses and ablations show scaling with model/data size, wavelet and low-rank benefits, strong within-modality and zero-shot transfer, and no demonstrated cross-modality generalization.

  • A.2 EXTENDED TRAINING DETAILS; A.2.1 FOUNDATIONAL TRAINING: MnemoDyn uses four residual-refinement operator blocks, 150-dimensional hidden states, a 5-dimensional low-rank bottleneck, and six db2 wavelet levels.Pre-training runs for 50 epochs with AdamW and a composite MSE-plus-MAE objective.
  • A.2.2 DENOISING AUTOENCODER; A.2.3 MASKED AUTOENCODER: Denoising reconstructs clean sequences from Gaussian-corrupted inputs, while masked autoencoding reconstructs five disjoint 80-step temporal blocks from unmasked context.Masked reconstruction supervises only occluded positions, encouraging inference from surrounding temporal context.
  • A.2.4 BRAIN-JEPA STYLE SCHEME: Brain-JEPA-style training masks temporal, spatial, and cross spatio-temporal subsets and trains a student against an EMA teacher using masked-index MSE.The scheme is intended to capture local and global structure in rs-fMRI time series.
  • A.2.5 DOWNSTREAM HEAD: For downstream tasks, the frozen backbone feeds pooled features to task-specific MLP heads using MSE for regression and cross-entropy for classification.The default head applies LayerNorm, GELU, and dropout before its final linear output.
  • A.3 ANALYSIS OF PRE-TRAINED OPERATOR: Wavelet-operator kernels dominate learned norm mass at approximately 1500 versus approximately 20 for dense layers, while output projections are over 95% sparse.Operator responses are strongest and most localized at fine temporal scales, becoming smoother and weaker at coarser levels.
  • A.4 MNEMODYN AS A FOUNDATION MODEL FOR RS-FMRI; A.4.1 SCALING BEHAVIOR: MODEL SIZE AND DATA SIZE; A.4.2 SCOPE OF TRANSFER: WITHIN-MODALITY GENERALITY AND REUSE: Increasing operator blocks from two to four and increasing UK-Biobank pre-training data both consistently improve downstream performance, supporting model-size and data-size scaling.The paper frames MnemoDyn as a foundation model for rs-fMRI, with large-scale self-supervised pre-training, reusable representations, and cross-dataset robustness.
  • A.5 DISTINCTIONS BETWEEN MNEMODYN AND CLASSICAL OPERATOR-LEARNING FRAMEWORKS; A.6 IMPORTANCE OF THE MULTI-RESOLUTION ANALYSIS AND LOW RANK STRUCTURE: MnemoDyn learns self-supervised latent dynamics from unpaired rs-fMRI sequences using wavelet-domain pseudo-differential operators, low-rank structure, and continuous-time evolution rather than supervised PDE operator regression.Removing wavelets reduces classification accuracy by 20–23 points on ADNI and 12 points on UKB sex prediction, while removing low-rank structure causes immediate memory blow up.
Loading 2608.23936v1…