PaperlayerView exact revision on arXiv

Source-linked AI summary

MtReNIRS: A Self-Supervised Foundation Model for Transferable Neurovascular Representations

Tony Chae, Kleanthis Avramidis, Dimitra Emmanouilidou, Ivan J. Tashev

arXiv:2610.36159v1eess.SPq-bio.NC

TL;DR

fNIRS modeling remains constrained by small-scale data and heterogeneous acquisition settings, leaving uncertain whether one encoder can transfer across subjects, tasks, devices, and montages. MtReNIRS uses montage-aware masked-autoencoder pretraining across public datasets, and its representations transfer to three unseen datasets, with the strongest reported benefits in few-shot subject adaptation.

  • Problem

    It remains unclear whether one fNIRS encoder can transfer across subjects, tasks, devices, and montages amid substantial dataset and acquisition heterogeneity.

    Source: §I · pp. 1–2

  • Method

    MtReNIRS uses a montage-aware masked autoencoder pretrained on 12 public datasets, mapping heterogeneous S-D geometries to a shared anatomical reference and combining signals with coordinates and chromophore identity.

    Source: §I · p. 2

  • Results

    On three unseen datasets, MtReNIRS outperformed most existing fNIRS pretraining methods,,, and fully supervised approaches, with the largest and most consistent gains in within-subject few-shot adaptation.

    Source: §I · p. 2, §VI · p. 8

  • Takeaways & Limitations

    The transfer and ablation results suggest anatomically aligned self-supervision can provide reusable neurovascular representations for future label-efficient fNIRS and multimodal BCI systems.

    Source: §VII · p. 8

    Downstream evaluation was limited to three held-out datasets and may not fully characterize generalization across the broader range of settings.

    Source: §VI · p. 8

Abstract

from arXiv · show

Functional near-infrared spectroscopy (fNIRS) offers a promising approach for monitoring the hemodynamic correlates of neural activity in naturalistic settings. However, existing modeling methods for fNIRS are largely confined to small-scale, single-session data with low sampling rates, driven by high heterogeneity in acquisition devices and optode montages. In addition, the complex spatiotemporal structure of fNIRS signals further restricts model scalability and transferability across settings. As a result, it remains unclear whether a single fNIRS encoder can learn representations that are usable across subjects, tasks, devices, and montages. Here, we show that a montage-aware masked autoencoder, pretrained on 12 public fNIRS datasets spanning 268 subjects and 567 recording hours, learns generalizable neurovascular representations that transfer across three external datasets. Our fNIRS foundation model, MtReNIRS, jointly encodes the slow hemodynamic nature of fNIRS, chromophore identity and cortical location by mapping heterogeneous montages onto a shared anatomical reference. MtReNIRS achieves cross-subject performance on par or better than fully supervised baselines and excels in few-shot within-subject calibration, where it outperforms prior state-of-the-art fNIRS models, while remaining robust to sparse montages. Our results indicate that montage-aware self-supervision yields physiologically grounded, transferable fNIRS representations, providing a scalable foundation for label-efficient brain-computer interfaces.

I. Introduction

fNIRS modeling remains limited by small datasets and substantial variation in devices, sampling, signal representations, and montages, leaving cross-setting transfer uncertain. MtReNIRS addresses this with montage-aware self-supervised pretraining and evaluation on unseen datasets.

I. Introduction

  • The paper asks whether one pretrained fNIRS encoder can transfer across subjects, tasks, devices, and montages, a question for which comparable evidence remains limited.Existing supervised models and early foundation-model studies largely focus on individual datasets or within-subject few-shot classification.Source: §I · p. 1
  • fNIRS datasets vary in devices, sampling rates, signal representations, source–detector geometries, and cortical coverage, while individual datasets remain relatively small.The paper notes that single-study models may learn acquisition-specific rather than reusable neurovascular structure.Source: §I · p. 2
  • HbO/HbR signals are slow, delayed, and highly autocorrelated, so fNIRS pretraining must address both anatomical alignment and physiology-matched reconstruction.The paper states that prior fNIRS pretraining had not combined montage-aware scale, physiology-matched masked reconstruction, and cross-subject evaluation.Source: §I · p. 2
  • MtReNIRS is pretrained on 12 public datasets spanning 268 subjects and 567.1 recording hours, mapping heterogeneous S-D midpoints to a shared 10–05 reference.It combines 10-s HbO/HbR segments with canonical coordinates and chromophore identity, then reconstructs randomly masked windows without task labels.Source: §I · p. 2
  • On three unseen datasets, MtReNIRS outperformed most existing fNIRS pretraining methods,,, and fully supervised approaches.Evaluation included cross-subject leave-one-subject-out classification and within-subject calibration using 2–8 labels per class.Source: §I · p. 2
  • The paper also examines token duration, spatial correspondence, pretraining duration, and robustness under reduced-channel settings.The introduction describes fNIRS as a portable, non-invasive modality for monitoring cortical HbO and HbR changes.Source: §I · pp. 1–2

II. Related Work

Prior fNIRS models have addressed supervised decoding and masked pretraining, but transfer across datasets and montage layouts remains underexplored. MtReNIRS is positioned to address those limits through montage-aware representations grounded in hemodynamics.

II. Related Work

  • Supervised fNIRS classifiers include statistical, convolutional, recurrent, and attention-based models, but fNIRS-T and its spatial-embedding extension use fixed dataset-specific input spaces.The paper says those fixed spaces limit transfer across devices and montages; MtReNIRS preserves S-D pair identity while accommodating variable sensor counts and arrangements.Source: §II · p. 2
  • BERT-style masked prediction in language and ViT-based masked autoencoding in computer vision established reconstruction of hidden inputs from visible context, later yielding useful representations for sleep, multimodal physiology, and fNIRS,,,,,.Source: §II · p. 2
  • Existing fNIRS pretraining is limited in corpus scale and focuses mainly on within-subject few-shot classification, leaving unseen-subject, task, device, and montage generalization underexplored.The paper also identifies insufficient grounding in delayed hemodynamics, coupled HbO/HbR responses, and montage anatomy.Source: §II · p. 3

III. Datasets and Methods

The study assembles diverse public pretraining data and three separate downstream datasets, then standardizes signals and maps heterogeneous sensor layouts to a shared anatomical reference. The methods retain dataset-specific preprocessing branches while applying common signal and coordinate representations.

A. Pretraining Datasets

  • The pretraining corpus spans 12 datasets, 268 subject entries, 1,389 runs, and over 567 recording hours, covering varied tasks, devices, sampling rates, and cortical regions.After quality control, the authors segmented the data into 203,420 non-overlapping 10-s windows and used subject-disjoint training and validation splits.Source: A. Pretraining Datasets · p. 3
  • Three downstream datasets were held out entirely from pretraining, covering mental arithmetic and rest, auditory and visual speech tasks, and left-, right-hand, and foot motor trials.Their corpora contain 8, 8, and 30 subjects, respectively.Source: A. Pretraining Datasets · p. 3
  • The study uses 10-s windows to capture the multi-second rise and peak of the hemodynamic response while limiting windows that could cross event boundaries or mix conditions.The authors note that longer windows could extend beyond labeled downstream event boundaries.Source: A. Pretraining Datasets · p. 3

B. Signal Pre-processing and Montage Harmonization

  • Preprocessing converts raw intensity when available, filters runs from 0.01 to 0.5 Hz, resamples to 10 Hz, removes unsuitable channels, and z-scores and clips retained HbO/HbR channels.For datasets supplied only with processed concentrations, the preceding raw-intensity conversion steps are skipped.Source: B. Signal Pre-processing and Montage… · p. 3
  • Montage harmonization maps physical S-D pairs to a shared canonical 10–05 slot, using centered and RMS-scaled 3-D midpoint alignment when valid geometry is available.The denser 10–05 reference was chosen to reduce midpoint slot collisions while retaining a scalp-coordinate vocabulary shared with EEG.Source: B. Signal Pre-processing and Montage… · p. 3
  • When 3-D geometry is unavailable, the method uses anatomically supported 2-D placement or canonical label assignment, retaining distinct S-D pairs even when they collide in a slot.The same canonical-slot assignment and coordinate normalization are applied to downstream inputs.Source: B. Signal Pre-processing and Montage… · p. 3

C. Model and Self-Supervised Pretraining

MtReNIRS represents fNIRS segments together with chromophore and anatomical information, then learns by reconstructing masked signals. Its training samples combine multiple temporal windows and channels, with datasets sampled uniformly.

C. Model and Self-Supervised Pretraining

  • Each token identifies an S-D pair and HbO or HbR chromophore, combining a signal segment with spatio-chromophore information.The six 10-s windows in a 60-s crop preserve temporal context while representing each channel over time.Source: C. Model and Self-Supervised Pretraining · pp. 3–4
  • Canonical coordinates and chromophore identity are encoded with Fourier features and combined with the signal embedding to distinguish regional activity and model coupled HbO/HbR dynamics.The resulting signal and spatio-chromophore embeddings are summed to form each token.Source: C. Model and Self-Supervised Pretraining · p. 4
  • The encoder uses six Transformer blocks and temporal rotary position encoding, while canonical coordinates carry spatial identity.All S-D pairs from a given temporal window share its temporal index.Source: C. Model and Self-Supervised Pretraining · p. 4
  • For each 60-s crop, the masked autoencoder exposes visible tokens to the encoder and reconstructs masked 10-s waveforms with a lightweight Transformer decoder.The reconstruction objective is mean squared error on masked real tokens only.Source: C. Model and Self-Supervised Pretraining · p. 4, Figure 2 · p. 4
  • Training samples select datasets and runs uniformly, then draw a random 60-s crop containing six 10-s windows per chromophore channel.Runs with more than 256 chromophore channels are randomly subsampled to 256 channels.Source: C. Model and Self-Supervised Pretraining · p. 4
  • Pretraining runs for 100 epochs, with the epoch-95 checkpoint selected by self-supervised validation loss without inspecting downstream labels.The reported training setup uses AdamW and one 32-GB NVIDIA Tesla V100 GPU per run.Source: C. Model and Self-Supervised Pretraining · p. 4

D. Downstream Evaluation Protocols

The evaluation compares four MtReNIRS adaptation strategies with four non-pretrained baselines using cross-subject and within-subject protocols. It also varies token duration to keep classification features comparable.

D. Downstream Evaluation Protocols

  • Four pretrained adaptation strategies—linear probe, MLP head, LoRA, and full fine-tuning—were compared with four fully supervised baselines.The baselines were Scratch-FT, CNN, LSTM, and Stat-SVM.Source: D. Downstream Evaluation Protocols · p. 5
  • Cross-subject transfer used leave-one-subject-out folds, with subject-level inner validation and one evaluation on each held-out subject.Stat-SVM used all non-test subjects because it required no epoch selection.Source: D. Downstream Evaluation Protocols · p. 5
  • Few-shot calibration sampled 2, 4, 6, or 8 examples per class from each subject and tested on that subject’s remaining examples.The protocol used no other subjects, validation set, or early stopping; each sampling condition was repeated 50 times.Source: D. Downstream Evaluation Protocols · p. 5
  • For varying token durations, encoder outputs were temporally averaged to keep feature dimensionality fixed; the primary 10-s model used one window per trial.With one window, temporal-mean and flatten pooling were equivalent.Source: D. Downstream Evaluation Protocols · p. 5

E. Metrics and Statistical Analysis

Macro-F1 was the primary metric, with subjects treated as independent units. Paired comparisons used subject-level differences, bootstrap confidence intervals, and Wilcoxon signed-rank tests.

E. Metrics and Statistical Analysis

  • Macro-F1 was the primary metric, and each subject was the independent unit for statistical analysis.For LOSO, scores were summarized across held-out subjects; for few-shot evaluation, seed-level scores were first averaged within each subject.Source: E. Metrics and Statistical Analysis · p. 5
  • Classification accuracy was additionally reported in Table 3 to facilitate comparison with prior work.Repeated few-shot draws from the same subject were not treated as independent samples.Source: E. Metrics and Statistical Analysis · p. 5
  • Paired comparisons used subject-level macro-F1 differences, 20,000 subject-resampled percentile bootstrap draws for 95% confidence intervals, and two-sided Wilcoxon signed-rank tests.Reported p-values were not adjusted for multiple comparisons.Source: E. Metrics and Statistical Analysis · p. 5

IV. Experimental Results

Across the three downstream datasets, MtReNIRS adaptation strategies matched or exceeded the fully supervised baseline. Pairwise results show that pretraining was competitive, though no adaptation strategy was uniformly best.

IV. Experimental Results

  • Across all three downstream datasets, MtReNIRS adaptation strategies matched or exceeded the fully supervised baseline.Table 1 reports 10-s cross-subject LOSO macro-F1, with boldface marking the highest point estimate per dataset rather than statistical significance.Source: A. Benchmark 1: Cross-Subject Transfer… · p. 5, Table 1 · p. 6
  • MtReNIRS-LP gains over Scratch-FT were 5.0 points on Gunther, 4.3 on Shader, and 1.2 on SujinBak, but none of the paired tests was significant.The corresponding p-values were 0.109, 0.383, and 0.416.Source: A. Benchmark 1: Cross-Subject Transfer… · pp. 5–6
  • MtReNIRS-LP exceeded CNN and LSTM on Gunther and SujinBak and exceeded LSTM on Shader, while no adaptation strategy was uniformly superior across datasets.MtReNIRS-LoRA exceeded MtReNIRS-LP on SujinBak by 2.4 points (p = 0.015).Source: A. Benchmark 1: Cross-Subject Transfer… · p. 6

B. Benchmark 2: Within-Subject Few-Shot Calibration

Within-subject few-shot calibration favored MtReNIRS-LP across datasets and shot counts, with dataset-specific significance. Pretraining gains emerged early and did not increase monotonically with additional epochs.

B. Benchmark 2: Within-Subject Few-Shot Calibration

  • MtReNIRS-LP had the highest point estimate in every dataset–shot condition, with gains over Scratch-FT significant on Gunther and SujinBak but not Shader.Gains were 6.8–7.7 points on Gunther, 5.2–11.0 on SujinBak, and 3.9–4.7 on Shader; Figure 4 summarizes subject-level trends and paired tests.Source: B. Benchmark 2: Within-Subject… · p. 6, Figure 4 · p. 7
  • On Gunther, MtReNIRS-LP achieved the highest accuracy among fNIRS-only pretrained models at matched 2–8-shot settings and surpassed the fully supervised baseline at 6–24 shots.The comparison is contextual rather than controlled because preprocessing, sampling, and evaluation details may differ across studies.Source: B. Benchmark 2: Within-Subject… · p. 6, Table 3 · p. 6

A. Effect of Pretraining Duration

  • At epoch 95, pretraining improved macro-F1 over a randomly initialized frozen-encoder probe by 3.9 points on Gunther, 9.3 on Shader, and 3.2 on SujinBak.Gains were significant for Shader and SujinBak but not Gunther; most gains emerged early, and they did not improve monotonically with more pretraining. Shader performance peaked around epoch 40 and then declined despite continued decreases in reconstruction loss.Source: A. Effect of Pretraining Duration · pp. 6–7, Figure 5 · p. 7
  • Table 4 presents a token-duration ablation under cross-subject LOSO, reporting macro-F1 as mean and population standard deviation across held-out subjects.The supplied passage identifies the ablation’s scope and reporting format but not its outcomes.Source: Table 4 · p. 7

B. Effect of Temporal Context

Among the tested token durations, the 10-second model performed best on cross-subject LOSO and within-subject few-shot evaluations. The study compared 2-, 5-, and 10-second pretraining tokens while keeping downstream examples 10 seconds long.

B. Effect of Temporal Context

  • Pretraining compared 2-, 5-, and 10-second token durations under 25% masking, while downstream examples stayed 10 seconds and temporal-mean pooling fixed feature dimension.Source: B. Effect of Temporal Context · p. 7
  • The 10-s model had the highest LOSO mean on all three datasets and the highest within-subject mean across three datasets and four shot counts.Source: B. Effect of Temporal Context · p. 7
  • The 10-s result aligns with prior fNIRS studies reporting benefits from longer temporal context for capturing slow hemodynamic response,,.Source: B. Effect of Temporal Context · p. 7

C. Effect of Anatomical Correspondence

Coordinate shuffling degraded few-shot performance across all 12 dataset–shot conditions, while reduced-montage evaluation found strong performance with 16 retained source–detector pairs. The reduced-coverage result is limited to one motor dataset.

C. Effect of Anatomical Correspondence

  • The shuffled-coordinate comparison preserved waveforms, chromophore identities, and coordinate values while permuting coordinate-to-channel assignments during pretraining.Both encoders received correct held-out coordinates at downstream evaluation, isolating anatomical correspondence during pretraining.Source: C. Effect of Anatomical Correspondence · p. 7
  • Coordinate shuffling reduced mean few-shot macro-F1 in all 12 dataset–shot conditions, with significant declines on Gunther and SujinBak but not Shader.Declines were 7.8–10.3 points on Gunther (all p = 0.0078), 5.0–8.7 points on SujinBak (all p ≤2.05 × 10−7), and 1.7–3.1 points on Shader, where no comparison reached p < 0.05.Source: C. Effect of Anatomical Correspondence · p. 7

D. Robustness to Reduced Sensor Coverage

  • At 16 retained S-D pairs, MtReNIRS-LP reached median macro-F1 0.683, compared with 0.689 for the symmetric 18-pair reference and 0.695 for the full 20-pair montage.Across retained-pair counts, MtReNIRS-LP had the highest median macro-F1; the analysis used SujinBak alone and does not establish robustness across other cortical regions or tasks.Source: Figure 6 · p. 8, D. Robustness to Reduced Sensor Coverage · p. 8

VI. Discussion

The discussion reports that pretraining benefits were largest and most consistent in within-subject few-shot adaptation, while cross-subject transfer remained challenging. The authors also identify evaluation limits and conclude that anatomically aligned self-supervision offers a basis for reusable neurovascular representations.

VI. Discussion

  • Across datasets, the largest and most consistent gains appeared in within-subject few-shot adaptation rather than pure cross-subject transfer.The discussion attributes remaining cross-subject difficulty to subject-specific physiological variability and says a small number of subject-specific labels appears sufficient for adaptation.Source: §VI · p. 8
  • Coordinate-shuffling degradation suggests transferability depends on preserving anatomical correspondence across heterogeneous montages, not only waveform structure.Source: §VI · p. 8
  • Downstream evaluation used three held-out datasets and could not isolate device effects because device, task, and montage differences co-occurred.The device invariance of the learned representations therefore remains unclear.Source: §VI · p. 8

VII. Conclusion

  • The authors conclude that anatomically aligned self-supervision can support reusable neurovascular representations and future label-efficient fNIRS and multimodal BCI systems.Source: §VII · p. 8

Related papers

Loading 2610.36159v1…