PaperlayerView exact revision on arXiv

Source-linked AI summary

M2LG-DG: A Multi-modal Local-Global Domain Generalization Framework for Cross-site Major Depressive Disorder Classification

Muhammad Asif Hasan, Yanming Zhu, Xuefei Yin, Alan Wee-Chung Liew

arXiv:2609.09186v1cs.CV

TL;DR

Cross-site rs-fMRI MDD classifiers can learn site-specific rather than diagnostic patterns, motivating source-only domain generalization that integrates multimodal and site-aware representations. M2LG-DG combines local-global fMRI encoding, structured multimodal fusion, and cross-site supervised contrastive learning, achieving a 69.48% AUC on four held-out REST-meta-MDD sites and supporting applicability on ABIDE.

  • Problem

    Cross-site MDD models may reflect scanner and acquisition conditions rather than diagnostic information, while existing psychiatric domain-generalization approaches often rely on one imaging modality.

  • Method

    M2LG-DG combines dual-stream local-global rs-fMRI encoding, shared-private multimodal fusion with cross-attention and gating, and cross-site supervised contrastive learning.

  • Results

    69.48% AUC was achieved across four held-out REST-meta-MDD sites, with the highest AUC at all four sites and additional evaluation on ABIDE.

  • Takeaways & Limitations

    The evaluated source-only framework supports cross-site psychiatric neuroimaging classification across held-out REST-meta-MDD and ABIDE target domains.

  • Takeaways & Limitations

    The study identifies acquisition-aware temporal standardization as future work for differences in repetition time, scan duration, and temporal sampling across sites.

Abstract

from arXiv · show

Classification models based on resting-state functional magnetic resonance imaging (rs-fMRI) often show lower performance at imaging sites not included during model development, which can limit their use in clinical settings. Domain generalization (DG) addresses this issue by learning representations from source sites that remain effective for unseen target sites. However, existing DG approaches for psychiatric disorder classification commonly rely on a single imaging modality and may not fully account for site-specific acquisition effects on the learned representation space. Subjects scanned at the same site share scanner hardware, acquisition settings, and preprocessing characteristics, which can cause representations to reflect acquisition conditions rather than diagnostic information. In this work, we present M2LG-DG, a source-only multimodal local-global framework for cross-site major depressive disorder (MDD) classification. The framework employs a dual-stream rs-fMRI encoder, where the global pathway models inter-regional dependencies through self-attention and the local pathway performs graph-constrained aggregation over functional connectivity-derived brain graphs. Imaging and non-imaging representations are decomposed into shared and private components and integrated through bidirectional cross-attention with a learned modality gate. A cross-site supervised contrastive objective forms positive pairs from same-class subjects acquired at different source sites, encouraging the fused representation to preserve diagnostic information across acquisition domains. On four held-out REST-meta-MDD sites, M2LG-DG achieves an AUC of 69.48% and exceeds the closest comparison method by 2.18 percentage points. Experiments on the Autism Brain Imaging Data Exchange (ABIDE) dataset further support its applicability to other psychiatric neuroimaging classification tasks.

1. Introduction

M2LG-DG targets cross-site MDD classification by learning source-only representations that combine complementary imaging and non-imaging information while reducing site-specific effects.

  • Site-specific acquisition characteristics can dominate learned representations, producing unreliable predictions at unseen imaging sites.
  • The framework separates shared and modality-specific information before integrating rs-fMRI and non-imaging representations through bidirectional cross-attention and a learned modality gate.
  • M2LG-DG is a source-only multimodal domain-generalization framework for cross-site MDD classification with every target site excluded from development.
  • Its dual-stream fMRI encoder combines unrestricted cross-region interactions in a global pathway with graph-constrained spatiotemporal modeling in a local pathway.
  • Cross-site supervised contrastive learning excludes same-site same-class positives to emphasize diagnostic consistency across acquisition sites, with REST-meta-MDD and ABIDE experiments supporting the framework.

2. Related work

Related work frames domain generalization as a source-only response to cross-site distribution shifts and identifies site-aware multimodal contrastive learning as an underdeveloped direction.

  • Domain generalization learns representations from source domains for unseen targets, avoiding the target-domain data required by domain adaptation.
  • Existing generalization methods include data manipulation, representation learning, and learning-strategy approaches that seek more transferable feature spaces.
  • Multimodal methods commonly align separate modality encoders in a shared embedding space or tokenize modalities for unified Transformer learning.
  • M2LG-DG’s overview combines local-global fMRI encoding, multimodal decomposition and fusion, and site-aware contrastive classification in one pipeline.
  • Most contrastive approaches do not explicitly account for the joint relationship between diagnostic labels and site membership when defining positive and negative pairs.

3. Proposed method

The proposed method processes rs-fMRI and non-imaging data through source-only preprocessing, dual-stream encoding, and a functional-connectivity graph constructed without target-domain information.

  • The dual-stream encoder uses a temporal-to-spatial global pathway with 1D convolutions and Transformers and a complementary local pathway for graph-based modeling.
  • rs-fMRI is represented as ROI time series, while non-imaging inputs include variables such as sex, age, and years of education.
  • The method defines source domains and a disjoint unseen target domain, aiming to learn diagnosis-related representations that generalize across domains.
  • Each ROI time series is resampled to T_fixed = 200 samples with cubic-spline interpolation to provide a consistent input dimension.
  • The FC-derived graph averages source-subject correlation matrices, retains connections above a percentile threshold, and adds self-loops to form the binary adjacency matrix.

3.3. Dual-stream fMRI encoder

The dual-stream fMRI encoder captures complementary global and local functional patterns by processing temporal and spatial information in opposite orders. Their embeddings are combined into a single fMRI representation.

  • 3.3.1. Global encoder: The global pathway extracts ROI temporal features, then uses Transformer self-attention to model unrestricted dependencies across regions.ROI identity embeddings preserve region identity, and averaging the ROI tokens produces the global embedding.
  • 3.3.2. Local encoder: The local pathway applies graph convolutions before temporal convolutions, using an FC-derived adjacency structure refined by learnable edge embeddings.The normalized adjacency combines data-driven and FC-derived connectivity information for graph-based aggregation.
  • 3.3.2. Local encoder: Graph convolution aggregates information across Chebyshev-defined neighborhood hops, followed by temporal modeling within each ST-GCN block.The neighborhood order K controls the number of graph hops, while residual connections are included in the blocks.
  • 3.3.2. Local encoder: Adaptive global average pooling over the final local encoder output yields a vector that is projected into the local embedding.The local encoder uses progressively increasing channels and temporal downsampling after the first block.
  • 3.3.3. fMRI representation fusion: The global and local embeddings are concatenated and projected with a linear layer and GELU activation to form the fMRI representation.The resulting h_fmri is used for subsequent multimodal fusion.

3.4. Non-imaging encoder

The non-imaging encoder maps preprocessed subject-level variables into a learned embedding using a multilayer perceptron.

  • 3.4. Non-imaging encoder: A two-hidden-layer MLP encodes the preprocessed non-imaging feature vector into a non-imaging representation.Each hidden layer uses batch normalization, GELU activation, and dropout, followed by a final linear layer.

3.5. Shared-private multi-modal decomposition

The shared-private decomposition separates multimodal information into cross-modal shared components and modality-specific private components. Auxiliary losses align shared information, reduce within-modality redundancy, and maintain private separation.

  • 3.5. Shared-private decomposition: Each fMRI and non-imaging representation is decomposed into shared and private components to preserve both common and modality-specific information.Four separate two-layer MLPs implement the decomposition.
  • 3.5.1. Shared alignment: The shared components are aligned by comparing their l2-normalized representations with a mean squared similarity loss.The comparison emphasizes directional agreement rather than magnitude and averages across batches and feature dimensions.
  • 3.5.2. Orthogonality constraint: An orthogonality loss encourages shared and private components within each modality to be close to orthogonal.The loss uses absolute inner products computed on the unnormalized components.
  • 3.5.3. Private separation: A margin-based difference loss keeps private components from the two modalities distinct without continuous repulsion after the margin is reached.The hinge term becomes zero once the average squared distance between normalized private components reaches m.

3.6. Cross-attention fusion

The framework fuses shared and private multimodal tokens through bidirectional cross-attention and a sample-specific modality gate. It then jointly optimizes classification with a site-aware contrastive objective that emphasizes cross-site diagnostic consistency.

  • 3.6. Cross-attention fusion: Bidirectional cross-attention lets fMRI tokens attend to non-imaging tokens and then reverses the direction so non-imaging tokens attend to updated fMRI tokens.Each modality contributes shared and private tokens to the cross-attention layers.
  • 3.6. Cross-attention fusion: A softmax modality gate assigns sample-specific weights to the imaging and non-imaging streams before concatenation and output projection.The gate is implemented as a two-layer MLP over the flattened token sequences.
  • 3.7. Classification and contrastive learning: The fused representation is passed through a three-layer classification MLP, with cross-entropy using label smoothing and source-fold inverse-frequency class weights.The class weights address imbalance across sites.
  • 3.7. Classification and contrastive learning: Cross-site supervised contrastive learning treats same-class subjects from different sites as positives, while same-site same-class subjects remain in the denominator but are excluded from positives.Subjects with different diagnostic labels are negatives regardless of site, and anchors without valid cross-site positives contribute only to classification loss.
  • 3.7. Classification and contrastive learning: The classification and cross-site contrastive losses are jointly optimized, applying classification to logits and contrastive learning to normalized fused representations.This objective combines diagnostic prediction with site-aware representation organization.

3.8. Optimization

The total optimization objective combines classification, decomposition, and cross-site supervised contrastive losses, with weights controlling their relative contributions.

  • 3.8. Optimization: The total loss combines classification, three decomposition losses, and cross-site supervised contrastive loss.The objective is expressed as L_total = L_CE + λ_sim L_sim + λ_orth L_orth + λ_diff L_diff + λ_con L_con.
  • 3.8. Optimization: A shared decomposition weight λ_decomp reduces the number of hyperparameters by controlling the three decomposition losses together.The decomposition losses use one common weight rather than separate coefficients.

4. Experiments and results

M2LG-DG performs strongly across held-out REST-meta-MDD sites, with ablations supporting its multimodal fusion and cross-site contrastive components. Sensitivity and attribution analyses further characterize parameter robustness and distributed brain-region contributions, while ABIDE results support broader applicability.

  • 4.2. Diagnosis performance of M2LG-DG: M2LG-DG achieved the highest values for all five metrics at Sites 9 and 20, and led four metrics at Site 15 and three at Site 21.At Site 15, MM-GTUNets narrowly exceeded M2LG-DG in SPE; at Site 21, the remaining reported metric was not a best result.
  • 4.2. Diagnosis performance of M2LG-DG: 69.48% AUC was achieved in the four-site macro-average, 2.18 percentage points above MM-GTUNets, with the highest values across all five metrics.The macro-average was 63.64% ACC, 62.42% SPE, 63.92% SEN, 69.48% AUC, and 63.62% F1.
  • 4.3. Ablation study: Replacing concatenation with cross-attention and modality gating increased ACC, AUC, and F1 by 2.79, 4.56, and 3.88 percentage points, respectively.The comparison is between configurations C2 and C4 in the nested ablation sequence.
  • 4.3. Ablation study: Removing the cross-site contrastive loss reduced ACC, AUC, and F1 by 1.33, 1.64, and 2.34 percentage points relative to the complete configuration.The reduced configuration achieved 62.31% ACC, 67.84% AUC, and 61.28% F1.
  • 4.4. Sensitivity analysis of hyperparameters: Moderate loss weights and a temperature of 0.07 performed best, graph thresholds from the 70th to 85th percentiles were stable, and T_fixed = 200 was best in the tested range.For temporal resampling, ACC ranged from 62.9% to 63.6%, AUC from 68.8% to 69.5%, and F1 from 62.7% to 63.6%.
  • 4.5. Interpretation of important brain regions: The ten highest-ranked ROIs spanned cingulate, temporal, default-mode, cerebellar, frontal, visual, insular, and sensorimotor systems.The attribution results associated classification decisions with distributed functional systems rather than a single anatomical region.
  • 4.5. Application to ABIDE: 65.15% ACC, 71.67% AUC, and 64.37% F1 were obtained across two held-out ABIDE sites, supporting applicability to another psychiatric neuroimaging task.The averaged results also included 65.87% SPE and 62.41% SEN.

5. Discussion

M2LG-DG achieves strong cross-site MDD performance, while ablations indicate that attention-based fusion contributes more than decomposition and that acquisition differences affect threshold-dependent metrics. The authors identify acquisition-aware temporal standardization as a future direction.

  • Cross-site performance: 69.48% average AUC, with the highest AUC at all four held-out sites, establishes M2LG-DG's cross-site performance.The site-level AUCs are 75.81%, 68.40%, 70.16%, and 63.53%.
  • Cross-site performance: Sensitivity and specificity vary across target sites, with the discussion associating these differences with cohort composition and acquisition conditions.Site 9 has higher specificity, whereas Site 21 has higher sensitivity; similar patterns occur for comparison methods.
  • Ablation analysis: 4.56 percentage points of additional AUC improvement comes from bidirectional cross-attention and modality gating, exceeding decomposition's 0.71-point gain.Across ACC, AUC, and F1, fusion accounts for the larger improvement in the nested ablation sequence.
  • Ablation analysis: Removing the cross-site supervised contrastive loss reduces AUC from 69.48% to 67.84%, alongside lower ACC and F1.The corresponding ACC and F1 differences are 1.33 and 2.34 percentage points.
  • Limitations and future work: The authors plan acquisition-aware temporal standardization to address differences in repetition time, scan duration, and temporal sampling across sites.They also identify federated learning as a possible complementary setting for distributed multi-center development.

6. Conclusion

The conclusion presents M2LG-DG as a multimodal local-global framework for source-only cross-site MDD classification using rs-fMRI and non-imaging features. Results across held-out REST-meta-MDD and ABIDE sites support its effectiveness across the evaluated target domains.

  • Conclusion: M2LG-DG addresses cross-site MDD classification by combining rs-fMRI and non-imaging features in a multimodal local-global domain generalization framework.
  • Conclusion: Experiments on four held-out REST-meta-MDD sites and two held-out ABIDE sites under a source-only protocol demonstrate effectiveness across the evaluated target domains.Each target site was excluded from training, validation, and model selection.

CRediT authorship contribution statement

The contribution statement assigns distinct roles across writing, software, investigation, validation, supervision, review, and project administration.

  • CRediT authorship contribution statement: The authors contributed across writing, software, investigation, validation, supervision, review and editing, and project administration.
Loading 2609.09186v1…