PaperlayerView exact revision on arXiv

Source-linked AI summary

CoPoE: Multimodal Fusion via Decomposable Disease-Coordinate Product-of-Experts for Missing-Modality Alzheimer's Diagnosis

Chihun An, Ikbeom Jang

arXiv:2610.11394v1cs.LGcs.CV

TL;DR

Irregular modality availability complicates multimodal AD classification, while clinical categories do not fully specify molecular pathology. CoPoE fuses observed modality experts in a structured RPNS space; on ADNI it achieves the highest mean AUROC across observed subsets among standardized baselines.

  • Problem

    Clinical AD categories do not fully specify amyloid/tau pathology, and relevant measurements are collected irregularly across subjects.

    Source: §I · p. 1

  • Method

    CoPoE maps each observed modality to a Gaussian expert over RPNS coordinates and fuses only observed experts, preserving a decomposable posterior without imputing missing inputs in that path.

    Source: §I · pp. 1–2

  • Results

    Across all 15 observed-modality subsets, CoPoE has the highest mean AUROC (77.82% vs. 76.04% for the strongest baseline) and the best all-modality performance, with better ECE, Brier score, and NLL.

    Source: C. Observed-Subset AUROC · p. 6, §V · p. 8

  • Takeaways & Limitations

    PET readouts show P-block enrichment under full evidence, with tau-related signal retained without biospecimen input.

    Source: §V · p. 8

    Ablations indicate gains are not explained by generic PoE fusion or the residual branch alone, assigning distinct roles to RPNS fusion and residual classification.

    Source: F. Discussion and Limitations · pp. 7–8

Abstract

from arXiv · show

Multimodal Alzheimer's disease (AD) diagnosis benefits from integrating heterogeneous clinical, imaging, genomic, and biomarker evidence, but clinical cohorts frequently suffer from irregular modality missingness. Existing fusion methods often synthesize absent inputs, risking the introduction of artificial surrogates, or pool available signals into uninterpretable latent spaces. We present CoPoE (Disease-Coordinate Product-of-Experts), a disease-coordinate framework that maps multimodal evidence into a structured latent space partitioned into four distinct biological and clinical axes: genetic Risk, molecular Pathology, Neurodegeneration, and clinical Stage (R/P/N/S). Each observed modality parameterizes a diagonal Gaussian expert over the full RPNS vector, and a masked Product-of-Experts architecture fuses only the available modalities. Consequently, absent modalities add no factor to the fusion path, allowing the network to preserve a robust, decomposable posterior for any non-empty modality subset without synthetic imputation in the RPNS path. Through extensive missing-modality experiments on the ADNI dataset, CoPoE achieves the best all-modality performance and the highest mean AUROC across all 15 observed-subset evaluations among standardized missing-modality fusion baselines under a shared non-PET ADNI embedding benchmark, while substantially improving raw-probability ECE, Brier score, and NLL. Furthermore, PET-supervised probing shows evidence enrichment within the pathology (P) block under full modalities, with tau-related signal retained even when direct fluid biospecimen inputs are withheld. Our code is available at https://github.com/labhai/CoPoE.

I. INTRODUCTION

AD diagnosis categories do not fully specify amyloid/tau pathology, and relevant measurements are collected irregularly. CoPoE addresses missing modalities by fusing observed evidence in a structured RPNS space and evaluating classification across observed subsets.

I. INTRODUCTION

  • Clinical CN/MCI/AD categories do not fully specify molecular pathology, while the measurements that do are collected irregularly across subjects.AD diagnosis here denotes computational classification, not a deployable clinical diagnostic system or a new taxonomy.Source: §I · p. 1
  • CoPoE maps each observed modality to a Gaussian expert over shared RPNS coordinates and fuses only observed experts, without imputing missing measurements in that path.This yields a decomposable posterior for any non-empty modality subset.Source: §I · pp. 1–2
  • RPNS uses genetic Risk, molecular Pathology, Neurodegeneration, and clinical Stage as disease-oriented coordinates, motivated by the A/T/N biomarker scheme,.The axes form a computational coordinate system, with each representing a primary biological anchor.Source: §I · p. 2
  • PET-derived amyloid and tau heads read only the fused P block, testing pathology coordinates with PET labels rather than using PET as an input modality.This probes whether the pathology coordinate captures molecular evidence.Source: §I · p. 2
  • The ADNI evaluation compares standardized missing-modality baselines, including PRA-PoE, across all 15 non-empty observed subsets on a shared complete-case panel.Component ablations assess parts of the RPNS-structured design using thresholded, ranking, and probability-quality metrics.Source: §I · p. 2

II. RELATED WORK

Prior missing-modality methods span completion, routed or pooled fusion, and related structured representations. CoPoE differs by fusing observed experts in disease-axis coordinates and testing its pathology block against held-out PET labels.

A. Missing-Modality Fusion

  • Missing-modality fusion includes hetero-modal aggregation, shared–specific modeling, transformer fusion, modality completion, and MoE routing –.These methods serve as controlled fusion baselines.Source: A. Missing-Modality Fusion · p. 2
  • Unlike CoPoE’s disease-axis posteriors, prior methods produce completion outputs, routed mixtures, interaction weights, or generic latent representations; imaging-genetics work assumes paired modalities.PRA-PoE combines prototype-anchored representation alignment, feature completion, and uncertainty-aware PoE; the comparison adapts its modules to shared non-PET embeddings and evaluation conditions.Source: A. Missing-Modality Fusion · p. 2

B. Structured, Biomarker-Tested Representations

CoPoE couples a disease-coordinate representation with observed-expert fusion and PET-based tests of the pathology block. Its structured probabilistic fusion replaces unstructured concatenation of available embeddings.

B. Structured, Biomarker-Tested Representations

  • Concept bottlenecks rely on dense concept annotations that are rare in heterogeneous AD cohorts, and interpretable imaging-genetics models generally target complete modality sets.CoPoE instead uses PET-derived amyloid and tau labels to test pathology-block enrichment, including in the no-biospecimen setting.Source: B. Structured, Biomarker-Tested… · p. 2

A. Overview

  • Each available modality maps to a Gaussian expert over shared RPNS coordinates, whose observed experts form a posterior; PET-supervised amyloid and tau heads read the P block.Classification uses the fused mean and a small residual branch over frozen embeddings.Source: A. Overview · p. 2, Figure 2 · p. 3
  • Unlike concatenation with zeros or masks for absent channels, CoPoE maps each available embedding to a probabilistic distribution over the same R/P/N/S coordinates.Only observed Gaussian experts are multiplied to obtain the fused posterior.Source: A. Overview · pp. 2–3

B. RPNS Gaussian Experts

CoPoE represents each modality as a Gaussian expert over shared RPNS coordinates, then fuses only the experts corresponding to observed modalities. The resulting posterior remains decomposable across non-empty subsets.

B. RPNS Gaussian Experts

  • Each modality encoder produces a fixed 128-dimensional embedding and is frozen before fusion; absent modalities produce no placeholder embedding in the RPNS path.The absent modality’s encoder is not evaluated.Source: B. RPNS Gaussian Experts · p. 3
  • RPNS coordinates anchor genetic Risk, molecular Pathology, Neurodegeneration, and clinical Stage; the main 48-dimensional configuration allocates 8/16/16/8 dimensions.Only the pathology coordinate is externally probed with PET labels; the other blocks impose disease-oriented structure rather than independently validated variables.Source: B. RPNS Gaussian Experts · p. 3
  • Each observed modality parameterizes a diagonal Gaussian expert over the full RPNS vector, with its mean estimating the state and variance controlling coordinate-wise precision.Experts span all four blocks, allowing remaining modalities to contribute to pathology when biospecimen is absent.Source: B. RPNS Gaussian Experts · p. 3

C. Observed-Expert Masked Gaussian PoE Fusion

  • For each subject, CoPoE fuses only experts from the observed modality set; missing modalities contribute no factor and are not imputed, synthesized, or represented by learned missing tokens.A zero-mean diagonal Gaussian prior has learnable per-coordinate variance.Source: C. Observed-Expert Masked Gaussian PoE… · pp. 3–4
  • Diagonal Gaussian priors and experts make fusion closed-form in each coordinate, yielding a prior-regularized, precision-weighted combination of observed experts.The prior shrinks estimates with few experts, while a confident added expert increases fused precision; the posterior stays decomposable into RPNS blocks for every non-empty subset.Source: C. Observed-Expert Masked Gaussian PoE… · p. 4

D. Block Alignment and Pathology Readout

CoPoE encourages selected modality-block specialization with soft alignment, then evaluates molecular-pathology readouts independently using PET-derived labels.

D. Block Alignment and Pathology Readout

  • Image evidence is softly aligned with neurodegeneration, and biospecimen evidence with pathology; genomic and clinical signals remain unassigned because they span several axes.The contrastive objective aligns each specialist modality’s block mean with the corresponding fused block mean.Source: D. Block Alignment and Pathology Readout · p. 4
  • Alignment pairs with no observed modality examples in a batch are skipped, while PET-derived amyloid and tau labels provide an independent test of pathology.The alignment loss encourages block specialization; the PET readouts test molecular pathology separately.Source: D. Block Alignment and Pathology Readout · p. 4
  • Two binary pathology heads read only the fused P block and are trained only for subjects with the corresponding PET label.They serve as molecular readouts, not alternative CN/MCI/AD classifications; biospecimen inputs may contain fluid assays, whereas the targets are PET-derived.Source: D. Block Alignment and Pathology Readout · p. 4

E. Classification, Training, and Inference

Classification uses the fused RPNS representation alongside a classification-only residual branch, while training combines classification, PET-readout, regularization, alignment, and subset objectives. At inference, the model fuses only observed modalities and returns class probabilities and the fused posterior.

E. Classification, Training, and Inference

  • The layer-normalized fused RPNS mean feeds a compact classifier, with a classification-only residual branch over frozen modality embeddings to avoid forcing all classification information through the RPNS bottleneck.Unobserved embeddings are zero-filled only for this residual branch; the convention does not enter the RPNS product or PET heads.Source: E. Classification, Training, and… · p. 4
  • Training minimizes a combined objective with classification, PET amyloid/tau, KL regularization, block alignment, and random-subset classification losses.PET losses apply only when the corresponding label is available; subset classification uses the original label.Source: E. Classification, Training, and… · p. 4
  • A low-weight cascade prior biases generalist experts’ block-wise precision distributions toward fixed targets, while specialists use image-to-neurodegeneration and biospecimen-to-pathology anchors.The paper describes this prior as a fixed implementation regularizer, not a separate methodological contribution.Source: E. Classification, Training, and… · p. 4
  • At inference, only observed-modality encoders and experts are used, so the same fusion rule handles any non-empty subset without network changes or a separate imputation step.Outputs include class probabilities, the fused RPNS posterior, and optional amyloid/tau probabilities from the P block.Source: E. Classification, Training, and… · p. 4

A. Experimental Setup

The study evaluates CoPoE in a standardized ADNI missing-modality fusion benchmark using shared frozen non-PET embeddings, subject-disjoint splits, and observed-subset comparisons. It assesses classification, probability quality, and PET-supervised pathology readouts under a common protocol.

A. Experimental Setup

  • All-modality results are reported for CN/MCI/AD classification, with BAcc, macro-F1, and AUROC in percent and raw ECE, Brier, and NLL; values are mean ± std over five seeds.Table I marks best and second-best values and indicates Holm-corrected paired patient-level bootstrap significance for CoPoE versus each baseline.Source: Table I · p. 5
  • ADNI supplies genomic, biospecimen, image, and clinical modalities; 3,018 subject-disjoint training profiles include 992 all-four and 2,026 naturally partial profiles.The 1,292 all-four subjects provide 100 validation and 200 test subjects; all-subset evaluation uses common complete-case validation/test subjects, with training subset loss sampling only naturally available modalities.Source: A. Experimental Setup · p. 5
  • The inputs are structural MRI, SNP dosages, clinical records, and CSF/biofluid assays; clinical features exclude cognitive or functional composites and diagnosis-defining fields.The benchmark uses frozen 128-dimensional modality embeddings, and encoder pretraining and feature-selection statistics use training data only.Source: A. Experimental Setup · p. 5
  • Amyloid and tau are PET-derived supervision labels for pathology readouts, not input modalities or class labels; no PET-derived feature enters fusion.The no-biospecimen condition removes direct fluid-biomarker inputs when assessing PET readout behavior.Source: A. Experimental Setup · p. 5
  • Classification metrics are BAcc, macro-F1, and macro one-vs-rest AUROC; probability quality uses 15-bin ECE, Brier, and NLL, while PET readouts use AUROC/AUPRC.Significance uses paired patient-level bootstrap tests with Holm correction, except the 15-subset AUROC comparison, which uses paired-block one-sided Wilcoxon tests with Holm correction.Source: A. Experimental Setup · pp. 5–6
  • The benchmark compares standardized fusion baselines using shared frozen non-PET embeddings, subject-disjoint splits, observed subsets, five seeds, and random-subset training augmentation.Baselines retain their published fusion and missing-modality handling; mean validation BAcc across 15 subsets selects models, and all-subset AUROC is the primary threshold-independent comparison.Source: A. Experimental Setup · p. 6

B. All-Modality Multi-Metric Overview

CoPoE leads on all six all-modality metrics and has the highest mean AUROC across the 15 observed-modality subsets, though it does not lead in every subset.

B. All-Modality Multi-Metric Overview

  • Against the strongest non-CoPoE baseline per metric, CoPoE improves BAcc by 5.42 percentage points, macro-F1 by 6.17 percentage points, and AUROC by 2.08 percentage points, while reducing ECE by 0.063, Brier by 0.074, and NLL by 0.180.Patient-level bootstrap tests with Holm correction found significant BAcc, macro-F1, Brier, and NLL gains over all baselines, and AUROC gains over most.Source: B. All-Modality Multi-Metric Overview · p. 6

C. Observed-Subset AUROC

  • On a common complete-case test panel, CoPoE achieves the highest AUROC in 12 of 15 subsets and the best subset average, 77.82% vs. 76.04% for ShaSpec.The same test subjects are used across subset rows, so differences reflect available input evidence rather than cohort composition.Source: C. Observed-Subset AUROC · p. 6
  • The three exceptions are biospecimen-only for MUSE and image-only and image+clinical for ShaSpec; a paired Holm-corrected test found a 1.78 percentage-point mean AUROC gain over ShaSpec (pHolm = 2.1×10−3).CoPoE is not uniformly dominant with minimal evidence; validation selection uses BAcc rather than AUROC.Source: C. Observed-Subset AUROC · p. 6

D. PET Biomarker Readout Analysis

PET-supervised probes find the strongest full-modality amyloid and tau readouts in the pathology block, with tau signal retained when biospecimens are withheld. Ablations assess RPNS allocation and component contributions.

D. PET Biomarker Readout Analysis

  • PET probes compare RPNS blocks, the full vector, and diagnosis logits under held-out labels, testing pathology-block enrichment rather than whether it emerges unsupervised.The probes control readout architecture and label availability, not the placement of PET supervision; test-label positive prevalence is 50.9% for amyloid and 33.0% for tau.Source: D. PET Biomarker Readout Analysis · p. 7
  • 91.83 AUROC and 92.82 AUPRC for amyloid, and 75.25 AUROC and 64.66 AUPRC for tau: P is the strongest readout for both under all modalities.Without biospecimens, tau remains strongest in P, while amyloid is best predicted by CN/MCI/AD logits; P-block amyloid remains above chance at 68.90 AUROC.Source: D. PET Biomarker Readout Analysis · p. 7

E. Ablation Studies

  • At dz = 48, the 8/16/16/8 allocation outperforms symmetric 12/12/12/12 and reversed 16/8/8/16 variants, and larger widths do not close the gap.The authors describe this as a sensitivity sweep, not evidence of unique optimality.Source: E. Ablation Studies · p. 7, Table IV · p. 7
  • Removing the RPNS partition degrades BAcc and AUROC; removing the residual pathway mainly degrades BAcc, Brier, and NLL, while removing biomarker heads slightly worsens Brier and NLL.The partition control keeps capacity and several components but also disables block-specific anchoring and cascade losses, so it does not isolate partition geometry; joint partition and residual removal gives the largest degradation.Source: E. Ablation Studies · p. 7, Table V · p. 7

F. Discussion and Limitations

CoPoE’s advantage grows as complementary evidence is added to the RPNS posterior; its RPNS fusion layer accumulates evidence across subsets, while the residual branch improves classification and probability quality. These findings are bounded by the standardized ADNI non-PET embedding benchmark and by the current validation and modeling scope.

F. Discussion and Limitations

  • The advantage grows as complementary evidence is added; gains are not explained by generic PoE fusion or the residual branch alone.The RPNS fusion layer provides a subset-invariant evidence accumulator, while the residual branch improves final classification and probability quality.Source: F. Discussion and Limitations · pp. 7–8
  • The standardized ADNI non-PET embedding benchmark does not establish deployment performance under natural acquisition policies or external cohort shifts.Independent validation of the R, N, and S blocks remains future work; PET supervision directly tests the pathology block.Source: F. Discussion and Limitations · p. 8
  • Diagonal Gaussian experts preserve closed-form fusion across modality subsets but do not model cross-coordinate covariance.External cohorts with natural missingness and broader PET-labeled samples are needed to assess site robustness, missingness mechanisms, and subgroup performance.Source: F. Discussion and Limitations · p. 8

V. CONCLUSION

  • CoPoE achieves the best all-modality performance and highest mean AUROC across 15 observed-modality subsets, with substantially better ECE, Brier score, and NLL.PET readouts show P-block enrichment under full evidence, with tau-related signal retained without biospecimen input.Source: §V · p. 8
Loading 2610.11394v1…