PaperlayerView exact revision on arXiv

Source-linked AI summary

PANDA - Prototype-Anchored Alignment for Partially Unpaired Multimodal Learning, with Applications to Alzheimers MRI and TCGA Pathology

Sheethal Bhat, Mahfuzur Rahman Chowdhury, Paula Andrea Perez-Toro, Stephan Wunderlich, Rose Dawn Bharat, Siming Bayer, Andreas Maier

arXiv:2608.25970v1cs.CVcs.AI

TL;DR

PANDA addresses incomplete or absent auxiliary modalities in medical prediction by transferring their information to a primary-modality model that uses only the primary modality at inference. Across ADNI MRI classification and TCGA-Lung pathology survival tasks, prototype alignment improves corresponding unimodal baselines, though the smaller TCGA-Lung cohort has wide confidence intervals and below-conventional-significance gains.

  • Problem

    Medical multimodal prediction often lacks auxiliary measurements for many subjects and cannot assume those modalities will be available at deployment.

  • Method

    PANDA learns auxiliary class prototypes from paired data, then trains the primary encoder on all subjects by aligning it to frozen prototypes without requiring auxiliary inputs at inference.

  • Results

    Across ADNI and TCGA-Lung, PANDA improves corresponding MRI-only or WSI-only baselines, reaching AUC 0.893 on ADNI and increasing TCGA-Lung 2-year OS AUC by 3.5pp and Cox PH C-index by 9.0pts.

  • Takeaways & Limitations

    PANDA provides a deployment-oriented mechanism for leveraging partially paired, externally paired, or unpaired auxiliary modalities while requiring only the primary modality at inference.

  • Takeaways & Limitations

    ADNI evaluation is confined to one cohort, and prototype alignment mitigates but does not eliminate scanner-related bias; TCGA-Lung gains also have wide confidence intervals below conventional significance.

Abstract

from arXiv · show

Multimodal medical prediction often faces incomplete pairing: auxiliary modalities with complementary signal are available for only a subset of subjects (or none) and cannot be assumed at deployment. We introduce PANDA (Prototype Anchored Data Alignment), a two-stage framework that transfers auxiliary information to a primary-modality model without auxiliary inputs at inference. Stage 1 learns a shared embedding from the paired subset and estimates class prototypes from auxiliary modalities; Stage 2 trains the primary encoder on all subjects using cross-entropy plus alignment to the frozen prototypes. Because supervision is defined at the class-prototype level, PANDA accommodates arbitrary pairing rates, including zero subject overlap. We evaluate PANDA on two applications. On a 1,021-subject multi-scanner ADNI cohort, we perform AD/CN classification with three auxiliary modalities at distinct pairing rates: tabular scores (44.8%), FDG-PET (18.7%), and external handwriting kinematics (0% overlap). Relative to the same-backbone MRI-only baseline, PANDA attains AUC 0.868 +-0.020 (+7.9pp) and reduces 1.5T CN false positives by 24.3pp; on a fully trainable Conv5-FC3 backbone it reaches AUC 0.893 (best overall). A pairing-rate ablation shows that the joint anchor remains within seed noise from 75% to 5% pairing. On TCGA-Lung survival prediction from whole-slide images with RNA-seq as auxiliary data, PANDA improves over WSI-only on 2-year OS (AUC +3.5pp) and Cox PH (C-index +9.0pts) and outperforms full-fusion training, which underperforms WSI-only, while requiring no RNA at inference; wide confidence intervals on this smaller cohort keep the gains below conventional significance. Overall, PANDA provides a deployment-oriented mechanism for leveraging incomplete auxiliary modalities to improve primary-modality prediction.

1 Introduction

PANDA addresses incomplete multimodal pairing by replacing per-subject correspondence with class-level prototype supervision, allowing MRI-only inference. It is evaluated on ADNI classification across heterogeneous pairing regimes and on cross-domain pathology survival prediction.

  • Framework: PANDA trains auxiliary and MRI encoders jointly on paired data, freezes auxiliary class prototypes, then aligns the MRI encoder using all subjects.At inference, auxiliary encoders are discarded and prediction uses MRI alone.
  • Evaluation scope: 44.8% tabular, 18.7% PET, and 0% external handwriting pairing demonstrate PANDA across subject-level, partial, and external-cohort regimes.The external handwriting prototype has no ADNI subject overlap.
  • Results: AUC 0.893 is achieved by the prototype-aligned fully trainable encoder, exceeding the strongest unimodal baseline on ADNI.Scanner-stratified analyses report different benefits: tabular alignment reduces 1.5 T false positives, whereas PET improves 3 T discrimination.
  • Pairing robustness: The joint Tab+PET anchor remains within seed noise as pairing decreases, indicating that full subject pairing is unnecessary for this alignment.The paper attributes this behavior to class-level geometry rather than per-subject correspondences.
  • Cross-domain validation: Cross-domain validation on TCGA-Lung indicates directionally consistent gains for whole-slide-image survival prediction with RNA-seq as auxiliary data.The supplied introduction passage notes that the smaller cohort limits the strength of this evidence.

2 Related Work

Prior multimodal methods commonly rely on complete pairing, imputation, fusion, or paired alignment and may require auxiliary data at deployment. PANDA instead transfers auxiliary class geometry through frozen prototypes and separates paired alignment from all-subject classification.

  • Existing approaches: Existing approaches address missing modalities through correction, fusion, modality dropout, imputation, distillation, or meta-learning, but retain pairing or deployment requirements.These methods differ in assumptions about scanner labels, paired subjects, synthesized modalities, or complete-modality teachers.
  • Scanner robustness: PANDA aligns MRI representations to auxiliary modalities designed to be scanner-invariant, implicitly suppressing scanner-correlated variability without scanner labels or target-domain scans.The discussion identifies cognitive scores, PET metabolism, and handwriting kinematics as examples.
  • Partial pairing: Training only on fully paired subjects underperforms the unimodal baseline on ADNI and TCGA because the paired subset is smaller and biased.PANDA avoids this failure by using paired subjects for alignment and all subjects for classification.
  • Prototype learning: PANDA computes prototypes from auxiliary encoders, fixes them after Stage 1, and avoids the circularity of learnable prototypes during MRI fine-tuning.Learnable prototypes collapse onto MRI class means, eroding their independent supervisory role.

3 Dataset

The study uses a multi-scanner ADNI AD/CN cohort, auxiliary clinical, PET, and handwriting data, held-out MCI evaluation, and a cross-domain TCGA-Lung WSI/RNA-seq task.

  • ADNI cohort: 1,021 ADNI subjects comprise 297 AD and 724 CN cases, split into 844 training/validation and 177 held-out test subjects.One highest-quality earliest T1-weighted MRI per subject was selected before modeling.
  • Scanner composition: 26.7% of scans were acquired at 1.5 T and 73.3% at 3 T, with class-imbalanced field strength motivating scanner-stratified analyses.AD subjects are more represented at 1.5 T than CN subjects.
  • Auxiliary modalities: The fully paired tabular subset contains 378 training subjects, corresponding to a 44.8% pairing rate across MMSE, CDR, and FAQ.Feature scaling is fit independently within each training fold.
  • MCI evaluation: A held-out MCI cohort contains 147 subjects and is used only for post hoc evaluation of ordinal disease-severity structure.No MCI scans, labels, or clinical scores enter model fitting.
  • TCGA-Lung: TCGA-Lung evaluation uses WSI as the primary modality and bulk RNA-seq as auxiliary data for 2-year overall survival and Cox proportional-hazards endpoints.RNA-seq is available for 99.5% of subjects in both endpoint cohorts.
  • Framework setup: Stage 1 jointly trains MRI and auxiliary encoders, computes frozen class prototypes, and Stage 2 trains the MRI encoder on all 844 subjects.Inference requires only MRI after auxiliary encoders are discarded.

4 Method

PANDA resolves incomplete multimodal training by using auxiliary data to estimate class-level anchors and aligning the primary encoder to those anchors across all available subjects.

  • Deployment constraint: The method supports partially paired, sparsely paired, and zero-overlap auxiliary data while requiring only the primary modality at deployment.Auxiliary data are used only to estimate class-level anchors.
  • Two-stage training: Stage 1 computes class prototypes from auxiliary modalities, while Stage 2 aligns the primary encoder to frozen prototypes for every subject.This separates paired multimodal alignment from all-subject primary-modality classification.

4.1 Framework

PANDA uses MRI as the deployment-time primary modality while incorporating partially paired or externally sourced auxiliary information through shared class-level representations. Its encoders and frozen prototypes support alignment across paired, unpaired, and zero-overlap settings.

  • Encoder architecture: The MRI backbone maps each 128^3 T1-weighted volume to a 512-dimensional feature, then projects it into a 256-dimensional unit-normalised embedding.The classification head uses the pre-projection 512-dimensional features.
  • Data and modalities: Every subject has an MRI scan, while each auxiliary modality is observed only for a subset with modality-specific pairing rates.The study uses tabular scores at approximately 45%, FDG-PET at approximately 19%, and handwriting kinematics with zero ADNI overlap.
  • Encoder architecture: Auxiliary encoders map modality-specific inputs into 256-dimensional ℓ2-normalised embeddings on the same shared hypersphere as MRI embeddings.The auxiliary encoders include dedicated architectures for tabular data and PET, with the shared embedding used for cross-modal alignment.
  • Prototype construction: For each auxiliary modality and class, PANDA computes an ℓ2-normalised mean embedding over paired training subjects and freezes the resulting prototype after Stage 1.Frozen prototypes provide stable geometric targets during subsequent MRI training.
  • External prototype transfer: Zero-overlap handwriting prototypes are computed on the external DARWIN cohort and transferred directly to test whether subject-level correspondence is necessary.Any benefit in this regime is attributed to transferred class separation rather than per-subject cross-modal pairing.

4.2 Training and Inference scheme

PANDA trains shared multimodal representations first, then aligns an MRI-only model to frozen auxiliary class prototypes using every training subject. At inference, auxiliary encoders are discarded, and classification uses MRI alone.

  • Stage 1: joint multi-modal training and frozen prototypes: Stage 1 jointly trains MRI and auxiliary encoders with modality-specific cross-entropy and supervised cross-modal contrastive losses on available paired subsets.The resulting auxiliary encoders are frozen before prototype computation for Stage 2.
  • Stage 2: prototype-anchored MRI training: Stage 2 trains the MRI encoder on the full dataset with cross-entropy plus temperature-scaled prototype cross-entropy terms.The prototype terms pull each normalised MRI projection toward the frozen prototype of its correct class.
  • Stage 2: prototype-anchored MRI training: The same class prototype is used for every subject, so paired and unpaired subjects receive identical geometric alignment targets.This makes alignment supervision available across the entire training cohort despite incomplete auxiliary measurements.
  • Inference: At test time, PANDA discards auxiliary encoders and classifies new subjects using the MRI backbone and linear classification head without additional computational cost relative to the MRI-only baseline.The reported score blends the linear-head softmax with a prototype-cosine softmax; when α1 = 0, it reduces to the MRI-only classifier.
  • Severity-axis extension: The severity extension adds a linear head on pre-projection MRI features and trains it with masked Huber regression against graded clinical-severity composites.The default sorth composite averages z-scored FAQ and NPI, while sdiag combines CDR and negated MMSE.

4.3 Information-Theoretic Motivation

PANDA’s motivation treats auxiliary class prototypes as stable geometric summaries that can guide MRI representations even when pairing is reduced. The information-theoretic discussion is explicitly heuristic and depends on simplifying assumptions.

  • Motivation and assumptions: The proposed interpretation assumes that Stage 1 embeddings and MRI projections share class-discriminative structure, making auxiliary class geometry useful as an alignment target.The discussion is presented as intuition rather than a rigorous proof.
  • Mutual-information interpretation: The InfoNCE term alone provides a variational lower bound on mutual information between paired MRI and auxiliary embeddings.The full Stage 1 objective also contains per-modality cross-entropy terms and therefore does not directly inherit that bound.
  • Prototype separation: Under a two-class Gaussian-channel approximation, larger inter-prototype separation corresponds to a stronger anchor through the deflection coefficient.This interpretation assumes equal class-conditional variances and a positive summed variance.
  • Prototype separation: Moderate pairing reductions can leave empirical prototype separation stable or slightly improved because subsampling increases estimator variance, but excessive reduction eventually makes class-mean estimates noisy.The paper reports this transition empirically at approximately 40 paired AD subjects per fold.

5 Experimental Setup

The experiments evaluate PANDA across multi-scanner ADNI MRI classification and TCGA-Lung whole-slide survival prediction, varying auxiliary coverage while keeping deployment primary-modality-only. Pairing-rate subsampling isolates prototype-estimation effects from changes in primary training-set size.

  • ADNI preprocessing and evaluation: ADNI MRI volumes are standardised to 1 mm isotropic RAS orientation, cropped to 128^3 voxels, intensity-normalised, and augmented during training.Experiments use three fixed random seeds and 5-fold stratified cross-validation.
  • Optimisation: Stage 1 and Stage 2 use AdamW with separate learning rates, fixed prototype weights, early stopping, and ReduceLROnPlateau scheduling for MRI fine-tuning.The experiments run on two A100 GPUs, with τ = 0.07 and fixed auxiliary loss weights selected using validation AUC.
  • ADNI evaluation: ADNI baselines train on 844 subjects and evaluate on a held-out 177-subject test set with MRI-only inference, comparing standard fusion, missing-modality, distillation, and published methods.Auxiliary modalities are used during training only.
  • Pairing-rate ablation: Pairing ablations subsample paired pools at predefined fractions, recompute only prototype vectors, and leave Stage 2 training on all 844 MRI subjects unchanged.This protocol isolates prototype-estimation quality from changes in MRI training-set size.
  • TCGA-Lung evaluation: TCGA-Lung uses H&E whole-slide images encoded by UNI2 and aggregated with attention-based multiple-instance learning, with bulk RNA-seq as the auxiliary modality.The study evaluates binary 2-year overall survival and Cox survival prediction.
  • TCGA-Lung evaluation: TCGA pairing sensitivity is tested by subsampling RNA-paired subjects at 25%, 50%, and 100% while reporting classification and survival metrics.The remaining architectural and optimisation settings are unchanged across this cross-domain evaluation.

6 Results

Across held-out ADNI evaluation, PANDA generally improves MRI-based classification and representation quality while reducing scanner-associated errors, including with sparse or zero-overlap auxiliary pairing. Joint prototype alignment remains effective at low pairing rates, although modality-specific benefits differ.

  • Main Results: 0.868 ± 0.020 AUC, a +7.9 pp gain versus MRI-only, is achieved by PANDA with MRI+Tab+PET+HW and zero-overlap handwriting prototypes.The model also raises CN recall to 0.863; the gain is significant after Holm–Bonferroni correction (padj = 0.011).
  • Representation Analysis: 1.14 PCA and 0.64 t-SNE Fisher ratios for MRI+Tab+PET, versus 0.01 for MRI-only, illustrate increasingly class-separated MRI embeddings.The full model reaches 0.96 in PCA and 1.01 in t-SNE; the projections are qualitative, while test AUC provides quantitative discrimination.
  • Scanner-Stratified Analysis: 52.2% to 27.9% is the reduction in 1.5 T CN false-positive rate from MRI-only to PANDA with tabular anchoring.Tabular alignment changes AUC minimally, whereas PET increases discrimination particularly at 3 T, reaching AUC 0.886 there.
  • Scanner-Stratified Analysis: 1.5 T AUC = 0.783 and FP% = 27.0, while 3 T AUC = 0.894 and FP% = 8.3 for the full MRI+Tab+PET+HW model.The full model has the strongest overall stratified profile, though HeMIS has a lower 1.5 T FP% with substantially reduced AD sensitivity.
  • Pairing Rate Sensitivity: 0.849–0.867 AUC is maintained by the joint Tab+PET anchor from 100% to 5% pairing, within overlapping seed variance.Tabular-only alignment is stable to 50% pairing but decreases to 0.758 at 25%, approximately 95 paired subjects.

6.4 Encoder Trainability and Backbone Generalizability

Prototype alignment improves MRI classification across encoder backbones and preserves ordinal CN→AD structure, while TCGA-Lung results show gains over WSI-only and Full Fusion without RNA at inference.

  • Encoder Trainability and Backbone Generalizability: AUC 0.893 ± 0.003 is achieved by the fully trainable Conv5-FC3 model with the full four-way anchor, the cohort’s highest overall performance.The same architecture raises MRI-only AUC from 0.881 ± 0.009 to 0.893 ± 0.003 after incorporating the full anchor.
  • Zero-Shot MCI Severity Inference: 0.0023 is the MRI-only three-group Kruskal–Wallis p-value, but its AD median ties the MCI median at 0.573 and fails the monotone CN→AD chain.The four-way joint model instead passes ordering intact with medians 0.027 < 0.125 < 0.436 < 0.462 < 0.720 across CN, EMCI, MCI, LMCI, and AD.
  • Zero-Shot MCI Severity Inference: 0.060 and 0.088 are the severity-head ordering-intact Kruskal–Wallis p-values for pAD and sev score, respectively, neither conventionally significant.The supported result is monotone CN→AD median ordering, not sharp separation of MCI substages; sev score correlates strongly with pAD (r = 0.941).
  • Cross-Domain Generalisation to TCGA-Lung Survival: +3.5 percentage points AUC and +9.0 C-index points are obtained by prototype transfer on TCGA-Lung versus WSI-only, while Full Fusion does not improve over WSI-only.PANDA uses no RNA at inference; matched-fraction comparisons also show C-index 0.529 versus 0.513 for Paired-only at 25% RNA.

7 Discussion

Across ADNI and TCGA-Lung, prototype alignment transfers complementary auxiliary structure into primary-modality representations without auxiliary inputs at inference. The discussion attributes gains to geometry, while identifying pairing, scanner coverage, cohort size, and validation scope as important boundaries.

  • Cross-application interpretation: Prototype alignment improves primary-modality calibration and scanner robustness across ADNI and TCGA-Lung without requiring auxiliary inputs at inference.Each auxiliary anchor reshapes the MRI/WSI embedding space and targets complementary geometric failure modes.
  • Modality-specific effects: Tabular alignment leaves AUC approximately unchanged but improves CN recall and macro-F1 through reduced within-class scatter around clinically defined centres.This geometric restructuring is consistent with the large reduction in 1.5 T false positives, whereas PET produces the largest single AUC gain but not improved 1.5 T robustness.
  • Cross-cohort transfer: 0% subject overlap still supports transfer: the handwriting anchor yields the strongest scanner-stratified performance despite no ADNI subject contributing to its prototype.The discussion interprets this as evidence that transfer requires prototype discriminability rather than semantic modality alignment.
  • Pairing-rate robustness: The joint Tab+PET anchor remains within seed noise from 75% to 5% pairing, while tabular-only alignment degrades below approximately 95 paired subjects per fold.The authors attribute joint-anchor stability to prototype quality and suggest modest pair sampling may be preferable to maximising pairing completeness.
  • Statistical qualification: TCGA-Lung effects are directionally consistent but underpowered at available test sizes, with the Cox C-index gain failing to reach the conventional significance threshold.The reported Cox gain is +9.0 points, from 0.460 to 0.550, with p = 0.059 and overlapping per-arm 95% confidence intervals.
  • Limitations and future work: Generalisability remains limited because evaluation is confined to ADNI for AD/CN classification, and prototype alignment mitigates but does not eliminate scanner-related bias.The persistent 1.5 T/3 T gap motivates external-cohort validation and dedicated harmonisation, while sparse PET pairing limits 1.5 T robustness.

8 Conclusion

PANDA improves primary-modality prediction and scanner robustness across partially paired settings, while preserving clinically meaningful severity structure without auxiliary inputs at test time.

  • +7.9 pp AUC: PANDA reaches 0.868 versus the same-backbone MRI-only baseline on ADNI.The fully trainable Conv5-FC3 backbone reaches AUC 0.893 with PANDA, versus 0.881 without alignment.
  • >24 pp: PANDA reduces the 1.5 T cognitively normal false-positive rate on ADNI.With Conv5-FC3, PANDA approximately halves the 1.5 T false-positive rate.
  • 5–10% pairing matches the fully paired configuration within seed noise, showing that full subject overlap is unnecessary for the joint anchor.The result suggests that collecting additional modalities at low overlap can be preferable to assembling a small fully paired cohort.
  • Prototype alignment preserves the CN→AD ordinal chain in held-out MCI representations, unlike the unimodal MRI baseline.A linear severity head reads out this axis without MCI-specific gradients and with no AUC cost.
  • A geometry-transfer variant was not supported because its four-arm ablation could not distinguish it from the no-op control.The authors interpret this as consistent with training-exposure confounding rather than a geometry-transfer effect.
Loading 2608.25970v1…