PaperlayerView exact revision on arXiv

Source-linked AI summary

Anchor-driven Multi-modal Multi-scale Expert Selection for Survival Prediction

Tao Zhou, Ying Hu, Huazhu Fu, Yi Zhou, Xiao-Jun Wu, Haibin Ling

arXiv:2610.07694v1cs.CV

TL;DR

Existing multimodal survival models lack explicit alignment between WSI morphology and transcriptomic signals and often do not adaptively prioritize tissue scales for individual patients. AM2ES uses patient-contextualized anchors and hierarchical expert selection to address these gaps; the authors report consistent outperformance of state-of-the-art methods across multiple cancer cohorts, with fine-grained interpretability.

  • Problem

    Existing methods typically lack explicit alignment between morphological and molecular features and mechanisms to adaptively prioritize prognostic tissue scales for individual patients.

    Source: §1 · pp. 1–2

  • Method

    AM2ES uses patient-contextualized semantic anchors to align transcriptomic and multi-scale pathology features, then H-MoE filters features within scales and adaptively integrates evidence across scales.

    Source: §1 · p. 2

  • Results

    AM2ES consistently outperforms state-of-the-art methods across multiple cancer cohorts and provides fine-grained interpretability of its decision process.

    Source: §5 · p. 13

  • Takeaways & Limitations

    Visualizations trace anchor-driven semantic shifts and patient-specific scale preferences in the model's decisions.

    Source: §1 · p. 2

    The survival setup represents censoring as right-censored observations, with c_i=1 for censored and c_i=0 for an uncensored death.

    Source: §3.1 · p. 4

Abstract

from arXiv · show

The integrative analysis of histopathological Whole-Slide Images (WSIs) and transcriptomic profiles holds significant promise for cancer survival prediction. However, existing methods typically project multi-modal features directly into a shared latent space without explicit alignment, leading to the entanglement of mismatched morphological cues and molecular signals. Furthermore, current fusion strategies often treat the extreme spatial heterogeneity of WSIs uniformly, lacking mechanisms to adaptively prioritize clinically relevant tissue scales for individual patients. To address these limitations, we propose an Anchor-driven Multi-modal Multi-scale Expert Selection (AM$^2$ES) framework for survival prediction. Specifically, we present an Anchor-driven Multi-modal Fusion (AMF) module, which introduces learnable semantic anchors as cross-modal mediators to bridge the semantic gap by enforcing a structurally regularized alignment between transcriptomic features and multi-scale pathology representations. Built upon this aligned semantic space, we further design a Hierarchical Mixture-of-Experts (H-MoE) selection module to decouple the hierarchical prognostic selection process. Mimicking the pathologist's diagnostic workflow, H-MoE performs (i) Intra-scale Expert Filtering to discriminatively identify salient tumor regions within each magnification, and (ii) Inter-scale Hierarchy Routing to dynamically weight and select the most informative resolution levels. Extensive experiments on multiple TCGA cancer cohorts demonstrate that our AM$^2$ES achieves state-of-the-art performance while offering fine-grained interpretability by visualizing how specific molecular pathways drive the expert routing decisions across tissue scales. The code will be released at https://github.com/taozh2017/AM2ES.

1 INTRODUCTION

Existing WSI–omics models struggle to align morphology with molecular signals while adapting to the prognostic hierarchy across tissue scales. AM2ES separates semantic alignment from patient-specific expert selection, and the authors report state-of-the-art performance across multiple TCGA cohorts.

1 INTRODUCTION

  • Flat multimodal fusion entangles semantic alignment with prognostic selection, which can lead to modality overfitting, feature collapse, and attention on background noise.The model must simultaneously match tissue patches to pathways and identify survival-driving features.Source: §1 · pp. 1–2
  • Existing early-fusion and fixed hierarchical-fusion strategies do not adaptively prioritize the resolutions most relevant to each heterogeneous patient's prognosis.Early fusion can confuse scales and propagate noise, while fixed cascades assume the same feature-importance order for every patient.Source: §1 · p. 2
  • AMF uses patient-contextualized semantic anchors and orthogonality and sparsity constraints to align pathological features with corresponding transcriptomic profiles.The anchors are conditioned on global patient context and serve as semantic mediators.Source: §1 · p. 2
  • H-MoE filters salient representations within each resolution, then uses rank-based alignment and feature-level mixture-of-experts fusion to recalibrate evidence across scales.The two stages are intra-scale filtering and inter-scale fusion.Source: §1 · p. 2
  • AM2ES achieves state-of-the-art performance across multiple TCGA cohorts and offers fine-grained visual interpretability of semantic shifts and patient-specific scale preferences.The visualizations trace how anchors shift semantics and how patients' scale preferences vary.Source: §1 · p. 2

2 RELATED WORK

Prior work develops multimodal fusion, multi-scale pathology representations, anchors, and mixture-of-experts methods, but the cited approaches leave gaps in explicit semantic mediation and patient-specific scale selection.

2.1 Multimodal Learning for Cancer Prognosis

  • Transformer-based multimodal methods such as MCAT and MOTCat use co-attention, but typically intertwine representation learning and prognostic fusion without explicit semantic mediators.Earlier fusion approaches include concatenation,, CNN-based joint modeling, and tensor-based gating.Source: §2.1 · p. 2

2.2 Multi-scale Representation in Computational Pathology

  • Multi-scale pathology methods range from MIL and graph models to Transformers and prompts, yet most use fixed fusion strategies rather than identifying each patient's prognostically informative scale.Biological signals span scales, from global tissue phenotypes at 5× to cellular atypia at 20×.Source: §2.2 · pp. 2–3

2.3 Anchor-driven Prototypes and Multi-modal Fusion

  • Existing anchor-based survival methods typically use fixed anchors, whereas AMF introduces dynamic, patient-contextualized anchors conditioned on global RNA profiles to align morphology through structural regularization.Prior dynamic prototype updating has rarely been adapted to patient-specific cross-modal fusion in survival tasks.Source: §2.3 · p. 3

2.4 Mixture-of-Experts Selection Mechanisms

  • Although MoE is used for conditional computation and medical-image variability, prior applications generally lack explicit semantic disentanglement and scale-aware or survival-aware expert selection in WSI–omics models.The cited medical-imaging applications generally use single-stage structures.Source: §2.4 · p. 3

3.1 Problem Definition

The task predicts survival from each patient's multi-scale WSI and genomic profile, using observed time and censoring status as outcomes. It adopts a discrete-time hazard formulation.

3.1 Problem Definition

  • Each patient is represented by multi-scale pathological WSIs and a genomic profile, with survival time and censoring status recorded as the outcome.The setup defines c_i=1 as right-censored and c_i=0 as an uncensored event (death).Source: §3.1 · p. 4
  • The continuous time horizon is partitioned into non-overlapping intervals, and the model estimates the conditional event probability for each interval.Interval boundaries are determined by survival-time quantiles from the uncensored subset; interval hazards yield the survival function.Source: §3.1 · p. 4

3.2 Overview of Architecture

AM2ES encodes multi-scale WSI and RNA data, aligns their representations with anchors, then selects and fuses prognostic features across scales.

3.2 Overview of Architecture

  • Independent encoders extract WSI features at 5×, 10×, and 20× and RNA embeddings, while Transformer-based clustering summarizes WSI features into semantic prototypes.These representations feed the anchor-driven fusion and expert-selection stages.Source: §3.2 · p. 4
  • AMF uses learnable anchors to align image prototypes and genomic embeddings in a shared representation space.Source: §3.2 · p. 4
  • H-MoE selects informative features within each scale and adaptively integrates them across scales for survival prediction.Source: §3.2 · p. 4

3.3 Multi-scale Histopathology and Genomic Feature Encoding

The framework encodes WSIs at multiple magnifications into representative morphological prototypes and preserves biological organization in the genomic representation. These structured inputs feed the subsequent anchor-driven fusion module.

3.3 Multi-scale Histopathology and Genomic Feature Encoding

  • Separate encoding streams produce multi-scale morphological representations from WSIs and biologically grouped descriptors from genomic profiles for subsequent fusion.The framework hierarchically integrates multiscale features rather than prematurely blending resolutions through heuristic pooling.Source: §3.3 · p. 4

3.3.1 Multi-scale WSI Representation Learning

  • At each magnification, a Transformer-based clustering module uses learnable queries and cross-attention to soft-assign patch embeddings to representative semantic prototypes.End-to-end optimization lets the queries capture distinct morphological patterns without heuristic clustering, preserving coarse-to-fine structure while controlling downstream computational complexity.Source: §3.3.1 · p. 4

3.3.2 Biologically-grouped Genomic Encoding

  • Genomic profiles are partitioned into biologically defined groups, encoded with category-specific networks, and projected into a shared latent dimension.The resulting genomic bag preserves pathway-level organization for anchor-driven fusion.Source: §3.3.2 · p. 4

3.4 Anchor-driven Multi-modal Fusion

AMF uses patient-conditioned anchors to align histological and genomic evidence at each scale, addressing the entangled representations associated with conventional fusion. It preserves aligned candidates for later selection and regularizes anchors for diversity and compact activation.

3.4 Anchor-driven Multi-modal Fusion

  • Learnable, scale-specific, patient-conditioned anchors bridge tissue representations and gene-expression embeddings, with anchor-driven integration yielding a more discriminative latent space that separates survival risk profiles.Conventional co-attention and concatenation are described as producing entangled spaces in which patients across risk levels are not well separated.Source: §3.4 · pp. 4–5, Figure 3 · p. 5

3.4.1 Dynamic Anchor Generation

  • At each scale, reference anchors are conditioned on global genomic and scale-specific morphological contexts to generate anchors tailored to the patient and resolution.The contexts are derived from encoded features using average pooling, and a dynamic calibration network injects these signals into reference anchors.Source: §3.4.1 · p. 5

3.4.2 Dual-stream Anchor Attention

  • Each anchor queries both morphological prototypes and the genomic bag, and the retrieved evidence is fused into aligned cross-modal features while retaining scale-specific candidates for later selection.Morphological attention is normalized over prototypes; the resulting distinct feature sets support context-aware selection within and across scales.Source: §3.4.2 · p. 5

3.4.3 Regularization for Stable Anchor Fusion

  • Orthogonality regularization encourages distinct semantic directions among anchors, while an ℓ1 penalty encourages compact anchor activations.The total anchor regularization is aggregated across all three magnification levels, with β balancing the sparsity penalty.Source: §3.4.3 · pp. 5–6

3.5 Hierarchical Mixture-of-Experts Strategy

H-MoE selects prognostic evidence in two stages: it filters multimodal descriptors within each resolution, then dynamically recalibrates evidence across resolutions. Its rank-based alignment links descriptors at corresponding importance ranks across scales.

3.5 Hierarchical Mixture-of-Experts Strategy

  • H-MoE sparsely filters multimodal representations within each resolution, then dynamically recalibrates and integrates evidence across resolutions to prioritize patient-relevant magnifications.The framework presents these as intra-scale expert selection and inter-scale MoE fusion stages.Source: §3.5 · p. 6

3.5.1 Intra-scale Experts Scoring and Selection

  • Within each scale, lightweight MLP experts score descriptors through sparse top-k routing, and the highest-scoring candidates are retained for inter-scale fusion.The experts are described as specializing in distinct risk patterns, and the selected descriptors are ranked by their routed ensemble scores.Source: §3.5.1 · p. 6

3.5.2 Inter-Scale Mixture-of-Experts Fusion

  • Across scales, descriptors with the same rank are concatenated, then a feature-level MoE generates a context-aware channel mask and projects the masked vector back to the latent space.The fused tokens are averaged to derive the patient-level holistic representation.Source: §3.5.2 · p. 6

3.6 Total Objective Function

The survival prediction head maps the patient-level representation to interval-wise hazard probabilities. Training combines the survival loss with anchor regularization in a weighted objective.

3.6 Total Objective Function

  • A fully connected layer with sigmoid activation maps the patient-level representation to interval-wise hazard probabilities.The prediction head uses learnable weight and bias parameters.Source: §3.6 · p. 6
  • The final training objective combines the survival loss and anchor regularization, with λ balancing the prediction task against regularization constraints.The survival loss is defined using each patient's observed time interval and censoring indicator.Source: §3.6 · p. 6

4 EXPERIMENTS AND RESULTS

Across five TCGA cohorts, AM2ES leads the reported C-index comparison, and Kaplan–Meier analyses show risk-group separation. Ablations and sensitivity analyses examine its multimodal, multi-scale, anchor, and expert-routing components.

4.1.1 Datasets

  • Experiments used five TCGA cohorts with paired WSI and genomic profiles, patient-level 5-fold cross-validation, and comparisons spanning unimodal and multimodal methods at different visual scales.WSI patches were extracted at 5×, 10×, and 20×; genomic profiles were organized into six functional categories, and WSI-based comparisons used the same UNI features and data splits.Source: §4.1.1 · p. 7, §4.1.3 · p. 7, §4.2.1 · p. 7

4.2.2 Result Comparison

  • 0.735 average C-index: AM2ES outperformed the second-best method, Survpath (0.709), across five TCGA cohorts.The study reports the best overall performance against existing state-of-the-art methods.Source: §4.2.2 · pp. 7–8, Table 1 · p. 8

4.2.3 Kaplan-Meier Survival Analysis

  • Across five TCGA cohorts, Kaplan–Meier analyses showed significant risk stratification, with clearer, more stable separation for AM2ES than Survpath and HiMT.Patients were divided at the median predicted risk score; the figure shows high-risk patients in red and low-risk patients in blue, with 95% confidence intervals.Source: Figure 4 · p. 8, §4.2.3 · p. 9

4.2.2 Result Comparison

  • AM2ES ranked best on both independent CPTAC cohorts in zero-shot evaluation, although absolute C-indices were lower than in TCGA evaluations.TCGA-trained models were applied without fine-tuning or retraining; the authors cite batch effects, cohort differences, and limited sample sizes as expected challenges.Source: Table 2 · p. 9, §4.2.2 · p. 9

4.3.1 Effectiveness of Each Component

  • Feature concatenation raised average C-index from 0.620 for either unimodal baseline to 0.650, while the full 5×, 10×, and 20× hierarchy reached 0.708 versus 0.683 at 20× alone.The comparison attributes complementary prognostic signals to genomic and pathological inputs and reports a further gain with all three visual scales.Source: §4.3.1 · p. 9
  • 0.714 C-index: anchor-driven fusion outperformed Cross-Attention (0.689) and Bilinear Pooling (0.682) under the same multi-scale setting.The authors attribute the improvement to patient-specific anchors aligning WSI features with genomic embeddings more effectively than generic attention maps.Source: §4.3.1 · p. 9, Table 3 · p. 10
  • Top-K selection reached 0.714 average C-index versus 0.686 for GAP and 0.691 for attention pooling; dynamic MoE routing reached 0.714 versus 0.687 for static concatenation.The inter-scale alignment experiments also tested global pooling, retaining descriptors in original order, and random alignment.Source: §4.3.1 · pp. 9–10

4.3.2 Hyperparameter Sensitivity

  • K = 16 anchors and K′ = 4 retained instances were the reported preferred settings; removing anchor regularization lowered average C-index from 0.714 to 0.700.The authors report diminishing returns at K = 32, declining performance when K′ increases to 8 or 16, and use of orthogonality and sparsity constraints in anchor regularization.Source: Table 4 · p. 10, Table 5 · p. 10, Table 6 · p. 10

4.4 Interpretability and Case Studies

The case studies show that AM2ES highlights subtype-consistent morphology and biologically meaningful genomic signals, while its cohort-level attributions vary by cancer type. The complexity discussion describes prototype-based sequence reduction and reports model size and inference time.

4.4.1 Morphological Attention Analysis

  • The GBM and ODG visualizations highlight distinct, subtype-consistent morphology across magnifications, without requiring pixel-level annotations.GBM attention concentrates on tumor-dense regions with disrupted architecture and nuclear atypia; ODG attention is more distributed and includes the characteristic “fried-egg” appearance.Source: §4.4.1 · pp. 11–12

4.4.2 Importance of Genomic Features

  • For the GBM case, SPP1 contributes positively to predicted risk, while PRF1 receives negative risk attribution in the ODG case.The authors interpret these attributions as consistent with known glioma biology, including immune-related signals in ODG.Source: §4.4.2 · p. 12

4.4.3 Multimodal Synergy

  • The GBM and ODG case studies pair aggressive or lower-grade tissue morphology with genomic signals, which the authors describe as complementary information for survival prediction.Examples include SPP1 in the GBM case and PRF1 in the ODG case.Source: §4.4.3 · p. 12

4.4.4 Importance of Global Pan-Cancer Features

  • Across five TCGA cohorts, top-ranked molecular features include both RNA-seq signals and CNVs, with prominent features varying by cancer type.GBMLGG ranks SPP1 highly, while LUAD and UCEC highlight different features; the authors also identify NAMPT and CDK4 CNV in glioma.Source: §4.4.4 · pp. 12–13

4.5 Discussion of Computational Complexity

  • Prototype queries aggregate patches at each scale, reducing attention aggregation complexity from O((M_i^s)^2) to O(P_i^s · M_i^s).The authors state that P_i^s is much smaller than M_i^s, making aggregation scale linearly with patch count.Source: §4.5 · p. 13
  • AM2ES has 12.66M parameters and an inference time of 9.5 ms per WSI, which the authors describe as a favorable efficiency–performance trade-off.The comparison is reported for a single NVIDIA GeForce RTX 4090 GPU with 24 GB of memory.Source: §4.5 · p. 13

5 CONCLUSION

AM2ES combines dynamic semantic anchors for cross-modal alignment with hierarchical expert selection across tissue scales. Experiments across multiple cancer cohorts show consistent outperformance of state-of-the-art methods, and ablations support the contribution of each key component.

5 CONCLUSION

  • AM2ES uses dynamic semantic anchors to mediate integration of transcriptomic profiles and multi-scale pathological features.Its H-MoE first selects informative features within each scale, then adaptively integrates evidence across scales for each patient.Source: §5 · p. 13
  • Across multiple cancer cohorts, AM2ES consistently outperforms state-of-the-art methods and provides fine-grained interpretability of its decision process.Source: §5 · p. 13
  • Ablation studies confirm contributions from each key component of AM2ES.Source: §5 · p. 13
Loading 2610.07694v1…