PaperlayerView exact revision on arXiv

Source-linked AI summary

Anatomy-aware Fine-grained Multimodal Fusion for Laryngopharyngeal Cancer T-Staging Prediction Using CT and Radiology Report

Xingyue Zhao, Yanzhou Su, Fang Zhang, Zhanghexuan Ji, Yirui Wang, Dazhou Guo, Sibo Ju, Yuehua Cheng, Yuzhen Chen, Ming Feng, Le Lu, Tsung-Ying Ho, Jian Wang, Dakai Jin, Na Shen

arXiv:2610.06837v1cs.CV

TL;DR

Laryngopharyngeal T-staging depends on complex invasion patterns, while existing approaches can lack explicit anatomical modeling or precise alignment between organs and report descriptions. The paper proposes an anatomy-aware multimodal framework that integrates CT-derived organ features with report evidence through graph construction, organ-level alignment, and report-guided refinement. The authors report that it consistently outperforms state-of-the-art unimodal and multimodal methods.

  • Problem

    Existing approaches can underrepresent anatomical invasion relationships or struggle to align individual structures with their corresponding radiology-report descriptions.

    Source: §I · p. 2

  • Method

    The framework represents CT-derived organ features in an anatomy-structured graph, aligns each organ with report evidence, and refines the graph using report-derived invasion information.

    Source: §I · p. 2, §VI · p. 12

  • Results

    The authors report consistent outperformance of state-of-the-art unimodal and multimodal methods on laryngopharyngeal cancer T-staging.

    Source: §VI · p. 12

  • Takeaways & Limitations

    The framework captures organ-specific invasion patterns together with their textual evidence for laryngopharyngeal cancer T-staging.

    Source: §VI · p. 12

Abstract

from arXiv · show

Accurate T-staging is crucial for guiding personalized treatment strategies for laryngopharyngeal cancer. However, current clinical practice relies on invasive biopsy procedures, whereas CT-based staging remains challenging due to the complex patterns of tumor invasion. Recent computer-aided approaches face two key challenges: 1) Structural relationship modeling: existing methods underrepresent anatomically structured patterns of tumor invasion, as they either process whole CT volumes without tumor-specific anatomical constraints or rely on labor-intensive tumor segmentation. 2) Fine-grained cross-modal alignment: while radiology reports contain organ-specific invasion details, current methods that apply global feature fusion struggle to accurately align individual anatomical structures with their corresponding textual descriptions. To address these issues, we propose an anatomy-aware multimodal framework that integrates organ-level CT context and radiology reports into a unified representation for laryngopharyngeal T-staging. The framework first constructs an Anatomy-Structured Organ Graph (AOG) that captures invasion patterns between primary sites and surrounding organs, then performs Organ-Anchored Cross-Modal Alignment (OCA) so that each organ node aggregates textual evidence from the radiology report, and finally refines this graph representation by injecting organ-specific invasion cues extracted from the report via Report-Enhanced Graph-Refinement (REG), yielding a multimodal organ graph that combines spatial and textual evidence. Extensive experiments demonstrate that the proposed framework achieves superior performance in T-staging of laryngopharyngeal cancer.

I. INTRODUCTION

Laryngopharyngeal cancer T-staging guides treatment, but CT staging must contend with complex invasion patterns and methods that lack explicit anatomical relationships or precise organ-to-report alignment. The paper proposes an anatomy-aware multimodal framework and evaluates it on a multi-center cohort.

I. INTRODUCTION

  • Invasive biopsy is the current reference standard, while CT review requires tracking tumor extension across many slices and can be subjective, especially in borderline cases.T-stage matters for treatment choices: early disease is often managed with partial laryngectomy, laser resection, or radiotherapy, whereas advanced disease often requires total laryngectomy and adjuvant therapies.Source: §I · pp. 1–2
  • Existing approaches either process whole CT volumes without tumor-specific anatomical constraints or rely on labor-intensive tumor segmentation, while global fusion struggles to align structures with relevant report descriptions.The paper frames these as structural-relationship modeling and fine-grained cross-modal alignment challenges.Source: §I · p. 2
  • AOG uses pre-trained organ segmentations and clinical prior knowledge to model primary sites and potentially invaded organs without manual tumor masks.Its anatomy-guided connections represent relationships between primary sites and surrounding organs.Source: §I · p. 2
  • OCA lets each CT-derived organ node query the report for relevant textual evidence, while REG uses reported invasion information to refine the graph without discarding anatomy-based connections.The authors note that subtle or equivocal extension may be omitted from reports, so anatomy-driven connectivity is retained.Source: §I · p. 2
  • The framework combines CT-derived organ features with organ-level textual cues in an anatomy-structured graph, modeling inter-organ relationships and invasion patterns rather than the tumor in isolation.The authors report experiments on 656 patients from four hospitals across binary and three-class settings, with internal and external test sets.Source: §I · pp. 2–3

A. T-staging

  • The paper situates its work against laryngopharyngeal and head-and-neck staging studies that are single-center, limited in scale, and based on imaging features without paired radiology reports.It identifies a need to integrate anatomical relationship modeling and textual evidence for complex invasion patterns.Source: A. T-staging · p. 3

B. Multi-modal Learning

Medical image–text methods offer ways to combine visual and textual evidence, but laryngopharyngeal T-staging requires both anatomical relationship modeling and structure-specific alignment. The framework addresses these needs with an organ graph and complementary report-based modules.

B. Multi-modal Learning

  • Medical image–text approaches span segmentation and disease-detection tasks, but the paper identifies a remaining challenge in precise correspondence between anatomical structures and report descriptions for laryngopharyngeal T-staging.The proposed framework jointly models anatomy and fine-grained alignment.Source: B. Multi-modal Learning · p. 3

A. Overview

  • The framework constructs an organ-level graph from pretrained segmentations and anatomy-guided connectivity, aligns report cues with individual organs, then refines the graph using report-derived invasion statements.The overview emphasizes retaining anatomy-based risk connections during report-guided refinement.Source: A. Overview · p. 3, Figure 2 · p. 4

B. Anatomy-Structured Organ Graph

  • Existing laryngopharyngeal T-staging methods either process whole CT volumes without explicit anatomical constraints or rely on manual tumor segmentation, limiting representation of invasion relationships between primary sites and at-risk organs.The paper identifies complex invasion patterns and the small size of tumor regions as challenges for feature extraction from whole CT volumes.Source: B. Anatomy-Structured Organ Graph · pp. 3–4, 1) Organ-specific Feature Extraction · p. 4
  • AOG uses pre-trained organ segmentation and clinical prior knowledge to connect primary sites with anatomically plausible invasion targets rather than relying only on tumor masks or reported invasion.This graph is intended to encode potential spread between primary sites and surrounding structures.Source: B. Anatomy-Structured Organ Graph · p. 4

1) Organ-specific Feature Extraction:

Organ-specific CT features provide the nodes for an anatomy-informed graph. The graph encodes plausible invasion pathways and refines organ representations through attention-based message passing.

1) Organ-specific Feature Extraction:

  • Pre-trained organ segmentation provides masks for each organ in a CT volume, and a feature network extracts organ-focused representations from masked regions.The paper motivates this intermediate representation by the more consistent appearance and clearer boundaries of anatomical organs than tumors.Source: 1) Organ-specific Feature Extraction · pp. 4–5

2) Anatomy-guided Graph Construction:

  • The directed graph represents organ features as nodes and uses anatomy-informed connections to capture relationships between primary sites and potential invasion targets.The graph structure is motivated by clinical knowledge of typical spread from primary sites to surrounding structures.Source: 2) Anatomy-guided Graph Construction · p. 5
  • Graph Attention Networks pass messages along the anatomically meaningful pathways, allowing attention to vary across relationships and incorporating local context and broader invasion patterns into node features.The refined organ representations support subsequent cross-modal enhancement.Source: 2) Anatomy-guided Graph Construction · p. 5

C. Organ-Anchored Cross-Modal Alignment

Global image–text fusion can leave organ features weakly grounded in the corresponding report descriptions. OCA instead anchors cross-modal interaction at the organ level.

C. Organ-Anchored Cross-Modal Alignment

  • Global fusion encodes CT volumes and entire reports holistically, making it difficult to associate specific anatomical structures with their relevant clinical descriptions.The paper presents this as a fine-grained alignment gap in current multimodal approaches.Source: C. Organ-Anchored Cross-Modal Alignment · p. 5
  • OCA uses CT-derived organ nodes as queries over the report, selectively aggregating textual evidence into organ representations while preserving the anatomy-structured graph.This creates organ-level image–text interaction rather than coarse global fusion.Source: C. Organ-Anchored Cross-Modal Alignment · p. 5
  • The report encoder uses BioBERT, a pretrained biomedical language model, to represent domain-specific vocabulary and context-dependent anatomical descriptions.BioBERT is identified as dmis-lab/biobert-base-cased-v1.2.Source: 1) Text Feature Encoding · p. 5

1) Text Feature Encoding:

The text encoder converts each radiology report into contextualized token representations. Organ-guided attention then lets each CT-derived organ feature select relevant report text, and the enhanced organ features replace the original graph nodes for downstream reasoning.

1) Text Feature Encoding:

  • Report tokens are processed by a text encoder into contextualized token representations, which are normalized while preserving medical semantics.The report is tokenized as a sequence of length L; each token representation is a hidden state in d dimensions.Source: 1) Text Feature Encoding · p. 5

2) Organ-guided Cross-modal Attention:

  • Each organ feature acts as a query over report tokens, enabling organ-specific selection of relevant textual descriptions for fine-grained cross-modal alignment.The organ-guided attention mechanism contrasts with global feature fusion by aligning individual organ features with relevant report text.Source: 2) Organ-guided Cross-modal Attention · p. 5
  • Text-enhanced organ features preserve organ-specific characteristics and replace the original graph-node representations for subsequent graph-based staging reasoning.This supplies downstream reasoning with both structural and textual evidence.Source: 2) Organ-guided Cross-modal Attention · p. 6

D. Report-Enhanced Graph-Refinement

Feature-level report fusion does not make explicit invasion findings alter the organ graph. REG adds a report node connected to primary sites and explicitly invaded organs while retaining the original organ connectivity.

D. Report-Enhanced Graph-Refinement

  • Feature-level fusion alone does not distinguish explicitly invaded organs from other anatomically at-risk organs, leaving report-based invasion evidence underused in graph reasoning.Reports may state which organs are invaded, but using that information only to enhance features leaves graph connections unchanged.Source: D. Report-Enhanced Graph-Refinement · p. 6
  • REG adds a dedicated text node connected to primary sites and organs explicitly described as invaded, while preserving the original organ–organ connectivity.The text node encodes invasion relationships stated in the report rather than treating text only as auxiliary features.Source: D. Report-Enhanced Graph-Refinement · p. 6, 1) Text Node Construction · p. 6

1) Text Node Construction:

The method extracts explicit organ-invasion statements from reports and encodes report information in a graph text node. Report-guided links and message passing refine the organ graph for staging classification.

1) Text Node Construction:

  • The prompt extracts only explicitly positive invasion statements for the esophagus, thyroid gland, trachea, and oral cavity, distinguishing thyroid gland from thyroid cartilage.The thyroid gland and thyroid cartilage are treated as different structures in the prompt design.Source: 1) Text Node Construction · p. 6
  • Claude 3.5 Haiku matched GPT-4o and Qwen-2.5-72B on all 439 reports, and manual review confirmed all extractions were correct.The reported agreement applies to the structured invasion extractions.Source: 1) Text Node Construction · p. 6

2) Graph Refinement:

  • The text node projects the report’s [CLS] representation to the organ-feature dimension and connects to primary sites and explicitly invaded target organs.These report-guided links are added while preserving the original organ–organ connectivity.Source: 1) Text Node Construction · p. 6, 2) Graph Refinement · p. 6
  • Message passing on the report-enhanced graph produces final node representations, followed by global average pooling for staging classification.The framework is trained end-to-end with cross-entropy loss for T-staging classification.Source: 2) Graph Refinement · p. 6, 3) Training Strategy · p. 6
  • Report-derived invasion evidence directly guides feature refinement in the enhanced graph, supporting more accurate staging decisions.The stated mechanism is the incorporation of structured invasion information into graph topology.Source: 2) Graph Refinement · p. 6

3) Training Strategy:

The framework is trained end-to-end for T-staging, and the study introduces a four-center dataset of 656 patients with paired CT reports. The primary cohort supports cross-validation, while three cohorts form the external test set.

3) Training Strategy:

  • The framework is trained end-to-end using cross-entropy loss for T-staging classification.The loss uses the ground-truth class label and predicted class probability.Source: 3) Training Strategy · p. 6

A. Dataset

  • The study reports a first and largest CT-based laryngopharyngeal T-staging dataset: 656 patients across four centers, with paired CT reports.Prior laryngopharyngeal and head-and-neck staging studies were single-center, had 118–319 patients, and lacked paired radiology reports.Source: A. Dataset · pp. 6–7
  • FUZH contributes 439 patients for five-fold cross-validation; CGMH, EENT, and TCGA form the external test set, with TCGA reports manually annotated and radiologist-reviewed.The four cohorts contain 439, 25, 87, and 105 cases, respectively.Source: 1) Dataset Composition and Evaluation… · p. 7

1) Dataset Composition and Evaluation Protocol:

The evaluation uses pathology-confirmed T-stage labels and tests binary and three-class classification. The proposed method leads the reported internal and external comparisons against imaging-only and multimodal baselines.

1) Dataset Composition and Evaluation Protocol:

B. Experimental Setup

  • CT volumes are resampled to 1 mm³ and resized to 128×128×128; organ masks come from one pretrained segmentation model without site-specific adaptation.The component ablation uses a representative internal fold and reports mean ± standard deviation over three random seeds.Source: 1) Implementation Details · p. 7, Table IV · p. 7

2) Evaluation Criteria:

C. Comparison to State-of-the-Arts

  • On external three-class testing, the method reaches 60.27% F1 and 63.78% accuracy, exceeding the second-best method by 8.73% F1 and 7.78% accuracy.It also leads the internal and external binary comparisons and internal three-class testing.Source: C. Comparison to State-of-the-Arts · p. 8

D. Ablation Study

Ablations and robustness tests assess the framework’s components, encoder strategy, organ-specific predictions, and sensitivity to perturbed inputs. The complete AOG+OCA+REG configuration performs best in the reported ablation.

D. Ablation Study

  • 83.96% and 64.53% macro F1 are the full model’s binary and three-class scores, respectively, the highest among the reported component combinations.Adding REG to AOG+OCA improves over that combination; the ablation is reported on a representative internal fold over three random seeds.Source: 1) Effectiveness of Individual… · p. 8

2) Impact of Text Encoder Freezing:

  • Fine-tuning yields 81.46% versus 80.83% F1 on internal binary validation and 77.77% versus 76.46% on external testing; freezing is adopted in the final framework.For three-class testing, the frozen encoder has marginal advantages: 61.43% versus 60.79% internally and 60.27% versus 58.33% externally.Source: 2) Impact of Text Encoder Freezing · p. 8, Figure 3 · p. 9

3) Effectiveness of Individual Invaded Organs:

  • The thyroid-focused model reaches average F1-scores of 0.716 internally and 0.582 externally, above global processing at 0.588 and 0.436.Trachea- and esophagus-focused models also outperform global processing, while the oral-cavity model is comparable.Source: 3) Effectiveness of Individual Invaded… · pp. 8–9

E. Robustness to Organ Segmentation Noise and Report Variation

  • Macro F1 falls by 1.1% and 2.8% under 5% and 10% organ-mask perturbations, respectively, beyond the segmentation errors already present in the baseline masks.Across original and rewritten reports, macro F1 stays within 80.4–82.1% and accuracy within 81.8–83.0%, with at most a 1.7-point macro-F1 change; rewrites preserve medical findings while varying detail.Source: E. Robustness to Organ Segmentation… · p. 9

F. Superiority of AOG

The section compares anatomical graph and attention designs, then evaluates the model against radiologists and examines errors. The reported results favor the proposed approaches, while failure analyses identify CT/report evidence discordant with pathology.

F. Superiority of AOG

  • AOG achieves 77.76% and 73.33% F1 on internal and external tests, exceeding the Fully-Connected Graph baseline by 3.85% and 4.33%.The comparisons also include Direct Fusion and Global Context.Source: F. Superiority of AOG · p. 9

G. Superiority of OCA

  • OCA reaches 81.93% F1 and 82.95% accuracy internally, and 76.10% F1 and 76.50% accuracy externally, beating Global+Attention by 7.63% and 7.84% on external testing.Attention examples link stage-specific organ nodes with location, adjacent-structure, or invasion-related report terms.Source: G. Superiority of OCA · pp. 9–10

H. Clinical Validation

  • On the external 80-sample comparison, the model reaches 59.71% F1 and 62.50% accuracy for T1, T2, Late, exceeding both radiologists by at least 6.08% F1 and 2.50% accuracy.For Early versus Late, its F1 and accuracy are 76.25%, comparable to the two radiologists.Source: H. Clinical Validation · p. 10, Table VII · p. 10
  • Among 16 misclassified cases in Fold 1, 9 were over-staged and 7 under-staged; the errors reflected CT/report invasion evidence discordant with pathological ground truth.In 7 over-staged cases, reports described cartilage invasion not confirmed by pathology; in 5 under-staged cases, reports lacked a positive invasion description despite pathological confirmation.Source: H. Clinical Validation · p. 10
  • The authors note that unreliable invasion descriptions can tip borderline cases, and propose uncertainty-aware fusion, borderline-case review, and multi-center training as future directions.Source: H. Clinical Validation · p. 11

V. DISCUSSION

The framework consistently outperformed unimodal and multimodal baselines across binary and multi-class staging tasks. Discussion experiments examined robustness to segmentation noise and report detail, while the authors identify prospective validation as necessary before clinical deployment.

V. DISCUSSION

  • Only 2.8% F1 degradation occurred under ±10% proportional organ-boundary noise, and the authors attribute robustness partly to masks locating crop regions rather than excluding organ content.Head-and-neck organ segmentation Dice scores of 80–95% are reported as routinely achievable for most structures.Source: §V · pp. 11–12
  • Thyroid-cartilage invasion descriptions did not encode pathological T-stage: 30.6% of reports with positive descriptions were early-stage, while 40.7% of advanced-stage cases lacked positive descriptions.The authors explain that reports reflect CT-visible findings rather than pathological diagnosis.Source: §V · p. 12, Table IX · p. 12
  • Report-detail rewrites changed macro-F1 by at most 1.7 points and accuracy by at most 1.2 points; cross-institutional performance was stable despite reporting variation.The rewrites retained medical terms and measurements while varying detail, and all reports were routine clinical documents.Source: §V · p. 12
  • Prospective validation on independent cohorts with different disease distributions is needed before clinical decision-support integration.Segmentation performance may vary by imaging protocol and scanner vendor, and reporting differences may affect OCA and REG.Source: §V · p. 12

VI. CONCLUSION

  • The framework consistently outperformed state-of-the-art unimodal and multimodal baselines in binary and multi-class staging, with gains over radiologist assessments.The authors also describe improved interpretability through alignment with clinically meaningful invasion pathways.Source: §VI · p. 12
Loading 2610.06837v1…