Source-linked AI summary
RT-Super: Learning Tumor Segmentation from Longitudinal Images and Reports
Pedro R. A. S. Bassi, Wenxuan Li, Hanxue Gu, Jieneng Chen, Xinze Zhou, Zheren Zhu, Sezgin Er, Ibrahim E. Hamamci, Bjoern H. Menze, Gulhan E. Akan, Kang Wang, Yang Yang, Alan L. Yuille, Zongwei Zhou
TL;DR
Tumor segmentation masks are costly and scarce across tumor types, despite hospitals routinely collecting reports and multiple scans. RT-Super uses a multi-image, report-informed teacher to train a student that segments from one image alone. The authors report improved detection and segmentation on mask-scarce tumor types, including when training without tumor masks.
Problem
Tumor masks are expensive to create and public datasets cover only a few tumor types, while hospitals routinely collect reports and multiple scans that may guide segmentation.
Source: §1 · p. 2Method
Results
Takeaways & Limitations
The authors conclude that routinely available reports and longitudinal and multi-phase images can support tumor segmentation across more tumor types than manual masks alone cover.
Source: §3 · p. 9The inference setting assumes reports are unavailable, and a first diagnostic image often has no prior scan to provide longitudinal context.
Source: §1 · p. 2
Abstract
from arXiv · showhide
Multi-tumor segmentation is important for early cancer detection and allows radiologists to visualize, verify, and understand AI predictions. However, tumor segmentation masks are expensive, time-consuming, and unavailable for many tumor types in public data. Instead, hospitals have vast, readily available data that can guide segmentation: radiology reports, longitudinal images, and multi-phase images. We use this readily available data to substitute for tumor masks in training AI for tumor segmentation. To this end, we propose a new architecture, RT-Super. It has a teacher network, which analyzes the patient's longitudinal images and reports to create high-quality tumor masks. These masks train a student network, which sees a single image and no report. At inference, when longitudinal images and reports are unavailable, we use the student. RT-Super uses a new CNN-Transformer architecture and novel Consistency Losses that exploit tumor location consistency across longitudinal images. We train RT-Super to segment esophagus, uterus and spleen tumors, which have few or no public masks. Even without training masks, RT-Super can segment these tumors and surpass public AI models. Overall, we demonstrate that learning from longitudinal images, multi-phase images, and reports can overcome mask scarcity and advance multi-cancer detection and segmentation. Code: https://github.com/MrGiovanni/RT-Super
1 Introduction
Tumor masks are costly and public datasets cover few tumor types, while hospitals routinely hold reports and multiple scans that may guide segmentation. RT-Super uses these sources to train a single-image student and reports improved detection and segmentation on mask-scarce tumor types.
1 Introduction
- Tumor masks are expensive and time-consuming to create, and public datasets provide masks for only a few tumor types.Public CT datasets and models mainly cover lung, liver, pancreas, kidney, and colon tumors.Source: §1 · p. 2
- Hospitals routinely collect longitudinal scans, contrast-phase images, and reports describing tumor size, location, and count, which can substitute for or augment scarce masks.The paper asks whether these sources can support segmentation with fewer or no masks.Source: §1 · p. 2
- RT-Super uses a teacher that learns from reports and longitudinal or multi-phase images to create masks for training a single-image, report-free student.The student is used at inference when the additional images and reports are unavailable.Source: §1 · p. 2
- On esophagus, spleen, and uterus tumors, RT-Super substantially surpassed public AI models and alternative training methods in detection.The dataset contains 34,152 CT scans and uses no masks or a limited set of 767 radiologist-created masks.Source: §1 · pp. 2–3
- RT-Super’s Consistency Loss uses consistency across longitudinal and multi-phase images, and the method improves performance when inference uses one image and no report.Its training data include reports and scans across time or contrast phases.Source: §1 · pp. 2–3
2 RT-Super
RT-Super combines a self-distillation teacher–student architecture with report supervision and consistency losses to train segmentation when masks are scarce. The teacher uses reports and multiple images to refine the student’s features, while the student learns to segment from one image alone.
2.1 RT-Super Architecture
- Across multiple scans, shared-weight teacher–student instances process one image each, while teacher instances exchange information through inter-image cross-attention.The system also provides image-date and contrast-phase tokens and sets missing report attributes or images to zero.Source: Figure 1 · p. 4, §2.1 · p. 6
- The teacher refines the student’s deep features using a report-aware transformer and convolutional path, then its mask trains the student when ground-truth masks are absent.The student can be any segmentation architecture; the paper uses MedFormer.Source: §2 · p. 3, §2.1 · p. 4, §2.2 · p. 6
- A report-aware transformer uses learnable queries, report attributes, and student features to generate convolutional kernels that iteratively refine the student’s features.The 3 × 3 × 3 kernels are weighted combinations from a learnable bank of 16 kernels.Source: §2.1 · pp. 4–5
2.2 RT-Super Loss Functions
- With ground-truth masks, Dice and BCE losses train both networks; without masks, report-based Volume and Ball losses train the teacher and distillation trains the student.Student distillation uses hard targets formed by binarizing the teacher output with report-based post-processing.Source: §2.2 · p. 6
- The Consistency Loss uses larger, clearer tumors in longitudinal scans to guide segmentation of smaller tumors, enforcing location consistency rather than size consistency.It is applied when reports indicate every tumor in the small-tumor scan is smaller than every tumor in the large-tumor scan; the large scan is registered and dilated before constraining the smaller-scan mask.Source: §2.2 · pp. 6–7
- The Table 1 caption reports improved multi-tumor detection at single-image, no-report inference and identifies the main gain over R-Super as occurring without training masks.It describes sensitivity, specificity, and F1 at the operating point that maximizes balanced accuracy.Source: Table 1 · p. 7
3 Results
RT-Super improved tumor detection and segmentation with few or no tumor masks, surpassing public AI models and other training methods. External validation used an unseen hospital, with detection evaluated on small tumors.
3 Results
- On unseen-hospital data, report-only RT-Super substantially surpassed R-Super in segmentation DSC; detection was evaluated on tumors under 2 cm.The segmentation DSC test set included 35% small tumors.Source: Table 2 · p. 8
- The caption attributes lower DSC than for pancreatic tumors to the difficulty of segmenting these tumor types on CT; radiologist-pair DSC values were 66 for spleen, 50 for esophagus, and 62 for uterus tumors.Source: Table 2 · p. 8
- Testing models had no access to patient reports or longitudinal images.Source: §3 · p. 8
- RT-Super† used CT-Report pairs, while RT-Super used CT-Report and CT-Mask pairs; Consistency Loss was skipped when registration DSC fell below 0.8.Consistency and Report Supervision losses were weighted at 0.1.Source: §3 · pp. 7–9
- At the end of training, the RT-Super teacher surpassed its student by 2.9% DSC.Source: §3 · p. 9
- RT-Super improved tumor detection and segmentation with few or no masks, surpassing eight dataset-trained methods and all listed public AI models.Source: §3 · p. 9
- The authors conclude that routine reports, longitudinal images, and multi-phase images can support tumor segmentation beyond manual masks and scale it to more tumor types.Source: §3 · p. 9