PaperlayerView exact revision on arXiv

Source-linked AI summary

Report Supervision

Pedro R. A. S. Bassia, Wenxuan Li, Jakob Wasserthal, Jieneng Chen, Xinze Zhou, Zheren Zhu, Chuntung Zhuanga, Sergio Decherchi, Andrea Cavalli, Kang Wang, Yang Yang, Alan Yuille, Zongwei Zhou

arXiv:2608.27668v1cs.CV

TL;DR

Tumor segmentation is constrained by scarce, expensive masks despite abundant radiology reports. R-Super directly converts report descriptions into segmentation supervision, improving external-validation detection and segmentation by up to +15% over mask-only training while outperforming CLIP and multi-task learning.

  • Problem

    Tumor masks are expensive and scarce, whereas radiology reports are routinely produced, abundant, and detailed.

  • Method

    R-Super uses an LLM to extract tumor count, size, and location from reports, then applies Volume Loss and Ball Loss alongside standard mask losses during training.

  • Results

    +15% F1-Score and DSC over mask-only training was achieved on external validation, with R-Super also surpassing CLIP and multi-task learning.

  • Takeaways & Limitations

    Reports can supplement scarce masks and improve tumor detection and segmentation from very small to large training-mask collections.

  • Takeaways & Limitations

    Performance depends on organ masks that may be inaccurate and on tumor types whose separate sub-segments may require poorly scalable handling.

Abstract

from arXiv · show

Segmentation models can surpass radiologists, classification models, and vision-language models in tumor detection. Importantly, segmentation models outline tumors, allowing radiologists to better verify and trust the AI output. Their main limitation is the scarcity of tumor masks: creating one 3D tumor mask takes up to 30 minutes, so most public CT datasets contain only a few hundred masks, and even the largest private datasets contain only a couple of thousand. Tumor masks are not produced in clinical routine, but radiology reports are. Public datasets contain tens of thousands of CT-Report pairs, and hospitals contain hundreds of thousands. These reports describe tumors in detail, providing large-scale, informative training data. Here, we introduce Report Supervision (R-Super), a training framework that uses reports to directly supervise and improve tumor segmentation. R-Super introduces new loss functions that teach segmentation models to segment tumors that match report descriptions of tumor count, sizes, and locations. Reports are only used for training. We evaluated R-Super on kidney and pancreatic tumor segmentation, exploring diverse training data sizes, up to 41,418 CT-Report plus 3,488 pancreatic tumor CT-Mask pairs. On external validation, R-Super increased tumor detection F1-Score and segmentation DSC by up to +15% with respect to mask-only training. It also surpassed alternative methods such as CLIP and multi-task learning. Leveraging numerous readily available reports to supplement scarce masks, R-Super strongly improves AI performance when very few training masks are available (e.g., 50), and when many masks are available (e.g., 3,488), unlocking scale in tumor segmentation.

1. Introduction

Tumor masks are scarce and costly, whereas radiology reports are abundant and detailed. R-Super directly uses report descriptions to supplement masks and improve tumor segmentation across data scales.

  • Over 50% of small pancreatic tumors are missed at first read, while kidney-tumor misses reach 22% on non-contrast CT and 11–23% on non-delayed contrast-enhanced CT.
  • Creating one 3D tumor mask can take 30 minutes, and public datasets generally contain fewer than 100 masks per tumor type.The JHH private dataset required 8 radiologists and 5 years to create 3,488 pancreatic tumor masks.
  • Radiology reports may unlock scale because hospitals store hundreds of thousands of reports that describe tumors in detail, unlike tumor masks produced in routine clinical work.
  • R-Super introduces Volume Loss and Ball Loss functions that penalize mismatches between segmented tumors and reported tumor count, sizes, and locations.An LLM extracts tumor information from reports before training, and reports are used only during training.

2. Related Work

Prior methods use reports indirectly through classification, contrastive pretraining, or pseudo-label generation. R-Super instead directly optimizes segmentation with report-derived tumor information and extends evaluation across public and private datasets.

  • CLIP: CLIP pretraining aligns images and texts before segmentation fine-tuning, so report supervision benefits segmentation indirectly and may be small or negative.
  • Multi-task Learning: Multi-task learning jointly trains classification and segmentation, but classification features are not guaranteed to improve segmentation.
  • Self-supervised Learning: Self-supervised methods learn image structure without reports, whereas R-Super exploits reported tumor counts, sizes, and locations.
  • Report-guided Pseudo-labels: Report-guided pseudo-labeling filters teacher-generated pseudo-masks by the number of tumors stated in each report before training a student model.
  • Report Supervision: R-Super was introduced with Volume Loss and Ball Loss, which directly optimize tumor segmentation using report descriptions rather than auxiliary tasks.
  • Extensions: Adding 1,848 public Merlin CT-Report pairs to PanTS improved tumor detection and segmentation on the largest public pancreatic mask dataset.
  • Extensions: Adding 41,418 CT-Report pairs to 6,212 private CT-Mask pairs further improved performance and enabled pancreatic-tumor detection on non-contrast CT.

3. R-Super

R-Super uses radiology reports to directly supervise tumor segmentation by converting reported tumor information into Volume and Ball losses. An LLM extracts tumor characteristics, while organ masks localize supervision and the losses accommodate measurement uncertainty.

  • 3.3. Ball Loss: Ball Loss enforces agreement with reported tumor count, location, and diameters by locating high-probability regions with fixed diameter-matched spherical kernels.Volume Loss is applied as deep supervision for easier optimization, while Ball Loss is applied at the final segmentation layer to match detailed tumor information.
  • 3.1. LLM for Extracting Tumor Information: An LLM extracts tumor diameters, locations, and counts from variable report text before segmentation training.The LLM is used once before training, with structured outputs generated from prompts refined through radiologist–computer scientist error analysis.
  • 3.1. LLM for Extracting Tumor Information: A radiologist found 96% accuracy for LLM tumor-information extraction, and R-Super showed minimal degradation with LLM error rates up to 10%.The extraction evaluation covered 447 reports, including 182 with tumors.
  • 3.2. Volume Loss: Volume Loss compares reported and segmented tumor volumes within each organ or sub-segment, while penalizing tumors outside reported locations.Reported volumes are estimated from one, two, or three tumor diameters; segmented volumes are soft, differentiable sums of tumor probabilities within organ masks.
  • 3.2. Volume Loss: A 10% tolerance margin prevents Volume Loss from penalizing segmented tumor volumes that are close but not equal to report-based estimates.The margin compensates for measurement and diameter-to-volume approximation errors.

4. Results

Across internal and external validation, R-Super improves tumor detection and segmentation by supplementing scarce masks with radiology reports. Gains persist across mask budgets, tumor sizes, datasets, and challenging scans, while ablations clarify the roles of report information, losses, organ masks, and LLM quality.

  • Training and validation data: 6,718 CT-Report pairs in UCSF-Train and 41,418 in UCSF-Huge provided report supervision alongside mask datasets.The experiments also used public AbdomenAtlas datasets and external testing on JHH-Test and Swiss-Test.
  • Reports improve performance with few or many masks: +10% F1-Score with 50 masks, +16% with 344 masks, and +4% with 1.7K masks were achieved over standard segmentation.DSC gains were +11% with 50 masks and +8% with 344 masks.
  • Generalization: R-Super outperformed standard segmentation on both UCSF-Test and the out-of-distribution JHH-Test, and surpassed six state-of-the-art methods.JHH-Test was from an unseen hospital, while UCSF-Test included varied resolutions, contrast phases, and non-contrast scans.
  • Small tumors: 80 F1-Score and 89 AUC were reached for small pancreatic tumors on external JHH-Test, exceeding standard segmentation by +15 F1 and +6 AUC.R-Super improved detection and segmentation across the full tumor-size spectrum, including tumors with diameter ≤2 cm.

5. Conclusion

R-Super turns radiology reports into direct supervision for tumor segmentation, enabling larger-scale training and improving performance across data scales and external hospitals. It also outperforms alternative report-based approaches because its losses directly supervise segmentation rather than an auxiliary task.

  • R-Super transforms radiology reports into supervision for tumor segmentation, enabling training to scale with abundant CT-Report pairs.The framework supplements scarce masks with reports already stored in hospitals and increasingly available in public datasets.
  • Reports substantially improve tumor detection and segmentation from small mask sets to datasets combining over 40,000 CT-Report pairs with over 6,000 CT-Mask pairs.The paper reports gains when adding over 6,000 reports to 200 CT-Mask pairs and over 40,000 reports to more than 6,000 masks.
  • R-Super supports cross-hospital training without masks from every hospital, improving training-data diversity and generalization on JHH-Test, JHH-Small, JHH-Large, and Swiss-Test.Experiments included validation on hospitals unseen during training.
  • R-Super outperformed CLIP-style pretraining and multi-task learning because reports directly supervise tumor segmentation instead of an auxiliary task.The segmentation model learns to produce tumors matching report descriptions.
  • The largest training set contained about 30 times more pancreatic tumor cases than the largest public pancreatic tumor segmentation dataset.The authors suggest that new public datasets and multi-institution collaborations could extend this scale further.

A.1. Weighted Cross-Entropy and Gaussian Ball

This section describes weighted cross-entropy and Gaussian Ball Convolution components used to focus supervision on confident tumor voxels and locate tumors using report-derived sizes.

  • Weighted Cross-Entropy: Kernel-weighted cross-entropy down-weights uncertain border voxels and concentrates gradient contributions on confidently predicted tumor voxels.The weighting replaces standard cross-entropy in the relevant loss.
  • Weighted Cross-Entropy: Rank weighting applies to voxels in the predicted tumor mask, while voxels outside that mask retain standard cross-entropy weight 1.The decay parameter c controls concentration on top-ranked voxels; c=0.5 gives smooth decay.
  • Gaussian Ball: The Ball Convolution uses a ball whose diameter matches the reported tumor diameter and sums tumor probabilities within the organ mask at each position.It is applied to per-voxel tumor probabilities while excluding probabilities outside the relevant organ.
  • Gaussian Ball: A Gaussian-tapered kernel favors probability concentrated near the ball center while remaining zero outside the reported tumor-sized ball.The kernel is largest at the center and smaller near the boundary.
  • Gaussian Ball: The Gaussian decay is scaled to each reported diameter, and normalization keeps Ball Convolution outputs comparable across kernel sizes.Normalization does not change the location of the maximum for a given tumor.
  • Gaussian Ball: Setting α=3 produces a mild boundary-to-center kernel ratio of exp(−1/18) ≈0.95.The decay breaks argmax ties while keeping the convolution close to a scaled sum of probabilities inside each ball.

A.2. Volume Loss for Reports without Tumor Size

For reports lacking tumor size or count, high-tolerance Volume and Ball Loss variants replace precise targets with permissive constraints that preserve useful minimum supervision.

  • Volume Loss: Standard Volume Loss cannot be applied when reports indicate tumors but provide neither diameters nor counts needed to compute the reported volume.The high-tolerance variant replaces the precise target with a tolerated volume range.
  • Volume Loss: The tolerated volume range is calibrated from report diameters as V_min=65 mm^3 and V_max=904,779 mm^3.Fewer than 5.5% of images had all tumors below 5 mm, and fewer than 1.66% had tumors above 120 mm.
  • Volume Loss: The high-tolerance Volume Loss gives zero loss and gradient for segmented volumes within [V_min,V_max], while pulling volumes toward the range outside it.The target equals the segmented volume inside the tolerated interval.
  • High-Tolerance Ball Loss: The high-tolerance Ball Loss locates a small high-probability ball, selects the most probable voxels inside it, and supervises only those voxels.Gradients at other voxels inside the organ are zeroed, leaving them unpenalized.
  • High-Tolerance Ball Loss: Without precise size or count information, the high-tolerance losses encourage at least one small tumor while allowing larger or multiple tumors without penalty.The model is still penalized for probabilities outside the organ.

B. Training Details

R-Super trains on organ-centered 3D CT patches with standardized optimization and preprocessing, while baselines share the MedFormer setup to isolate training-method differences.

  • Data and preprocessing: Models use 128 × 128 × 128 voxel 3D CT patches at isotropic 1 mm spacing, centered on a target organ for CT-Report pairs.Organ-centered patches prevent reported tumors from falling outside the input.
  • Optimization: Optimization uses AdamW, weight decay 5 × 10^-2, gradient clipping at 1, batch size 2, and 100 epochs of 1,000 batches.The learning rate starts at 1 × 10^-4, uses a 5-epoch warmup, and then polynomial decay.
  • Data and preprocessing: CT intensities are clipped to [−991, 500] HU and normalized, with rotation, brightness, gamma, contrast, blur, and noise augmentation.Augmentations are applied per sample following MedFormer settings.
  • Baselines: Most baselines use the same MedFormer backbone and hyper-parameters as R-Super, so performance differences reflect training methodology rather than architecture.MTL additionally uses a classification head, while standard segmentation uses CT-Mask pairs and MTL uses CT-Mask plus CT-Report pairs.
  • Baselines: nnU-Net uses a different architecture and training scheme, with its standard procedure and automatic hyper-parameters except for matched 1 mm isotropic spacing.Its architecture is ResEncL and it is trained on CT-Mask pairs.

C. Result Variability

The reported uncertainty reflects substantial patient-level variation in segmentation difficulty, while detection uncertainty is quantified with bootstrap confidence intervals and formal statistical tests.

  • Uncertainty estimation: 95% confidence intervals for detection metrics were estimated using 1,000 nonparametric bootstrap resamples of test cases.Intervals use the 2.5th and 97.5th percentiles.
  • Statistical testing: F1-Score and AUC comparisons used paired permutation and DeLong tests, respectively, with orange highlights marking significant gains over standard segmentation at p < 0.05.The table reports intervals for detection metrics and standard deviations for DSC and NSD.
  • Result variability: Segmentation DSC and NSD variability is reported as standard deviation because tumor size, shape, and texture differ considerably across patients.These differences produce varying segmentation difficulty and scores.

D. LLM Prompt

The LLM prompt converts radiology reports into structured lesion descriptions for report-supervised segmentation, while accompanying tables quantify uncertainty around reported comparisons. It captures lesion identity, certainty, size, organ, location, and attenuation, including rules for ambiguous or multiple lesions.

  • LLM prompt: The prompt asks the LLM to list each lesion’s type, certainty, size, organ, location, and attenuation from relevant report sections.It directs attention to findings, clinical history, and impressions when available.
  • Lesion extraction: Lesions include benign, malignant, confirmed, and suspicious focal abnormalities, with suspicious lesions assigned low certainty.The prompt distinguishes focal lesions from findings such as uncomplicated stones or diverticula.
  • Result uncertainty: 95% confidence intervals use 1,000-test-case bootstrap resamples, while DSC and NSD are summarized with standard deviations.The corresponding tables compare detection metrics using paired permutation tests for F1-Score and DeLong’s tests for AUC.
  • Lesion extraction: Multiple unspecified lesions in one organ receive an additional entry with size = multiple, alongside individual entries for specifically described lesions.The prompt also provides an explicit no-lesion response when the report contains no lesion.
  • Type assignment: Lesion type is inferred from findings, history, or impressions only under specified rules, while unspecified types are recorded as type = U.More specific diagnoses take precedence over the generic malignant label.
Loading 2608.27668v1…