PaperlayerView exact revision on arXiv

Source-linked AI summary

Report Supervision

Pedro R. A. S. Bassia, Wenxuan Li, Jakob Wasserthal, Jieneng Chen, Xinze Zhou, Zheren Zhu, Chuntung Zhuanga, Sergio Decherchi, Andrea Cavalli, Kang Wang, Yang Yang, Alan Yuille, Zongwei Zhou

arXiv:2608.27668v1cs.CV

TL;DR

Tumor segmentation is constrained by scarce CT masks, although radiology reports describe tumors in detail. R-Super uses report-derived tumor descriptions to supervise segmentation during training. On external validation, it improved tumor detection F1-Score and segmentation DSC by up to +15% over mask-only training.

  • Problem

    Tumor segmentation is constrained by scarce CT masks, while radiology reports are routinely written and describe tumors in detail.

    Source: §1 · p. 1

  • Method

    R-Super uses report-derived tumor count, size, and location descriptions in Volume and Ball losses to directly supervise tumor segmentation during training.

    Source: §1 · p. 2

  • Results

    Up to +15% F1-Score and DSC over training with masks alone on external validation, across settings with few or many training masks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    R-Super demonstrates that reports can supplement masks across multiple data scales in tumor segmentation.

    Source: §5 · p. 15

    A per-tumor-type approach scales poorly as more tumor types are added.

    Source: §4.4 · p. 15

Abstract

from arXiv · show

Segmentation models can surpass radiologists, classification models, and vision-language models in tumor detection. Importantly, segmentation models outline tumors, allowing radiologists to better verify and trust the AI output. Their main limitation is the scarcity of tumor masks: creating one 3D tumor mask takes up to 30 minutes, so most public CT datasets contain only a few hundred masks, and even the largest private datasets contain only a couple of thousand. Tumor masks are not produced in clinical routine, but radiology reports are. Public datasets contain tens of thousands of CT-Report pairs, and hospitals contain hundreds of thousands. These reports describe tumors in detail, providing large-scale, informative training data. Here, we introduce Report Supervision (R-Super), a training framework that uses reports to directly supervise and improve tumor segmentation. R-Super introduces new loss functions that teach segmentation models to segment tumors that match report descriptions of tumor count, sizes, and locations. Reports are only used for training. We evaluated R-Super on kidney and pancreatic tumor segmentation, exploring diverse training data sizes, up to 41,418 CT-Report plus 3,488 pancreatic tumor CT-Mask pairs. On external validation, R-Super increased tumor detection F1-Score and segmentation DSC by up to +15% with respect to mask-only training. It also surpassed alternative methods such as CLIP and multi-task learning. Leveraging numerous readily available reports to supplement scarce masks, R-Super strongly improves AI performance when very few training masks are available (e.g., 50), and when many masks are available (e.g., 3,488), unlocking scale in tumor segmentation.

1. Introduction

R-Super directly uses radiology reports to supervise tumor segmentation, addressing scarce masks with detailed information available in reports. On external validation, it improved detection and segmentation by up to +15% over mask-only training.

Motivation

  • Segmentation reveals tumor locations for radiologist verification and can outperform classification, vision-language, and human detection, despite frequent misses of small tumors.Over 50% of small pancreatic tumors are missed at first read; reported miss rates for small kidney tumors range from 11% to 23% on contrast-enhanced CT.Source: §1 · p. 1
  • Mask scarcity motivates R-Super: one 3D mask can take 30 minutes, public datasets have none exceeding 2,000 masks per tumor type, while reports are abundant and describe tumor count, location, and size.The public Merlin dataset alone contains over 25,000 CT-Report pairs.Source: §1 · p. 1

Approach

  • R-Super directly trains segmentation with LLM-extracted tumor counts, sizes, and locations, using Volume and Ball losses on report pairs and standard Dice and cross-entropy losses on mask pairs.Reports are used only during training, not inference.Source: §1 · p. 2

Related work

  • R-Super consistently surpassed alternatives, while prior MTL and CLIP approaches used reports for auxiliary tasks whose segmentation benefits were small or negative.The alternatives included CLIP, MTL, nnU-Net, and report-guided pseudo-labels.Source: §1 · p. 2

Results

  • Up to +15% F1-Score and DSC over mask-only training was achieved on external validation.R-Super improved tumor detection and segmentation across settings with few or many training masks.Source: §1 · p. 2

Contributions

  • R-Super improved performance on public and private mask datasets, boosted pancreatic tumor detection on non-contrast CT, and the authors released a public CT-Report dataset.The released dataset contains 684 CT-Report pairs, 445 with pancreatic tumors; experiments also scaled PanTS and JHH with report pairs.Source: §1 · p. 2

2. Related Work

R-Super differs from related approaches by using reports to directly supervise tumor segmentation rather than relying on indirect pretraining, auxiliary tasks, or report-filtered pseudo-labels. The paper also substantially extends the original framework with new datasets, experiments, and a more formal methodology.

2. Related Work

  • Unlike alternatives that ignore reports or use them indirectly, R-Super directly optimizes tumor segmentation from report descriptions.The comparison spans mask-supervised segmentation, self-supervised pretraining, CLIP, multi-task learning, and report-guided pseudo-labels.Source: §2 · p. 2, §2.1 · p. 3

2.1. Alternative Training Methods

  • CLIP matches images with reports during pretraining before mask-based fine-tuning, but its learned features may offer small or negative segmentation benefits.A low CLIP loss does not ensure high-quality segmentation, whereas R-Super losses directly penalize disagreement between predicted tumors and report descriptions.Source: §2.1 · p. 3
  • Multi-task learning trains classification and segmentation together, but classification may not improve segmentation, unlike R-Super’s direct report-based segmentation supervision.The passages note that mutual benefit can be marginal or absent, and that strong classification can coexist with poor segmentation.Source: §2.1 · p. 3
  • Bounding boxes, scribbles, and promptable models can reduce mask effort, but spatial annotations must still be created, and SAM-like prompts are required at inference.Because SAM assumes a tumor has already been detected and roughly localized, it is not directly designed for tumor detection; R-Super uses routinely produced reports without new annotations.Source: §2.1 · p. 3
  • Self-supervised pretraining learns from CT scans without reports, while R-Super exploits report details about tumor counts, sizes, and locations.The study adopts Models Genesis as an established, independently validated 3D medical self-supervised baseline.Source: §2.1 · p. 3
  • Report-guided pseudo-labeling filters teacher masks by report tumor count but can discard false-negative cases and does not use reported tumor sizes or locations.R-Super’s Volume Loss penalizes false negatives, and its loss functions use report details beyond tumor count.Source: §2.1 · p. 3

2.2. Our Previous Work

  • This paper extends the earlier R-Super framework with new public and large-scale private data, non-contrast CT experiments, and formalized loss functions.Additions include 684 public CT-Report pairs (445 pancreatic tumor cases), experiments combining public reports with PanTS, and 41,418 reports added to a private dataset of 6,212 CT-Mask pairs created by 8 radiologists over 5 years; it also tests non-contrast CT reports with contrast-enhanced CT masks.Source: §2.2 · pp. 3–4

3. R-Super

R-Super uses tumor information extracted from radiology reports to supervise segmentation through complementary losses for matching reported tumor volume and detailed tumor characteristics.

3. R-Super

  • R-Super combines CT-Mask pairs with CT-Report pairs, using report-derived tumor count, size, and location to guide segmentation through Volume Loss and Ball Loss.The Volume Loss is applied as deep supervision, while the stricter Ball Loss is applied at the final segmentation layer; mask pairs use Dice and cross-entropy losses.Source: §3 · p. 4
  • Organ and sub-segment masks restrict report-guided losses to diseased organs; masks are dilated by 2 cm, and the losses remain effective without them.Kidney guidance uses right- and left-kidney masks because the nnU-Net does not separately segment kidney sub-segments; ablations also retained high performance with masks trained on 50 CT scans.Source: §3 · p. 4

3.1. LLM for Extracting Tumor Information

  • A Llama 3.1 70B AWQ LLM extracts report findings using clinical-guideline prompts, with 96% accuracy on 447 reports and minimal performance degradation at error rates up to 10%.The LLM runs once before segmentation training; in-context examples support generalization without fine-tuning.Source: §3.1 · pp. 4–5

3.2. Volume Loss

  • The Volume Loss matches estimated segmented and reported tumor volumes, penalizes tumors outside the reported organ, and uses a 10% tolerance to accommodate volume-estimation uncertainty.When reports omit tumor count or diameters, a relaxed variant requires at least one small tumor without penalizing larger or multiple tumors.Source: §3.2 · pp. 6–7

3.3. Ball Loss

  • The Ball Loss locates each reported tumor with a diameter-matched ball, then guides probabilities at the most likely voxels to match tumor count, location, and size without enforcing spherical shapes.For multiple tumors, it locates them from largest to smallest while preventing overlap; a high-tolerance variant handles reports missing counts or diameters.Source: §3.3 · pp. 7–8

4. Results

The results evaluate R-Super across public and private settings, reporting improved tumor detection and segmentation from supplementing scarce masks with reports. Ablations examine the effects of report quality, tumor-diameter information, complementary loss functions, and the scalability of handling tumor types individually.

4.1. Reports Improve Performance with Few or Many Masks

  • R-Super improved F1-Score by up to +10% with 50 masks, +16% with 344, and +4% with 1.7K, and pancreatic DSC by +11% and +8% with 50 and 344 masks versus mask-only training.It outperformed alternative methods across internal and external tests; for small pancreatic tumors on JHH-Test, it reached F1-Score 80 and AUC 89, exceeding standard segmentation by +15 F1 and +6 AUC.Source: Figure 3 · p. 9, Table 1 · p. 10, §4.1 · p. 10

4.2. Reports Scale the Largest Public Pancreatic Tumor Segmentation Dataset

  • R-Super achieved +11% higher F1-Score and +9% higher AUC than mask-only training on Merlin-Test, and +4% F1-Score, +3% AUC, and +3% DSC on external JHH-Small.This public-data experiment used reports to scale PanTS, the largest public pancreatic tumor mask dataset, and also surpassed alternative training methods.Source: §4.2 · p. 11, Table 3 · p. 12

4.3. Reports Scale the Largest Private Pancreatic Tumor Segmentation Dataset

  • +19% sensitivity at matched specificity on non-contrast CTs and +8% on external Swiss-Test show that R-Super improved detection beyond mask-only training, even with thousands of masks available.Training combined 3,488 pancreatic tumor masks with 28,295 tumor reports; the authors report that preparing the reports took about a month, versus five years and substantial funding to create the masks.Source: §4.3 · pp. 11–13, Table 4 · p. 12

4.4. Ablation Studies

  • Adding Ball Loss to Volume Loss raised AUC by +3 points for pancreas and +2 for kidneys overall, with gains of +3 for multifocal kidney and +6 for complex pancreatic tumors.Volume Loss alone outperformed Ball Loss alone; together they performed best on UCSF-Test, as Volume Loss stabilizes early training while Ball Loss refines locations as predictions improve.Source: §4.4 · pp. 13–14, Table 5 · p. 13
  • With synthetic report errors in 20% of training pairs, AUC fell by at most 8%; at 100% errors it fell by up to 22%, while smaller LLMs also substantially reduced performance.R-Super remained above standard segmentation with up to 20% synthetic errors, but fell below it with 100% errors; Qwen 3 0.6B was often outperformed by standard segmentation.Source: Table 5 · p. 13, §4.4 · p. 14
  • With organ masks from nnU-Net trained on 50 scans, mean DSC was 81.6% for pancreas and 97.4% for kidneys; without organ masks, R-Super still beat standard segmentation despite up to 9% AUC loss.Organ masks improve precision, but the ablations show they can be generated from few scans or omitted entirely.Source: Table 5 · p. 13, §4.4 · pp. 14–15
  • Removing tumor diameters reduced AUC by up to 14% and DSC by 9%, compared with up to 9% AUC and 1% DSC after removing sub-segments, identifying diameters as the stronger report signal.Both diameters and sub-segments contributed to supervision.Source: Table 5 · p. 13, §4.4 · p. 15
  • Single-pancreas training raised JHH-Test F1/AUC to 92/98 from 91/92, but lowered NSD and sensitivity versus the two-organ model and requires separate models.On JHH-Test, the single-organ model’s DSC/NSD/sensitivity/specificity were 62/64/88/96 versus 59/69/94/88 for the two-organ model.Source: §4.4 · p. 15
  • An approach applied per tumor type scales poorly as the number of tumor types increases.Source: §4.4 · p. 15

5. Conclusion

R-Super turns radiology reports into direct supervision for tumor segmentation, substantially improving tumor detection and segmentation across training-data scales. It also supports cross-hospital generalization and outperforms report-based alternatives by teaching models to match tumor descriptions.

5. Conclusion

  • R-Super substantially improves tumor detection and segmentation, from adding over 6,000 CT-Report pairs to 200 CT-Mask pairs to adding over 40,000 reports to over 6,000 masks.The smaller mask set includes 100 tumors; the largest mask dataset took 8 radiologists 5 years to create.Source: §5 · p. 15
  • The study includes private- and public-data experiments and releases Swiss-Test, a new public CT-Report test set.Experiments across data scales assess generalization on hospitals unseen during training.Source: §5 · p. 15
  • Consistent gains on JHH-Test, JHH-Small, JHH-Large, and Swiss-Test show that merging report data across hospitals can improve generalization to unseen hospitals without creating masks for each hospital.Experiments at all data scales validated on hospitals not seen during training.Source: §5 · p. 15
  • R-Super outperformed CLIP-style pretraining and multi-task learning on the same datasets by using reports to supervise tumor segmentation directly rather than an auxiliary task.Minimizing its losses teaches the model to segment tumors that match report descriptions.Source: §5 · p. 15
  • The largest training set contains about 30 times more pancreatic tumor cases than the largest public pancreatic tumor segmentation dataset, and public datasets and multi-institution collaborations could extend that scale further.The study suggests that learning at scale from readily available text may also advance tumor detection and segmentation.Source: §5 · p. 15

A. Details on the Loss Functions · A.1. Weighted Cross-Entropy and Gaussian Ball

The loss-function details refine report-supervised segmentation with rank-weighted cross-entropy and a Gaussian-tapered Ball Convolution kernel. The weighting focuses supervision on confident tumor voxels, while the kernel favors tumor-probability concentrations near reported tumor centers.

A.1. Weighted Cross-Entropy and Gaussian Ball

  • Rank-weighted cross-entropy reduces border-voxel gradients and concentrates supervision on confidently predicted tumor voxels.It ranks tumor voxels by predicted probability and applies geometrically decaying weights only within the binary mask of most probable tumor voxels; other voxels retain weight 1.Source: §A.1 · p. 18
  • With c = 0.5, the least confident voxel in the mask receives half the weight of the most confident, while the most confident half carries about 59% of total tumor-voxel weight.The weighting is normalized to match unweighted cross-entropy scale, and c approaching 0 recovers standard cross-entropy.Source: §A.1 · p. 18
  • The weighted loss zeros gradients at margin voxels, preserving the margin treatment used in the original loss.The normalization ensures the weights within the tumor mask sum to the number of mask voxels.Source: §A.1 · p. 18
  • The Gaussian-tapered Ball Convolution kernel favors positions where tumor probability is concentrated near the kernel center, unlike a uniform ball kernel.The kernel is largest at the center, decreases toward the ball boundary, and is zero outside it.Source: §A.1 · p. 18
  • Ball Convolution uses a report-sized ball kernel for each tumor and sums predicted tumor probabilities within each ball, excluding probabilities outside the organ mask.The uniform binary kernel is insensitive to where probability is concentrated within the ball, motivating the Gaussian taper.Source: §A.1 · p. 18
  • The Gaussian kernel’s standard deviation scales with reported tumor diameter, preserving the same decay shape across diameters.Its normalization keeps outputs comparable across kernel sizes and does not change the location of the convolution maximum.Source: §A.1 · p. 18
  • Setting α = 3 yields a boundary-to-center kernel ratio of about 0.95, providing a mild decay that breaks argmax ties while keeping outputs close to scaled probability sums.The reported ratio is exp(−1/18) ≈ 0.95.Source: §A.1 · p. 18

A.2. Volume Loss for Reports without Tumor Size

R-Super adapts its Volume Loss to reports without tumor sizes or counts by accepting a tolerated tumor-volume range. Its high-tolerance Ball Loss uses incomplete reports to require a small tumor while allowing larger or additional tumors without penalty.

A.2. Volume Loss for Reports without Tumor Size or Count

  • For reports without diameters or counts, high-tolerance Volume Loss assigns zero loss within a tolerated volume range and penalizes segmented volumes outside it.The range is calibrated from reported diameters: V_min = 65 mm3 and V_max = 904,779 mm3, corresponding to spheres with diameters of 5 mm and 120 mm.Source: §A.2 · p. 18

A.3. Ball Loss for Reports without Tumor Size or Count

  • For reports missing diameters or counts, high-tolerance Ball Loss requires at least one small tumor while leaving larger or additional tumors unpenalized.It identifies likely voxels for a 5 mm tumor and selects the most probable voxels, using a voxel count 20% below that of a 5 mm sphere; other voxels inside the organ receive no loss or gradient.Source: §A.2 · p. 19

B. Training Details

Models used 1-mm isotropic CT patches, with report-pair patches centered on a target organ to keep the reported tumor visible. R-Super and most baselines shared the MedFormer backbone and hyperparameters, while nnU-Net used its standard self-configuring setup.

B. Training Details

  • Report-pair patches were centered on a target organ to ensure the reported tumor was visible to the report-based losses.Patches were 128 × 128 × 128 voxels at 1 mm isotropic spacing; the target organ was selected with 80% probability from organs reported as containing tumors.Source: B. Training Details · p. 19
  • Optimization used MedFormer defaults, with R-Super loss weights tuned once and reused across experiments.Training used AdamW, batch size 2, and 100 epochs of 1,000 batches; tuned weights were 1 for Dice and cross-entropy segmentation losses and 0.1 for Volume and Ball Losses. CT intensities were clipped and normalized, with per-sample augmentations; combined training oversampled CT-Mask pairs to 50% of epoch samples.Source: B. Training Details · p. 19
  • R-Super and most baselines shared the MedFormer backbone and hyperparameters, isolating differences in training methodology.Baselines included standard segmentation, MTL, CLIP, Models Genesis, and report-guided pseudo-labels, each trained using its specified data and pretraining or pseudo-mask procedure.Source: B. Training Details · p. 19
  • nnU-Net was the sole baseline with a different architecture and training scheme, using its standard procedure and ResEncL architecture.It was trained on CT-Mask pairs with automatic hyperparameters; isotropic 1-mm spacing was the only non-default setting.Source: B. Training Details · p. 19

C. Result Variability

The section explains how uncertainty is reported across results and why segmentation scores vary substantially across patients. Wide intervals or standard deviations do not by themselves rule out statistically significant differences between methods.

C. Result Variability

  • DSC and NSD are reported with standard deviations, while sensitivity, specificity, F1-Score, and AUC use 95% confidence intervals from 1,000 bootstrap samples.Table 6 also specifies paired permutation tests for F1-Score and DeLong’s tests for AUC; orange marks statistically significant gains over standard segmentation (p<0.05).Source: C. Result Variability · p. 20, Table 6 · p. 20
  • Tumor size, shape, and texture vary across patients, causing segmentation difficulty and large DSC and NSD standard deviations; organ segmentation also has large standard deviations.Wide confidence intervals or standard deviations for individual results do not necessarily mean method differences are statistically insignificant.Source: C. Result Variability · p. 20

D. LLM Prompt

The section presents an organ-adapted LLM prompt, iteratively developed by computer scientists and radiologists, to extract structured lesion information from radiology reports.

D. LLM Prompt

  • The prompt requests a separate entry for each lesion, with its type, certainty, size, organ, location, and attenuation.It directs the model to examine findings, clinical history, and impressions when available.Source: D. LLM Prompt · p. 20
  • The prompt includes benign and malignant lesions, including suspicious lesions, while excluding specified non-focal findings and allowing a justified “No lesions mentioned” response.For suspicious lesions, it instructs the model to mark certainty as low.Source: D. LLM Prompt · p. 20
  • The prompt preserves reported 1D, 2D, or 3D size formats and requires an additional “multiple” entry when multiple lesions in an organ are unspecified.It also calls for individual entries for every specifically described lesion.Source: D. LLM Prompt · pp. 20, 22
  • The prompt standardizes organ names and asks for specified intra-organ locations, using “U” when location is not reported.It gives pancreas regions and kidney sides as examples of locations to capture.Source: D. LLM Prompt · p. 22
  • Type rules infer labels such as cyst, metastasis, malignant, or benign from report context, while certainty distinguishes certain, high, low, and unknown.When a type is available, the prompt asks for its standard name followed by its acronym; attenuation is mapped to prescribed categories or unknown.Source: D. LLM Prompt · p. 22
  • The prompt requires report-section-based justification, sentence-level explanations for size, location, and type, and syntactic analysis distinguishing current from past measurements.For measurements, the analysis should identify the corresponding lesion and whether it is malignant or benign.Source: D. LLM Prompt · pp. 22–23

Related papers

Loading 2608.27668v1…