Source-linked AI summary
BS: Take the Hint - Interactive Multitracer PET/CT Lesion Segmentation with a Scribble-Conditioned ResEnc U-Net
Marven Sherif, Amgad Elmasry, Youssef Ghazal, Ayman Elghotni
TL;DR
Whole-body PET/CT lesion segmentation must handle differing tracer uptake patterns, while autoPET/CT V requires models to use interactive foreground and background scribbles. The paper presents a scribble-conditioned residual encoder U-Net with tracer-agnostic PET normalization and fold-specific training. Across five-fold validation, interaction substantially improves segmentation, with most improvement arriving after the first scribble, although the evaluation excludes lesion-free studies and does not measure the submitted ensemble directly.
Problem
Whole-body PET/CT lesion segmentation is complicated by differing physiological tracer uptake patterns, and autoPET/CT V requires algorithms to exploit supplied foreground and background scribbles.
Method
A scribble-conditioned residual encoder U-Net uses CT, PET, and separate foreground and background scribble channels, with PET normalized against a per-scan aortic blood-pool reference.
Results
Dice rises from 0.5539 unaided to 0.7512 after five scribbles, while lesion-level F1 rises from 0.5281 to 0.7326, with about 85% of the gain arriving after the first scribble.
Takeaways & Limitations
Interactive correction consistently improves segmentation and primarily recovers lesions initially missed rather than suppressing spurious predictions.
Takeaways & Limitations
The reported cross-validation results are single-model results, while the submitted five-fold ensemble cannot be scored under the protocol.
Abstract
from arXiv · showhide
Automated lesion segmentation in whole-body PET/CT is complicated by the variety of physiological tracer uptake patterns and by the differing appearance of lesions across tracers. The autoPET/CT V challenge addresses this by making segmentation interactive: user scribbles marking foreground and background are supplied alongside the image, and the algorithm is expected to exploit them. We present our submission, a scribble-conditioned residual encoder U-Net operating on four input channels: CT, PET, and a sparse scribble map for each of foreground and background. The network is initialised from the autoPET-III winning weights and extended from two to four input channels, with the two scribble channels zero-initialised so that the pretrained representation is preserved exactly at initialisation. Every model is fine-tuned per fold from the corresponding autoPET-III fold checkpoint, so that no validation case is seen during pretraining. PET intensities are normalised against a per-scan aorta blood-pool reference derived from a CT segmentation, which removes tracer- and centre-specific scaling without requiring lesion labels. At inference the five fold models are ensembled by averaging their softmax outputs per sliding-window patch, before Gaussian-weighted stitching. On the challenge's five-fold split, with each fold evaluated on its own validation cases, mean Dice is 0.554 and mean lesion-level F1 is 0.528 without scribbles, rising to 0.751 and 0.733 after five correction rounds. About 85% of that gain follows the first scribble, and the spread between fold models narrows five-fold over the same rounds, so interaction largely compensates for how well or badly a given model segments unaided.
1 Introduction
Whole-body PET/CT lesion segmentation is clinically important but difficult to automate across tracers and physiological uptake patterns. autoPET/CT V therefore evaluates interactive models that use foreground and background scribbles, and this submission reports improving quality across correction rounds.
- Manual whole-body lesion delineation is slow and subject to inter-reader variation.
- autoPET/CT V is multitracer, combining 18F-FDG and 68Ga/18F-PSMA studies with substantially different physiological uptake patterns.
- The challenge is interactive: algorithms receive foreground and background scribbles marking lesion and non-lesion voxels.
- The submission uses a scribble-conditioned three-dimensional residual encoder U-Net initialized from the autoPET-III winning solution and fine-tuned on challenge data.
- Segmentation quality improves consistently across interactive correction rounds, with most of the gain arriving after the first scribble.
2 Methods
The study uses only challenge-provided whole-body PET/CT data and evaluates fold-specific models on held-out validation cases. The same five-fold split and corresponding autoPET-III checkpoints support per-fold initialization without validation-case exposure during training.
- The dataset contains 1611 whole-body PET/CT studies with manual lesion annotations and no external or private data.It includes 1014 18F-FDG and 597 PSMA examinations.
- The PSMA cohort combines 18F-labelled and 68Ga-labelled PSMA ligand studies.
- Five-fold splits divide the studies into training and validation sets, with each study predicted once by a model that did not see it during training.
- Each fold is initialized from the corresponding autoPET-III reference-solution checkpoint using the same distributed split.
2.2 Data pre-processing
Pre-processing places CT and PET on a common grid while standardizing their intensities. PET uses a robust, label-free aortic blood-pool reference to make normalization tracer-agnostic.
- All channels are resampled onto a common grid with 2.0364 × 2.0364 × 3.0 mm voxel spacing.
- CT intensities are clipped to [−811.73, 1137.42] HU and standardized using dataset-wide mean and standard deviation.
- PET standardized uptake values are divided by a per-scan reference, transformed with inverse hyperbolic sine, and then dataset-wide z-scored.
- The PET reference is robust aortic blood-pool activity derived from a CT segmentation, with fallback references for insufficient positive uptake.This makes normalization label-free and tracer-agnostic.
Scribbles.
Foreground and background scribbles are supplied as separate sparse volumes on the PET grid. They are generated from connected components, represented as planar curves in single axial slices, and resampled for model input.
- Reference scribbles sample up to k = 5 annotation components using 26-connectivity and select the dominant axial slice for each.
- Each scribble is a planar curve confined to one axial slice rather than a three-dimensional structure.
- Scribble maps are binary point maps with σ = 0 and extremely sparse occupancy of approximately 10^-6 of the volume.
- Foreground and background scribbles are stored separately, resampled with linear interpolation, clipped to [0, 1], and quantised to uint8.
- Lesion masks are resampled to the target spacing with nearest-neighbour interpolation.
2.3 Algorithm/model
The submission uses a four-channel three-dimensional residual encoder U-Net that incorporates CT, PET, and separate foreground/background scribble maps. It preserves the pretrained two-channel function at initialization while learning the scribble pathways during fold-specific fine-tuning.
- The model is a three-dimensional ResEncL residual encoder U-Net with four input channels, two output classes, and deep supervision.
- The four channels are CT, normalized PET, foreground scribble, and background scribble maps.
- The input stem expands from two to four channels by zero-initializing the added scribble weights, preserving the pretrained function exactly at initialization.
- Each fold starts from its corresponding autoPET-III checkpoint, while the auxiliary organ segmentation head is disabled during fine-tuning.
2.4 Data post-processing
Inference processes resampled volumes with overlapping sliding windows, combines patch probabilities using Gaussian center weighting, and thresholds after resampling to preserve small lesions.
- Inference uses 192 × 192 × 192 sliding-window patches with 0.5 overlap, corresponding to a stride of 96 voxels along each axis.Short axes are zero-padded and the output is cropped afterward.
- Gaussian importance weighting down-weights poorly contextualized patch borders relative to patch centers before probabilities are stitched.The Gaussian standard deviation is 0.125 of the patch size, or 24 voxels for a 192-voxel patch.
- The final probability volume is the element-wise ratio of weighted class-probability and weight accumulators, making it independent of window coverage count.
- The foreground probability is resampled to the acquisition grid and thresholded at 0.65 only afterward to produce the binary lesion mask.Post-resampling thresholding avoids nearest-neighbour interpolation of a binary mask, which could discard small lesions.
2.5 Training and test parameters
Training combines cross-entropy with a false-negative-weighted soft Tversky term, uses stochastic lesion-aware patch sampling, and selects the lowest-validation-loss checkpoint for five-fold ensemble inference.
- Each fold trains for 500 epochs using SGD with learning rate 10−3, Nesterov momentum 0.99, weight decay 3 × 10−5, batch size 2, and gradient clipping at norm 12.No learning-rate schedule is used, and training runs in mixed precision.
- The loss combines cross-entropy with a soft Tversky term using α = 0.3, β = 0.7, and ϵ = 10−5.The Tversky term is averaged over foreground classes, while cross-entropy supervises the background.
- The Tversky asymmetry penalizes false negatives more heavily than false positives, reflecting the stated clinical cost of missing a lesion.
- Training patches are sampled stochastically, including hard-negative tumour-free windows with probability 0.33 and lesion-centered windows otherwise.
- The lowest-validation-loss checkpoint is selected per fold, and five fold models average softmax outputs per patch before Gaussian stitching.
2.6 Github repository
The paper provides a GitHub repository for the autoPET/CT V submission.
- The submission code is linked at the listed GitHub repository.
3 Results
Five-fold evaluation measures interactive segmentation with volumetric Dice and lesion-level F1 across six rounds, showing large gains after scribble correction and reduced variation between fold models.
- Metrics and scoring: Both metrics are computed over tumour-bearing cases because lesion-free annotations produce NaN, and lesion-level F1 counts detections using connected components with IoU at least 0.1.Dice is evaluated on the original acquisition grid; the reported protocol excludes lesion-free studies from both averages.
- Evaluation protocol: The six-round evaluation uses r0 without scribbles followed by five rounds that append one error-derived scribble per round.The challenge loop draws candidates from false-positive and false-negative regions.
- Overall performance: 0.5539 Dice and 0.5281 lesion-level F1 were achieved without scribbles, averaged over the five validation folds.Each fold model was evaluated only on its own held-out validation cases.
- Interactive correction: 0.7512 Dice and 0.7326 lesion-level F1 were reached after five correction rounds, corresponding to gains of +0.1973 and +0.2045.The first scribble accounted for 85% of the total gain in both metrics.
- Interactive correction: Dice increased at every round on all five folds, while F1 increased on four folds and dipped by 0.0006 once on fold 4.At the individual-study level, 497 of 1040 cases improved in F1, 461 were unchanged, and 82 deteriorated from r0 to r5.
- Robustness across folds: The fold-to-fold Dice spread narrowed from 0.1729 unaided to 0.0344 after five rounds, a five-fold compression.The corresponding F1 spread fell from 0.1748 to 0.0371, and fold ordering was not preserved.
4 Discussion
Interactive correction consistently improves segmentation, with most of the gain arriving after the first scribble and arising from recovering missed lesions. PET blood-pool normalisation supports the multitracer setting, while evaluation remains subject to ensemble, threshold, cohort, and simulator limitations.
- Segmentation quality increases with every interactive round, showing that the model follows scribbles and progressively corrects its own errors.
- +0.1973 Dice and +0.2045 F1 are obtained across the loop, with roughly 85% of the gain supplied by the first scribble.The four subsequent rounds refine rather than transform the segmentation.
- 829 false negatives become true positives after five rounds, while false positives increase slightly from 6869 to 6998.The improvement is therefore almost entirely due to recovering missed lesions rather than suppressing spurious ones.
- Aorta blood-pool PET normalisation is label-free and tracer-independent, addressing differing uptake scales in the combined FDG and PSMA cohort.
- The cross-validation results do not measure the submitted ensemble, reuse a fold-0 threshold, exclude 571 lesion-free studies, and use optimistic ground-truth-derived scribbles.The background channel also limits further gains because evaluation-style false-positive scribbles differ from the training boundary shells.
5 Conclusion
The submission is a scribble-conditioned three-dimensional residual encoder U-Net with per-fold pretrained initialisation, zero-initialised scribble pathways, and five-fold ensemble inference. Cross-validation shows consistent interactive improvement, with most gains from the first scribble and reduced variation between fold models.
- The method extends a pretrained two-channel residual encoder U-Net to four channels with zero-initialised foreground and background scribble pathways.It uses a false-negative-weighted Tversky and cross-entropy objective and averages fold predictions per sliding-window patch.
- 0.5539 to 0.7512 Dice and 0.5281 to 0.7326 lesion-level F1 are reported from unaided prediction to five scribbles.About 85% of the gain arrives with the first scribble, while fold-model spread narrows five-fold.
- All authors are employees of Brightskies and report no relevant competing interests or affiliations with challenge-organizer institutes.