PaperlayerView exact revision on arXiv

Source-linked AI summary

The autoPET3 Challenge: Automated Lesion Segmentation in Whole-Body PET/CT $\unicode{x2013}$ Multitracer Multicenter Generalization

Jakob Dexl, Katharina Jeblick, Andreas Mittermeier, Balthasar Schachtner, Anna Theresa Stüber, Johanna Topalis, Maximilian Rokuss, Fabian Isensee, Klaus H. Maier-Hein, Hamza Kalisch, Jens Kleesiek, Constantin M. Seibold, Hussain Alasmawi, Lap Yan Lennon Chan, Yixuan Yuan, Alexander Jaus, Rainer Stiefelhagen, Pauline Ornela Megne Choudja, Konstantin Nikolaou, Christian La Fougère, Sergios Gatidis, Matthias P. Fabritius, Maurice Heimer, Gizem Abaci, Lalith Kumar Shiyam Sundar, Rudolf A. Werner, Jens Ricke, Clemens C. Cyran, Thomas Küstner, Michael Ingrisch

arXiv:2605.05775v2cs.CVcs.AI

TL;DR

autoPET3 addresses multitracer, multicenter generalization in automated whole-body PET/CT lesion segmentation. The challenge established a benchmark and released a large annotated PSMA dataset; its analyses identified false positives as a primary improvement area and found cross-ligand generalization within PSMA-LMU.

  • Problem

    Automated PET/CT lesion segmentation must handle domain shifts from different tracers and institutions, motivating explicit evaluation of multitracer, multicenter generalization.

  • Method

    The challenge established a transparent benchmark for automated whole-body PET/CT lesion segmentation using two tracers and centers, with 17 teams submitting 27 algorithms.

  • Results

    False positives remained the primary improvement area, while algorithms generalized across 68Ga-PSMA-11 and 18F-PSMA-1007 within the PSMA-LMU subset.

  • Takeaways & Limitations

    A post-hoc majority-vote ensemble substantially reduced false-positive volume while leaving DSC and false-negative volume largely unchanged.

  • Takeaways & Limitations

    Tracer–center conditions confound scanner hardware, reconstruction protocol, annotation method, and patient demographics, preventing isolation of a single source of variation.

Abstract

from arXiv · show

We report the design and results of the third autoPET challenge (MICCAI 2024), which benchmarked automated lesion segmentation in whole-body PET/CT under a compositional generalization setting. Training data comprised 1,014 [18F]-FDG PET/CT studies from the University Hospital Tübingen and 597 [18F]/[68Ga]-PSMA PET/CT studies from the LMU University Hospital Munich, constituting the largest publicly available annotated PSMA PET/CT dataset to date. The held-out test set of 200 studies covered four tracer-center combinations, two of which represented unseen compositional pairings. A complementary data-centric award category isolated the contribution of data handling strategies by restricting participants to a fixed baseline model. Seventeen teams submitted 27 algorithms, predominantly nnU-Net-based 3D networks with PET/CT channel concatenation. The top-ranked algorithm achieved a mean DSC of 0.66, FNV of 3.18 mL, and FPV of 2.78 mL across all four test conditions, improving DSC by 8% and reducing the false-negative volume by 5 mL relative to the provided baseline. Ranking was stable across bootstrap resampling and alternative ranking schemes for the top tier. Beyond the benchmark, we provide an in-depth analysis of segmentation performance at the patient and lesion level. Three main conclusions can be drawn: (1) in-domain multitracer PET/CT segmentation is sufficient and probably approaching reader agreement; (2) compositional generalization to unseen tracer-center combinations remains an open problem mainly driven by systematic volume overestimation; (3) heterogeneity and case difficulty drive performance variation substantially more than the choice of algorithm among top-ranked teams.

1. Introduction

Whole-body PET/CT lesion segmentation is clinically valuable but difficult because manual delineation is labor-intensive, physiological uptake and imaging artifacts can mimic disease, and performance degrades under multicenter distribution shifts. autoPET3 addresses these challenges through a multitracer, multicenter benchmark focused on generalization to unseen tracer-center combinations.

  • Clinical motivation: Manual or semi-manual lesion segmentation limits the routine adoption of quantitative PET/CT metrics because interpretation can take up to 90 minutes per patient.Typical readings take around 30 minutes.
  • Segmentation challenges: Distinguishing pathological from physiological uptake is difficult because FDG and PSMA tracers have different normal biodistribution patterns.FDG uptake is prominent in the brain, myocardium, and urinary tract, whereas PSMA uptake is prominent in several glands and abdominal or urinary organs.
  • Segmentation challenges: False positives and false negatives arise from tracer-specific pitfalls, small or low-uptake lesions, artifacts, inflammation, and adjacent physiological activity.Examples include brown fat or muscle uptake for FDG, ganglia or bladder activity for PSMA, attenuation-correction artifacts, respiratory misregistration, and partial-volume effects.
  • Generalization gap: Segmentation models often lose performance when applied across institutions because scanner, reconstruction, population, disease, and annotation differences shift jointly.These composite shifts make it difficult to disentangle the source of degraded performance.
  • Challenge contribution: autoPET3 introduces a multitracer, multicenter benchmark in which models must generalize to unseen combinations of tracer type and acquisition site.The challenge also releases 597 annotated PSMA scans, described as the largest publicly available annotated PSMA PET/CT dataset at that time.

2. Related Works

PET segmentation research has progressed from small or narrowly scoped benchmarks and rule-based methods toward large, learning-based whole-body datasets. Prior work shows strong in-domain results but persistent uncertainty about robustness across centers, tracers, and other clinically distinct domains.

  • Earlier challenges: Early PET tumor-segmentation challenges established benchmarking value but used relatively small, heterogeneous datasets with limited clinical representativeness.The first MICCAI PET tumor-segmentation challenge included phantom, synthetic, and clinical images of isolated solid tumors.
  • Earlier challenges: HECKTOR expanded PET/CT segmentation to 883 cases from nine institutions and combined tumor delineation with survival and recurrence prediction tasks.Across its first three editions, the challenge progressed to multiclass segmentation of primary and nodal tumor volumes.
  • autoPET series: The first autoPET iteration provided 1,014 single-site FDG studies and evaluated generalizability using a 150-study test set containing same-hospital and external-hospital cases.It focused on whole-body lesion segmentation in lung cancer, lymphoma, and melanoma patients.
  • autoPET series: autoPET2 extended domain-generalization evaluation across scanner, demographic, pathology, and tracer differences, highlighting limitations on out-of-distribution data.Models were trained on one source distribution and evaluated across multiple clinically distinct target domains, including PSMA.
  • Learning-based methods: PET tumor segmentation has shifted from classical or semi-automatic methods toward fully learning-based approaches as large annotated PET/CT datasets became available.FDG research progressed from candidate-region classification toward end-to-end detection and segmentation.
  • Generalization evidence: External validation remains limited, with prior studies reporting either modest DSC drops or substantial quantitative bias on external cohorts.Reported examples include a 2% DSC drop in one multicenter evaluation and 20.8% TMTV underestimation in another.
  • PSMA segmentation: PSMA segmentation evidence is newer and smaller, ranging from a single-center DSC of 0.44 to multicenter internal DSC of 0.70 with lower external performance.These studies used 3D nnU-Net cascade frameworks on [68Ga]Ga-PSMA-11 scans.

3. Material and Methods

autoPET3 evaluates whole-body PET/CT lesion segmentation across tracers and centers, separating model generalization from data-centric improvements. Its datasets combine heterogeneous FDG and PSMA acquisitions with held-out in-domain and unseen tracer-center pairings.

  • Challenge design: autoPET3 evaluates automated lesion segmentation across two tracers and institutions in a compositional generalization setting.The challenge used separate award categories for unrestricted model development and fixed-model data-centric strategies.
  • Challenge datasets: 1,014 FDG studies from UKT and 597 PSMA examinations from LMU formed the training datasets.The PSMA data included two tracers and three scanners, while the FDG data came from one institution and a standardized protocol.
  • Challenge datasets: 200 held-out studies covered four tracer-center combinations, including 100 studies from unseen cross-center pairings.The cross-center evaluation comprised PSMA from UKT and FDG from LMU.
  • Challenge datasets: PET images were converted to standardized uptake values using tissue activity, injected activity, and patient body weight.SUV was calculated as Ctissue/(Ainj/W).
  • Annotation procedure: Annotations combined 3D specialist delineation with expert verification, but annotation procedures differed between FDGUKT and the other datasets.FDGUKT used slice-by-slice segmentation, whereas the other datasets used thresholding with manual adjustments.
  • Metrics: DSC measured overlap between predicted and ground-truth masks, while FPV and FNV quantified non-overlapping predicted and missed lesion volumes.The evaluation used the same three measures as previous autoPET editions.

4. Results

Seventeen teams submitted 27 algorithms, predominantly 3D nnU-Net-based PET/CT models, and the leading methods showed strong but uneven performance across tracers and centers. Rankings were robust at the top, while reader comparisons and volume analyses exposed domain-specific limitations and oversegmentation.

  • Submitted algorithms: 17 teams submitted 27 algorithms, predominantly using 3D nnU-Net models with concatenated PET and CT inputs.Most AC1 teams used nnU-Net, while only one submitted a 2D U-Net.
  • Overall performance: LesionTracer A ranked first overall, followed by IKIM A and HussainAlasmawi A, with the top three showing balanced DSC across tracers and centers.The data-centric baseline separated a larger well-performing group from algorithms with less stable predictions.
  • Overall performance: Median DSC among well-performing teams ranged from 0.64 to 0.70, while the best DSC occurred on FDGUKT and the worst on PSMAUKT.The most severe false negatives occurred predominantly on LMU datasets, and FPV was generally lower in-domain than on composite datasets.
  • Model combinations: The ensemble would have ranked first, achieving the best DSC and FNV ranks, whereas perfect routing of single-tracer specialists would have ranked only 10th.The ensemble also achieved the third-best FPV rank after AiraMatrix and BAMF.
  • Ranking stability: Bootstrap analysis separated algorithms into two tiers, with LesionTracer A consistently leading while many subsequent algorithms had overlapping ranking intervals.IKIM showed greater ranking variation than LesionTracer A.
  • Ranking stability: Across five ranking methods, top-ranked teams remained in positions 1–4, bottom-tier teams remained stable, and mid-field teams shifted by up to five positions.No lower-tier algorithm crossed above the data-centric baseline under any ranking method.

4.5. Patient-level analysis

Patient-level difficulty dominated algorithm choice: case-to-case variation was substantial, while team differences were small. Lesion volume and dataset condition were associated with DSC, and out-of-domain cases showed systematic volume overestimation.

  • Patient-level heterogeneity: Patient-level median DSC ranged from zero to above 0.9 across the top-18 algorithms.Near-zero cases generally contained only one small lesion, whereas FDGUKT cases had higher median scores and narrower interquartile ranges.
  • Factors driving segmentation performance: 61% of unexplained DSC variance came from patient heterogeneity, versus 1.3% from mean algorithmic differences.The expected DSC gap between randomly selected patients was roughly seven times the gap between randomly selected algorithms.
  • Factors driving segmentation performance: Each doubling of reference lesion volume was associated with a +0.039 increase in DSC.After volume adjustment, UKT had a +0.14 center effect, while tracer and tracer×center interaction were not significant.
  • Ablation PSMA ligands: No performance difference was observed between 68Ga-PSMA-11 and 18F-PSMA-1007 within the PSMA-LMU subset.The estimated difference was β = −0.03 with 95% CI [-0.21, 0.16].
  • Patient-level classification: True-positive counts ranged from 136–152 of 156 positive cases, while true-negative counts ranged from 0–35 of 44 negative cases.LesionTracer A had sensitivity 0.97, specificity 0.27, and accuracy 0.82.
  • Volume estimation: Out-of-domain datasets were systematically shifted toward oversegmentation, reaching roughly 1.7 times the reference volume.In-domain medians were generally near zero, while out-of-domain predictions also showed greater within-algorithm variance.

4.6. Lesion-level analysis

Lesion-level performance depended strongly on matching strictness, lesion characteristics, and test condition. Sensitivity fell under stricter overlap criteria, while larger and more avid lesions were detected more reliably; qualitative errors included merges, splits, false positives, and false negatives.

  • Lesion detection sensitivity: Median lesion-detection sensitivity fell from approximately 0.83 at the one-voxel criterion to 0.48 at IoU τ = 0.5.The sharp increase near the one-voxel threshold indicates that many detections relied on marginal overlap.
  • Dataset-condition effects: FDGLMU started at 0.74 sensitivity under the one-voxel criterion and declined more steeply across increasing IoU thresholds.The other three dataset conditions started above 0.84.
  • Structural error analysis: At the one-voxel threshold, median merge, split, and split-merge associations were 146, 92, and 24, respectively.As τ increased, these ambiguous associations decayed toward correct detections, detection failures, and false alarms.
  • Factors influencing lesion detection: Detection sensitivity increased with lesion volume and tracer uptake across all four test conditions.Lesions smaller than 0.1 mL were detected at roughly 40–60%, while lesions with SUVmax above 15 exceeded 95% detection.
  • Factors influencing lesion detection: SUVmax below 4.1 incurred a detection rate below 40%, whereas SUVmax above 15 exceeded 95%.SUVmax appeared to be the dominant driver at the one-voxel criterion, with greatest inter-team variability for SUVmax between 4 and 8.
  • Qualitative error analysis: Representative errors included elevated FPV of 1.68–3.01 in an FDGUKT skeletal-metastasis case and high FNV of 1.11–1.81 in a PSMALMU case.The latter case contained annotated lesions with minimal tracer uptake.
  • Qualitative error analysis: PSMA datasets concentrated uptake in bones and near the bladder, while many false positives appeared in the head and extremities.Lacrimal-gland uptake was identified as a systematic error source in PSMAUKT.

5. Discussions and conclusion

autoPET3 found that top algorithms perform well on in-domain multitracer PET/CT, while unseen tracer-center combinations remain difficult. Case heterogeneity, false positives, evaluation design, and confounded challenge conditions qualify both performance interpretation and clinical use.

  • Methodological findings: 17 teams submitted 27 algorithms, with most using nnU-Net and PET/CT channel concatenation; no single architectural choice explained the performance gap.Twelve of 15 AC1 teams used nnU-Net, while teams varied in backbones, fields of view, training, sampling, organ masks, and preprocessing.
  • Methodological findings: Data-centric submissions performed only slightly better than the baseline, while limited participation and implementation constraints weaken conclusions about data-handling strategies.The fixed field of view and backbone, heterogeneous strategies, and migration from nnU-Net to MONAI may have restricted the track.
  • Performance and biomedical findings: In-domain performance was clinically relevant for most cases and likely approached reader variability, although stronger reader studies are needed for confirmation.FDGUKT showed nearly all relative volume differences within ±20% for lesions larger than 1 mL, while PSMALMU performance was somewhat lower.
  • Performance and biomedical findings: Unseen tracer-center combinations produced distinct errors, including missed small lesions, over-segmentation, and false positives from rare or unseen physiological patterns.Removing the head region did not eliminate elevated false-positive volume, confirming that out-of-domain generalization remains difficult.
  • Performance and biomedical findings: False positives remain the primary improvement target, but majority-vote ensembling substantially reduced false-positive volume while leaving DSC and false-negative volume largely unchanged.Errors included both annotation omissions and clinically obvious uptake from contamination, muscle activation, glands, or the bladder.
  • Performance and biomedical findings: Patient heterogeneity explained 61% of unexplained DSC variance, versus 1.3% for team differences, shifting attention toward case triage and physician verification.Long-tailed difficult cases make mean metrics underestimate typical performance.
  • Performance and biomedical findings: Current models should not be used for binary classification because many flagged uptake in lesion-free cases, often with small associated volumes.Post-treatment changes can produce suspicious uptake patterns in negative cases.
  • Challenge validity and metrics: Top-tier rankings were broadly stable across bootstrap resampling and alternative methods, but subgroup ranking and sensitivity-weighted composite metrics introduce documented biases.The metric design undervalues false-positive performance and can incentivize small positive volumes.

CRediT authorship contribution statement

The contribution statement assigns conceptualization, data, analysis, methodology, software, validation, visualization, project administration, resources, and writing roles across the listed authors.

  • Jakob Dexl is credited with conceptualization, data curation, formal analysis, investigation, methodology, project administration, resources, software, validation, visualization, and writing.
  • Katharina Jeblick and Andreas Mittermeier are credited with conceptualization, data curation, project administration, resources, software, and writing.
  • The statement credits additional authors with methodology, resources, software, data curation, validation, and writing contributions.

Declaration of competing interest

One author reports a relationship with Novartis involving speaking and lecture fees; the other authors report no known competing interests or relationships.

  • Rudolf A. Werner reports a relationship with Novartis involving speaking and lecture fees.
  • The other authors declare no known competing financial interests or personal relationships that could have influenced the reported work.

Declaration of generative AI and AI-assisted technologies in the manuscript preparation process

The authors used generative AI and language tools for manuscript language, clarity, structure, analysis, visualization code, and programming support, then reviewed and edited the resulting content.

  • Opus 4.6, GPT 5.2, and Grammarly were used to improve language, clarity, and structure during manuscript drafting.
  • Opus 4.6 also supported refinement of analysis and visualization code as a programming companion.
  • The authors state that they reviewed and edited the outputs and retain full responsibility for the published article.

Appendix A. Top performing teams

The appendix describes top-performing challenge approaches spanning dual-headed, tracer-specific, vanilla, synthetic-data, and pruning-based strategies. Most methods build on nnU-Net while varying tracer handling, normalization, augmentation, data synthesis, or dataset filtering.

  • Lesion segmentation approaches: LesionTracer uses a dual-headed nnU-Net with separate lesion- and organ-segmentation heads, trained on 3D patches with global PET normalization.Its augmentation includes standard nnU-Net transforms and a novel misalignment augmentation.
  • Lesion segmentation approaches: IKIM first classifies the tracer from PET maximum-intensity projections, then applies a tracer-specific nnU-Net for lesion segmentation.The classifier uses fused features from coronal and sagittal ResNet18 backbones.
  • Lesion segmentation approaches: HussainAlasmawi uses a vanilla nnU-Net trained only on tumor-positive patients, with submissions differing in PET normalization and loss aggregation.The variants use either z-score or global normalization and different Dice-loss formulations.
  • Data-centric approaches: Lennonlychan adapts DiffTumor to generate paired PET/CT latent samples conditioned on lesion and organ masks for data augmentation.The approach trains an autoencoder followed by a latent diffusion model on AutoPET PET/CT data.
  • Data-centric approaches: ZeroSugar prunes training data using fixed-model losses and segmentation metrics to address PSMA-specific FPV shifts and lesion-presence imbalance.Its analysis identified greater false-positive volume and different lesion prevalence patterns in PSMA studies than in FDG studies.

Appendix B. Mixed-effects model specifications

The appendix specifies mixed-effects models for DSC and radionuclide effects, including tracer, center, lesion volume, patient, and team terms. Reported variance components indicate substantially larger patient-level than team-level random-effect variability in both models.

  • Estimation: The models were fitted as linear mixed models using lme4 in R with restricted maximum likelihood.The appendix reports approximate 95% confidence intervals for the fitted models.
  • Model specifications: The DSC model uses tracer-by-center interaction, log2 lesion volume, and random intercepts for team and patient.Its formula is dsc ∼ tracer * center + log2(V) + (1 | team) + (1 | patient).
  • Variance components: Patient random-effect variability exceeds team variability in the DSC model, with standard deviations of 0.171 and 0.025, respectively.The model includes 2,808 observations from 156 patients and 18 teams.
  • Model specifications: The radionuclide model for the PSMA–LMU subset uses radionuclide as a fixed effect with patient and team random intercepts.Its formula is dsc ∼ radionuclide + (1 | patient) + (1 | team).

Appendix C. Additional tables and figures

Appendix C documents ranking rules, alternative metrics, ablations, and case- and lesion-level performance analyses. These materials show substantial variation across dataset conditions, including larger variance and lower DSC for PSMAUKT and reduced detection of very small or low-uptake lesions.

  • Ranking and evaluation: Official rankings combine per-metric ranks and retain only the best-performing submission from each team.The bottom three algorithms are excluded from ranking, while bold values mark the best performance per column.
  • Additional metrics and ablations: Additional analyses report voxel-level, lesion-level, whole-sample, lesion-positive, and classification metrics across the four test conditions.The classification columns include correctly identified lesion-positive and lesion-negative cases.
  • Additional metrics and ablations: The pathology ablation removes approximately 50 voxels covering the lacrimal gland region from each PSMAUKT case and reports score differences.Reported outcomes include DSC and FPV before and after removal.
  • Per-algorithm performance: Per-algorithm plots stratify DSC, false-negative volume, and false-positive volume by FDGUKT, PSMALMU, FDGLMU, and PSMAUKT.FNV and FPV use inverted logarithmic scales, so higher boxes represent lower volumetric errors.
  • Per-case performance: Per-case distributions order patients by average DSC and identify a cluster of near-zero DSC cases associated with single small lesions.Volume axes are logarithmic and box plots are color-coded by dataset condition.
  • Per-case performance: PSMAUKT consistently exhibits larger variance and lower DSC than the other dataset conditions.The comparison appears in the per-case performance analysis.
  • Lesion detection: Lesion-level sensitivity begins above 0.84 at the one-voxel criterion in three conditions, whereas FDGLMU begins near 0.74 and declines more steeply.The figure also reports detected-lesion counts and highlights the top three teams.
  • Detection errors: Detection-error analyses separate correct detections, merges, false alarms, splits, split-merge clusters, and detection failures across IoU thresholds.The figure also tracks cumulative false-positive and false-negative volume.
Loading 2605.05775v2…