Source-linked AI summary
Deep learning to achieve clinically applicable segmentation of head and neck anatomy for radiotherapy
Stanislav Nikolov, Sam Blackwell, Alexei Zverovitch, Ruheena Mendes, Michelle Livne, Jeffrey De Fauw, Yojan Patel, Clemens Meyer, Harry Askham, Bernardino Romera-Paredes, Christopher Kelly, Alan Karthikesalingam, Carlton Chu, Dawn Carnell, Cheng Boon, Derek D'Souza, Syed Ali Moinuddin, Bethany Garie, Yasmin McQuinlan, Sarah Ireland, Kiarna Hampton, Krystle Fuller, Hugh Montgomery, Geraint Rees, Mustafa Suleyman, Trevor Back, Cían Hughes, Joseph R. Ledsam, Olaf Ronneberger
TL;DR
Manual delineation of head and neck organs at risk is time-consuming and variable, motivating clinically applicable auto-segmentation. This paper evaluates deep learning segmentation against experts, introduces surface DSC for clinically relevant contour comparison, and reports human-level performance with generalisability across unseen datasets.
Problem
Manual delineation of head and neck organs at risk is time-consuming and can introduce inter-operator variability in radiotherapy planning.
Method
The study develops a deep learning OAR segmentation algorithm for routine clinical planning CT scans and evaluates it using surface DSC against segmentations from independent experts across multiple datasets.
Results
The algorithm achieved human expert-level segmentation performance for head and neck OARs and maintained human performance on 19 of 21 OARs across previously unseen international sites.
Takeaways & Limitations
The algorithm has potential to improve the speed, efficiency, and consistency of radiotherapy workflows and support hospital-based OAR segmentation and quality assurance.
Takeaways & Limitations
The study used planning CT scans only, excluding certain OAR classes that require MRI for optimal delineation; protected-characteristic status was also unavailable for fairness analysis.
Abstract
from arXiv · showhide
Over half a million individuals are diagnosed with head and neck cancer each year worldwide. Radiotherapy is an important curative treatment for this disease, but it requires manual time consuming delineation of radio-sensitive organs at risk (OARs). This planning process can delay treatment, while also introducing inter-operator variability with resulting downstream radiation dose differences. While auto-segmentation algorithms offer a potentially time-saving solution, the challenges in defining, quantifying and achieving expert performance remain. Adopting a deep learning approach, we demonstrate a 3D U-Net architecture that achieves expert-level performance in delineating 21 distinct head and neck OARs commonly segmented in clinical practice. The model was trained on a dataset of 663 deidentified computed tomography (CT) scans acquired in routine clinical practice and with both segmentations taken from clinical practice and segmentations created by experienced radiographers as part of this research, all in accordance with consensus OAR definitions. We demonstrate the model's clinical applicability by assessing its performance on a test set of 21 CT scans from clinical practice, each with the 21 OARs segmented by two independent experts. We also introduce surface Dice similarity coefficient (surface DSC), a new metric for the comparison of organ delineation, to quantify deviation between OAR surface contours rather than volumes, better reflecting the clinical task of correcting errors in the automated organ segmentations. The model's generalisability is then demonstrated on two distinct open source datasets, reflecting different centres and countries to model training. With appropriate validation studies and regulatory approvals, this system could improve the efficiency, consistency, and safety of radiotherapy pathways.
1. Introduction
Head and neck radiotherapy depends on accurate OAR delineation, yet manual contouring is time-consuming and variable. Deep learning is proposed to address limitations of existing automated methods and improve clinically meaningful evaluation.
- Clinical motivation: 550,000 people are diagnosed with head and neck cancer worldwide each year, and radiotherapy can cause dose-dependent adverse effects through incidental irradiation of OARs.
- Clinical motivation: Accurate OAR and tumour segmentation is essential for radiotherapy efficacy and safety, but manual contouring produces substantial variability even among experts.
- Clinical motivation: Four hours or more may be required to segment a single case, contributing to treatment delays associated with increased local recurrence and overall mortality risk.
- Existing automation: Atlas-based algorithms may be prone to systematic error because they inadequately account for post-surgical changes, anatomical variation, and tumour effects on local anatomy.
- Existing automation: Existing clinical algorithms often require substantial manual editing, achieve expert performance for only a small number of organs, and have not significantly improved workflows.
- Study approach: The study addresses unresolved standards for expert performance and clinical acceptability by evaluating deep learning across many OARs with a clinically meaningful metric and independent expert comparisons.
2. Results
The study evaluates a deep-learning segmentation model against expert radiographers on clinically representative and independent datasets, using surface DSC to measure clinically relevant contour agreement. Performance was similar to humans across all 21 OARs on UCLH scans and remained within human variability for 19 of 21 OARs on TCIA scans.
- Datasets: The evaluation used UCLH training, validation, and held-out test subsets, alongside open-source TCIA and PDDCA datasets to assess generalisability.The UCLH scans came from routine curative-intent radiotherapy planning at a single high-volume centre; TCIA and PDDCA provided data from previously unseen distributions.
- Evaluation design: The study compared model segmentations with radiographer segmentations against a gold standard produced by independent radiographers and specialist oncologists.The model and radiographers were evaluated independently on test images, while the gold standard used further radiographers and oncologist arbitration.
- Clinical metric: Surface DSC measures agreement between predicted and reference organ surfaces within organ-specific tolerances, rather than overlap between volumes.The metric normalises acceptable surface portions by combined surface area and penalises both false-positive and false-negative surface deviations.
- Clinical metric: The organ-specific tolerance τ was derived from inter-observer surface variation, using the 95th percentile of distances measured across expert segmentations.The tolerance defines the maximum deviation accepted without penalty and is calculated separately for each OAR.
- UCLH performance: On the UCLH held-out test set, the model showed no clinically meaningful difference from radiographers across all 21 OARs.Figure 4 compares model and radiographer surface DSC values at organ-specific tolerances, with differences interpreted against a -5% to +5% non-substantiality range.
- Generalisability: On the independent TCIA test set, the model performed within the threshold defined for human variability in 19 of 21 OARs.Performance was lower for the brainstem and right lens than in UK data, possibly because several TCIA scans had poorer image quality.
3. Discussion
The study presents a deep learning segmentation algorithm designed for clinical head and neck radiotherapy, with performance comparable to experienced radiographers and evaluated across varied datasets. It also emphasizes surface DSC as a clinically relevant metric, while identifying limits in comparison, imaging scope, fairness assessment, and clinical workflow evidence.
- Clinical performance: The model performed as well as experienced radiographers for head and neck radiotherapy planning.It was developed using CT scans derived from routine clinical practice.
- Generalisability: Human-level performance was maintained on 19 of 21 OARs across previously unseen international datasets.The right lens and brainstem were below radiographer performance, potentially because of lower image quality and unclear brainstem boundaries.
- Evaluation metric: Surface DSC was introduced to reflect the clinical task of correcting OAR contours more closely than volumetric DSC.Surface DSC is more sensitive to clinically significant border deviations and has been associated with the time required to correct segmentations.
- Evaluation metric: Volumetric DSC remains useful for literature comparison, although differing datasets and labelling protocols make published comparisons difficult.The model and radiographers achieved similar scores for OARs where the model compared unfavourably with published results.
- Limitations: Clinical acceptability, time savings, and intersectional fairness remain to be evaluated in future clinical studies.The test sets lacked protected-characteristic status, preventing study of intersectional fairness.
- Limitations: The study used planning CT scans and omitted some OAR classes because of limited conforming training examples.Future work will incorporate MRI, while the ground truth may not fully capture segmentation variability or may reflect a particular atlas interpretation.
4. Methods
The study used clinically sourced UCLH CT scans and independently labelled open-source datasets to train, validate, and test segmentation of 21 head and neck organs at risk. It also introduced a surface-based metric and less biased linear-complexity implementation for evaluating delineation.
- Datasets: UCLH scans were split into training, validation, and test sets, with 663 scans allocated to training and 75 to testing.The UCLH cohort included 486 patients and 838 scans meeting inclusion criteria.
- Datasets: The study added TCIA and PDDCA datasets to assess generalisability across different centres, countries, selection criteria, and imaging characteristics.TCIA scans were filtered to CT planning scans with 2.5 mm slice thickness, producing 31 scans divided into validation and test sets.
- Clinical labelling & annotation: Twenty-one organs at risk were selected from the Brouwer Atlas to represent diverse head and neck anatomical regions relevant to routine radiotherapy.Regions requiring additional MRI, lacking routine relevance, or not used clinically at UCLH were excluded.
- Clinical labelling & annotation: Training segmentations were reviewed against the Brouwer Atlas, while test and validation ground truths were densely labelled and independently arbitrated by experienced radiographers.The TCIA ground truths were initially segmented by one experienced radiographer, arbitrated by another, and further reviewed by a radiation oncologist.
- Performance metrics: Surface DSC evaluates agreement between organ surfaces at a specified tolerance rather than between enclosed volumes.This formulation targets contour differences relevant to manual correction of automated segmentations.
- Implementation of surface DSC: The surface implementation uses points shifted between voxels to reduce rasterisation bias while retaining linear complexity, O(N), in the number of voxels.Foreground- and background-voxel surface representations can respectively underestimate or overestimate surface area.
5. Code availability
The deep learning framework contains proprietary components and cannot be publicly released, but the methods provide sufficient implementation detail for independent replication with non-proprietary libraries.
- Code availability: The surface DSC performance-metric code is publicly available, while the proprietary deep learning framework code is not.The released code is hosted at https://github.com/deepmind/surface-distance.
6. Data availability
Clinical training and validation data were collected and de-identified at UCLH under local and national permissions but are not publicly available.
- Data availability: Access to the UCLH data is restricted and may require local and national ethical approvals.Released test and validation data came from datasets hosted on The Cancer Imaging Archive.
8. Author contributions
The project involved distinct contributions spanning initiation, dataset creation, software engineering, model architecture, clinical expertise, project management, and manuscript writing.
- Author contributions: Contributors created the dataset, engineered software, designed model architectures, manually segmented images, supplied clinical expertise, managed the project, and wrote the paper.The author-contribution statement assigns these activities across the listed collaborators.
9. Competing financial interests
The authors disclose that several contributors were paid contractors of DeepMind and/or Google Health, with no other competing interests reported.
- G.R., H.M. and the D.R.C. were paid contractors of DeepMind and/or Google Health.
- The authors report no other competing interests to disclose.
- The disclosure identifies affiliations involving both DeepMind and Google Health.