Development and Validation of a Deep Learning Algorithm for Improving Gleason Scoring of Prostate Cancer

Introduction

• 30–53% Gleason score discordance has been reported, reflecting the subjectivity and suboptimal inter- and intraobserver variability of a scoring system central to prognosis and patient management.Gleason score and tumor stage are powerful predictors of prognosis, and Gleason score informs clinical risk stratification and management guidelines.Source: Introduction · p. 3
• Prior AI work addressed prostate cancer detection in needle biopsies and grading of research tissue microarrays, but not Gleason grading of specimens used for clinical diagnosis.The microarrays comprised selected tumor subregions outside routine clinical workflow.Source: Introduction · p. 3
• Expertise and consistency in Gleason scoring improve its prognostic utility, motivating a whole-slide clinical tool to address grading variability, improve prognostication, and optimize patient management.9,30Source: Introduction · pp. 3–4
• The study developed a DLS for Gleason scoring and quantitation on prostatectomy specimens, comparing accuracy with pathologists against a genitourinary specialist reference and evaluating progression risk stratification.It also explored finer-grained tumor grading measures and their potential for more precise prognostication.Source: Introduction · p. 4

Overview of the Deep Learning System (DLS) and Data Acquisition

• The two-stage DLS used a deep convolutional neural network for regional Gleason-pattern classification, followed by k-nearest-neighbor whole-slide grade-group classification; training used 112 million patches from 912 slides and 1,159 pathologist-provided slide-level classifications.Source: Results · p. 4

Comparison of DLS to pathologists on Whole-Slide Gleason Scoring

• 0.70 accuracy (95% CI 0.65–0.75) exceeded the 29 pathologists’ mean 0.61 (95% CI 0.56–0.66; p=0.002), and the DLS outperformed 8 of 10 pathologists who reviewed all slides.The other 19 pathologists reviewed overlapping subsets, with individual accuracies ranging from 0.31 to 0.74 (mean 0.60).Source: Results · p. 5
• 0.95–0.96 AUCs were achieved at each evaluated threshold; at GG≥4, the DLS had higher sensitivity and specificity than 9 of 10 pathologists.Source: Results · p. 5

Comparison of DLS to pathologists on Gleason Pattern Quantitation

• 4–6% lower mean absolute error than the average pathologist was observed for quantitating patterns 3 and 4, and 8% lower error for grade groups 2 and 3.The trend was similar for grade groups 4 and 5, where pattern 5 quantitation is significant.Source: Results · pp. 5–6

Insights from DLS Region-Level Classifications

• 97% of regions with unanimous pathologist classifications matched the DLS classification, and for regions labeled as a Gleason pattern, the DLS selected the same pattern 88% of the time.Source: Results · p. 6
• DLS prediction scores changed smoothly across pathologist classification distributions, and fine-grained patterns such as GP3.3 or GP3.7 represented gradual differentiation transitions beyond canonical pattern categories.Source: Results · p. 6

Measuring Effectiveness of Gleason Scoring in Risk Stratification for Disease Progression

• 0.65 c-index for the DLS-predicted grade group compared with a 0.63 median for pathologist-provided grade groups and 0.69 for the specialist reference; Kaplan–Meier and hazard-ratio analyses showed the same trend.Source: Results · p. 7
• 0.697 c-index was obtained by Cox models using DLS-quantified patterns, versus 0.674 for the 29-pathologist cohort and 0.690 for the specialist reference; adding GP3.5 raised the c-index to 0.704, while adding GP4.5 yielded 0.702.Source: Results · p. 7

• The DLS was more accurate than 29 board-certified pathologists scoring prostatectomy whole-slide images.Source: Discussion · p. 7
• 66% Gleason score concordance and 61% Gleason grade group concordance were achieved by pathologists against genitourinary specialists.The Gleason score concordance was at the high end of previously reported inter-pathologist concordances of 47%-70%.Source: Discussion · p. 8
• The study’s risk-stratification performance followed the trend of prior work, but clinical impact on prognostication and therapy decisions still requires evaluation.DLS-predicted Gleason scores could be computed on demand as decision support in routine workflows.Source: Discussion · p. 8

Implications of DLS region-level pattern classifications and quantitation

• Where pathologists disagreed on pattern categorization, the DLS reflected ambiguity in prediction scores and assigned finer-grained Gleason patterns.This may mitigate variability from discrete categories imposed on continuous glandular differentiation and create avenues for more precise risk stratification.Source: Discussion · pp. 8–9
• The DLS directly quantitated Gleason patterns from region categorizations, avoiding visual-quantitation variability and outperforming pathologists against a specialist-adjudicated reference standard.This suggests an opportunity for more precise prognostication.Source: Discussion · p. 9

Relation to previous works

• The study extended prior deep-learning work to routine-practice-representative Gleason grading specimens and directly compared algorithm performance with pathologists using a rigorous reference standard.Developing the DLS required complex, large-scale annotations, and titrated-data experiments suggested performance benefited greatly from dataset size and may improve with more or better-quality data.Source: Discussion · pp. 9–10
• The DLS used quasi-online hard-negative mining, with more frequent training-set inference improving performance; the approach exceeded 30,000 stage-1 inferences per second.The authors anticipate the approach may also benefit other histopathology algorithms.Source: Discussion · p. 10

Limitations and future work

• Clinical implementation requires addressing digital-review limitations, and future work must validate the DLS for biopsies, other variants and prognostic categorizations, and larger clinically annotated datasets.The study focused on acinar adenocarcinoma in prostatectomy specimens; larger datasets are needed to assess the statistical significance of prognostication trends.Source: Discussion · p. 11

Conclusions

• The DLS outperformed a cohort of 29 generalist pathologists in Gleason scoring prostatectomy whole-slide images.The conclusions describe the system as having potential to enhance the clinical utility of Gleason scoring for treatment decisions.Source: Conclusions · p. 11
• The DLS enabled more accurate quantitation of Gleason patterns and finer-grained characterization of differentiation, creating opportunities for better risk stratification.The conclusions frame these capabilities as potential improvements to Gleason scoring and treatment decisions for patients with prostatic adenocarcinoma.Source: Conclusions · pp. 11–12

Methods

• The study assembled prostatectomy specimens from TCGA, a U.S. teaching hospital, and an independent laboratory, then assigned cases to development or validation datasets.The final validation dataset comprised 331 slides: 183 from TCGA, 144 from the hospital, and 4 from the laboratory.Source: Methods · pp. 13–14
• A genitourinary specialist established each slide’s reference standard using prior general-pathologist reviews and final Gleason-pattern percentages.Slide-level Gleason scores and grade groups were derived from the specialist’s predominant and next-most-common patterns, avoiding variability from inconsistent tertiary replacement.Source: Methods · pp. 14–15
• Twenty-nine pathologists provided independent slide-level reviews, while three pathologists exhaustively annotated tissue regions on a subset of 79 validation slides.The region-level annotations covered more than 95% of tissue on each selected slide; only regions labeled by all three pathologists were used for validation.Source: Methods · p. 15
• Clinical risk-stratification analyses used progression-free interval for TCGA and biochemical recurrence for the hospital cohort; follow-up was available for 320 of 331 validation slides.The paper notes that endpoints were unavailable for the laboratory and a small number of TCGA and hospital cases.Source: Methods · pp. 15–16
• The DLS has two stages: regional classification followed by whole-slide Gleason grade group classification; a nearest-neighbor classifier uses a summary of first-stage heatmap output to categorize each slide’s grade group.Source: Methods · p. 16
• Accuracy was defined as the fraction of exact matches with the reference, making exact agreement with that reference the criterion used to measure accuracy.Source: Methods · p. 17

Code availability.

• TensorFlow and the Python libraries used for computation and plotting, including SciPy, NumPy, Lifelines, and MatPlotLib, are available.Source: Code availability · p. 18

Data availability.

• The TCGA dataset is available through the Genomic Data Commons, but the other datasets are not publicly available because of data-sharing restrictions.The validation-set reference standard was established through pathologist review and specialist adjudication; grade groups 4 and 5 were merged because of their low prevalence.Source: Data availability · pp. 18, 27
• Across 79 slides and 41 million annotated patches, DLS Gleason-pattern likelihoods changed smoothly with pathologists’ classifications, and its learned continuum enabled finer categorization.The figure illustrates regions spanning the Gleason patterns 3–5 spectrum.Source: Figure 4 · p. 23, Data availability · p. 24

Datasets and variant handling

• Training and tuning datasets used 1–7 slides per patient, reviewed by 3–5 pathologists; ungradable slides were excluded, and variant handling followed specified rules.Cancer and histological variants not Gleason gradable were excluded except intraductal carcinoma; other variants were graded consistently with ISUP 2014 recommendations.Source: Table 1 · p. 36, Table 2 · p. 37

Excluded validation slides

• Of 12 excluded validation cases with consensus among the original adjudicator and two consulting GU experts, the DLS was concordant on 9 and within one grouping on 3.Twenty slides were excluded from the main analysis because the specialist adjudicator could not provide a confident diagnosis.Source: Supplementary Information · p. 39

Pathologist comparisons

• The DLS was more accurate than 14 of 19 pathologists in an analysis of overlapping validation-set subsets.Collectively, the 19 pathologists provided three annotations per slide across all 331 validation slides.Source: Table 5 · p. 41

Sensitivity analysis

• Excluding slides flagged by pathologists as non-confident produced results qualitatively similar to Supplementary Table 4, with no material differences.The sensitivity analysis compared DLS and pathologist Gleason scoring.Source: Table 9 · p. 45, Supplementary Information · p. 45

Clinical-event models

• Supplementary Table 10 describes Cox models for progression or biochemical recurrence using Gleason-pattern quantitation from pathologists, GU specialists, or the DLS, including fine-grained pattern features.The models were trained and evaluated on the 331-slide validation set; the supplied passage does not report comparative model results.Source: Table 10 · p. 46, Supplementary Information · p. 46

Supplementary Fig. 1: Confusion Matrices for the DLS and two pathologist subgroups

• The DLS was more accurate for GG1, GG2, and GG4–5, but less accurate for GG3, than both pathologist subgroups on the validation set.The subgroups comprised 10 pathologists who each reviewed every slide and 19 pathologists who collectively provided three reviews per slide.Source: Figure 1 · p. 47
• On patches assigned mixed-grade labels by pathologists, the DLS represented in-between patterns with mixed likelihood across multiple Gleason patterns.The mixed labels were 3+4, 4+3, 4+5, and 5+4, extensions that let pathologists represent patterns characteristic of both grades at the gland level.Source: Supplementary Fig. 1: Confusion… · p. 48
• The DLS learned a continuum of Gleason patterns, with quantitative gradations between the defined GP3, GP4, and GP5 categories.The visualization’s endpoint columns showed 100% confidence in GP3, GP4, and GP5; intervening columns represented patterns between those categories.Source: Supplementary Fig. 1: Confusion… · p. 49
• The annotation tool let users select a label, outline regions on slides, and zoom between objective powers; most annotations were made at 5–20X.Screenshots also show annotations at 2.5X and 10X.Source: Supplementary Fig. 1: Confusion… · p. 50
• Pathologists provided region-level and slide-level training data, while genitourinary expert pathologists were involved in creating slide-level tuning and validation datasets.The passage directs readers to the Methods and Supplementary Methods for further dataset details.Source: Supplementary Fig. 1: Confusion… · p. 51

Pathologist Slide-Level Gleason Scoring Protocol

• Training slides were reviewed by 3–7 pathologists (median 4); labels used the most common annotation, ties favored the more severe grade, and slide-level Gleason scores came from predominant and next-most-common patterns.Source: Pathologist Slide-Level Gleason… · p. 52

Pathologist Region-Level Annotation Protocol

• Region labels distinguished artifacts, uncertain consult regions, and mixed-grade areas; ambiguous training labels were resolved by majority vote and mixed grades mapped to the primary pattern.Perineural and lymphovascular invasive tumor and intraductal carcinoma were labeled non-Gleason-gradable.Source: Pathologist Region-Level Annotation… · pp. 52–53
• Training slides received non-exhaustive region annotations, while three pathologists exhaustively annotated selected tuning-slide regions and only concordant patches were retained.Training annotations covered a median 57% of tissue; tuning annotations covered a median 14%, using eight randomly selected square regions per slide.Source: Pathologist Region-Level Annotation… · p. 53

Development of the Deep Learning System

• The DLS summarized patch predictions as tumor and grade-group proportions, then used k-nearest-neighbor models for four-class and binary grade-group predictions.The selected models used k=24 with uniform neighbor weighting.Source: Development of the Deep Learning System · p. 54
• Continuous hard-negative mining repeatedly analyzed the training patches and increased sampling of patches with higher cross-entropy loss.Inference ran hourly across more than 112 million patches, updating sampling probabilities so incorrectly classified examples were sampled more often.Source: Development of the Deep Learning System · p. 55
• >30,000 DLS stage-1 inferences per second supported the quasi-online hard-negative mining approach used here.Source: Development of the Deep Learning System · p. 55
• The authors anticipate that continuous hard-negative mining may benefit development of other deep learning algorithms on histopathology images.Source: Development of the Deep Learning System · p. 55

Fine-grained Gleason Pattern (GP)

• The quantitative GP interpolates between the two highest-confidence predictions using likelihood1/(likelihood1 + likelihood2), yielding a continuous value across existing GPs 3, 4, and 5.Source: Fine-grained Gleason Pattern (GP) · p. 56
• With GP predictions [0.7, 0.2, 0.1], the interpolation gives 0.78 and a quantitative GP of 3.78.Source: Fine-grained Gleason Pattern (GP) · p. 56
• CIELAB color space visualizes quantitative GPs, while validation-slide patches closest to a desired value, such as 3.5, identify representative regions.Source: Fine-grained Gleason Pattern (GP) · p. 56

Comparison with the Cohort-of-29

• The modified permutation test compared DLS accuracy with the cohort-of-29 mean by swapping each slide’s model rating with a pathologist rating and generating a two-tailed null distribution over 5,000 iterations.The test statistic was the difference between DLS accuracy and mean pathologist accuracy.Source: Comparison with the Cohort-of-29 · p. 56
• For risk stratification, annotations were sampled to approximate equal representation of all 29 pathologists, selecting among the subgroup-of-10 and subgroup-of-19 annotations with specified probabilities.Each subgroup-of-10 annotation had probability 1/29; each of the three available subgroup-of-19 annotations had probability (19/29)*(1/3).Source: Comparison with the Cohort-of-29 · pp. 56–57

Bootstrap Approach for Confidence Intervals

• Confidence intervals for pools of 10, 19, and 29 were computed by resampling slides and annotators with replacement, separately resampling the 10- and 19-pathologist subsets for the pool of 29.Source: Bootstrap Approach for Confidence… · p. 57

DLS Region-level Errors

• Localization errors included imprecise pattern boundaries, tumor predictions across narrow non-tumor strips, and difficulty delineating GP5 from stroma because individual cells were impractical to outline.The underlying region-level annotations were also described as imprecise for this task.Source: DLS Region-level Errors · p. 57
• Ambiguous histology caused errors, such as tangentially cut GP3 resembling GP4, and the DLS could not use context beyond its input image patch.The authors attribute some remaining errors to true prediction mistakes that may improve with more data.Source: DLS Region-level Errors · pp. 57–58
• The second stage was fairly robust to region-level errors by summarizing predictions from all slide regions as a small number of features.The authors note that remaining true prediction mistakes may improve with more data.Source: DLS Region-level Errors · p. 58