Her2 Challenge Contest: A Detailed Assessment of Automated Her2 Scoring Algorithms in Whole Slide Images of Breast Cancer Tissues

Introduction

• Visual assessment of Her2 IHC can show inter- and intra-observer variability, and borderline scores remain challenging despite scoring guidelines.Automated scoring is presented as a potential source of objective quantitative analysis to assist pathologists.Source: Introduction · pp. 4, 6
• The contest provided a platform to assess automated Her2 scoring on IHC slides and compare pathologists’ and algorithms’ agreement.The paper describes this as an initial step toward computer-assisted diagnosis for digitised breast cancer histology slides.Source: Introduction · p. 6

Ethics

• The study received approval from Nottingham Research Ethics Committee 2, reference REC 2020313.The R&D reference was (N) 03HI01.Source: Ethics · p. 7

Image Data Acquisition and Ground Truth

• The dataset comprised 172 whole-slide images from 86 invasive breast carcinoma cases, including H&E and Her2-stained slides.H&E slides help identify invasive tumour and distinguish it from in situ disease; Her2 scoring is performed on IHC slides.Source: Image Data Acquisition and Ground Truth · p. 7
• Ground truth came from clinical reports for cases reported or reviewed by at least two specialist consultant histopathologists.The referral centre also provided regular internal Her2 quality control and participated in UK NEQAS.Source: Image Data Acquisition and Ground Truth · p. 7

Contestants

• 18 submissions from 14 teams were received for the off-site contest; four pathologists submitted markings for the Man vs Machine contest.105 teams from more than 28 countries had registered for the training dataset.Source: Contestants · pp. 8–9

Evaluation

• The algorithms were evaluated by agreement points, weighted confidence, and combined points, with the combined criterion multiplying agreement and weighted-confidence scores per case.Weighted confidence rewards confidence in correct predictions and penalizes confidence in incorrect predictions; it may identify cases requiring expert examination.Source: Evaluation · pp. 9, 11
• The evaluation framework treated errors differently according to clinical implications, including whether a score would lead to FISH testing or Herceptin treatment.A 0/1+ call reported as 3+, or the reverse, was characterized as a serious error.Source: Evaluation · p. 9
• Agreement points penalized erroneous predictions by deviation from ground truth; to resolve ties, bonus points used PCMS (complete membrane-staining cells regardless of intensity), with predicted PCMS within ±2% of ground truth.Source: Evaluation · pp. 9–10

Contest Leaderboards

• Some tied teams were distinguished by PCMS prediction accuracy, including MUCS-1 and VISILAB, which earned more bonus points or had more precise PCMS predictions, respectively.The figure summarizes the top-ranked teams’ agreement and bonus points and notes the importance of correctly predicting the percentage of cell membrane staining.Source: Contest Leaderboards · p. 12, Figure 3 · p. 14

Summary of Proposed Automated Methods

• Most teams used supervised patch-based classification, typically with deep learning; CS_UCCGIP and Huangch instead used handcrafted characteristic curves with standard machine learning.The described pipeline included preprocessing and region identification, classification, and post-processing to aggregate slide-level Her2 scores and estimate PCMS.Source: Summary of Proposed Automated Methods · pp. 12, 14
• CNN approaches dominated, accounting for 8 of the top 10 methods.Several leading teams used architectures inspired by AlexNet and GoogLeNet [9,10].Source: Summary of Proposed Automated Methods · p. 13

Contest Leaderboards

• VISILAB, FSUJena, and Huangch were the top three teams under combined points, while VISILAB, FSUJena, and MTB NLP led weighted confidence.Team Indus, MUCS-1, and MUCS-2 led point-based assessments; the table summarizes all three assessment criteria.Source: Contest Leaderboards · p. 12, Table 3 · p. 13

Summary of Proposed Automated Methods

• Team Indus estimated PCMS using both IHC and H&E slides, whereas the remaining teams used only IHC images.All submissions used images at 10× or higher except MUCS and Rumrocks, which used low-resolution images.Source: Summary of Proposed Automated Methods · p. 15

Organization

• The event compared expert and automated scores with GT agreed by at least two consultant breast pathologists, using a 15-case off-site subset and paired IHC and H&E images.Pathologists scored each case with a Her2 score, PCMS, and confidence value.Source: Organization · pp. 15–16

Man vs Machine Results Comparison

• One automated method slightly outperformed the top-performing participant pathologist in the event’s overall evaluation.Table 4 summarizes each participant’s evaluation score under the contest criteria.Source: Man vs Machine Results Comparison · p. 16, Table 4 · p. 17
• The submitted automated algorithms were not ready for deployment and require extensive validation on significantly larger-scale data, with substantial expert input to prepare GT.Source: Man vs Machine Results Comparison · pp. 16–17
• For score 3+, participants and automated methods showed 100% agreement with GT, while scores of 1+ and 2+ showed disparities.Source: Man vs Machine Results Comparison · p. 17
• 6/9 (67%) score 1+ cases were overcalled as 2+ by humans, compared with 4/9 (44%) by machine algorithms.Source: Man vs Machine Results Comparison · p. 17
• 7/15 (46%) score 2+ cases were overcalled as 3+ by humans, compared with 1/15 (6%) by machines; machines also undercalled 1/15 (6%) as 1+.Source: Man vs Machine Results Comparison · p. 17
• No score 2+ cases were misclassified as 1+ or 0 by participant pathologists, while one automated method predicted 1+ for a 2+ case.The authors note that score 2+ cases are recommended for FISH testing in routine practice.Source: Man vs Machine Results Comparison · pp. 17–18
• Most incorrect participant-pathologist predictions occurred in heterogeneous cases, where subtle morphological differences make complete membrane staining difficult to assess, especially for scores 2+ and 3+.The paper illustrates two such examples in Fig 4.Source: Man vs Machine Results Comparison · p. 19, Figure 4 · p. 20

• The contest provided the first independent comparison of Her2 scoring platforms on a benchmark dataset and a starting point for cross-system learning.Source: Discussion · pp. 21, 24, 25
• Automated analyses performed comparably to histopathologists, and many algorithms achieved high accuracy, often close to the maximum.Source: Discussion · p. 21
• All participants correctly identified GT 3+ cases, while overcalls of score 0 or 1+ never exceeded 2+, avoiding denied treatment or treatment without further testing.Source: Discussion · p. 21
• A fairer human comparison would involve more pathologists and microscope-based slide review; effective routine algorithms also require collaboration and training across tissue types and staining variation.Source: Discussion · pp. 22, 24
• Semi-automated analysis of selected regions could quantify staining to assist PCMS estimation and Her2 scoring, particularly in borderline cases.The task of estimating PCMS is subjective, and Her2 assessment depends on membrane completeness as well as staining intensity.Source: Discussion · pp. 22–23
• Automated methods could support objective diagnosis and predictive modeling, with automated or semi-automated scoring offering potential for routine practice.Source: Discussion · p. 23
• Deployment faces practical and generalization limits: some methods took over three hours per case, and the contest used images from one site and one scanner.The authors proposed testing across laboratories, experts, scanners, and staining conditions to reduce overfitting to one dataset.Source: Discussion · p. 23
• Automated or semi-automated scoring and computer-assisted diagnosis tools are intended to assist pathologists in daily routine analysis.Source: Discussion · p. 25
• The contest is over, but registration and the web portal will remain open for future participants to contribute to automated scoring.Source: Discussion · p. 25

Supplementary Material

• The supplementary material section is titled “Description of Automated Methods.”Source: Supplementary Material · p. 27

Contest Format

• The contest released 52 training cases and 28 off-site test cases, with test-slide ground truth withheld for evaluation.Source: Contest Format · p. 31
• Teams submitted predicted Her2 scores, confidence values, and PCMS estimates for test slides before results were announced at the workshop.Source: Contest Format · pp. 31–32
• Six remaining cases were used in an on-site competition, but those results are not discussed here because some algorithms required substantial computation.Source: Contest Format · p. 32
• The training-set table reports ground-truth scores and complete-membrane-staining percentages for 52 cases; case 63 was deemed negative with an amplification ratio of 1.92.Source: Table 1 · p. 34

Related Work on Automated IHC Scoring

• Automated image analysis has been proposed to address inter- and intra-observer variation in conventional assessment.Source: Related Work on Automated IHC Scoring · p. 35
• Thresholding methods that perform well on one dataset may fare less well on unseen data as stain hues vary.Source: Related Work on Automated IHC Scoring · p. 35
• Deep-learning approaches outperformed classical machine-learning approaches in the contest.Source: Related Work on Automated IHC Scoring · p. 36
• The contest offered a shared dataset on which participants could develop and validate their algorithms, unlike prior work using different datasets.Source: Related Work on Automated IHC Scoring · p. 36

Team Indus

• A trained network scores each WSI patch, and the patch scores are aggregated into one Her2 score; a stated criterion is n3/N > 0.08, with n3 denoting patches scored 3+.Source: Team Indus · pp. 37–38
• Team Indus estimates PCMS as membrane-staining extent relative to tumor regions identified from H&E images.The method detects tumor nuclei after stain normalization and morphological filtering, then estimates membrane extent from a chicken-wire staining pattern in IHC images.Source: Team Indus · p. 38
• The MUCS submissions used AlexNet or GoogLeNet, with MUCS-2 adding augmentation and hand-picked regions from test images without slide classifications.Source: Team Indus · p. 39
• Team Indus trained on hand-selected representative regions from 52 IHC training images, mapped from low resolution to 40× and divided into patches.Source: Team Indus · p. 39
• For testing, common H&E and IHC regions were mapped to high resolution, and adaptive thresholding filtered patches before neural-network classification.Source: Team Indus · pp. 39–40
• The method estimates PCMS from the proportion of score-2+ and score-3+ patches among non-background patches.Source: Team Indus · p. 40

• A CNN trained on 128 x 128 WSI patches predicts Her2 scores, while a post-processing Random Forest aggregates patch metrics into slide-level scores and percentages.Source: MTB NLP · pp. 40–41
• The method manually annotated 272 IHC regions, labeled them by the corresponding WSI ground-truth score, and used AlexNet-like architecture for the submitted model.Source: MTB NLP · p. 40
• Training used 207 annotated regions with the remainder for validation; patches were flipped and rotated, and dropout was used to prevent overfitting.The model was trained on 8,575,000 patches using cross-entropy loss.Source: MTB NLP · p. 41
• At test time, non-overlapping patches were classified and connected-component analysis merged them into clusters for slide-level pixel-percentage metrics.Source: MTB NLP · p. 41
• Five-fold cross-validation on the 52 training images produced averaged test-slide score and percentage estimates.Source: MTB NLP · p. 41

VISILAB

• Training used 5,750 hand-selected 68 x 68-pixel patches across four Her2 classes and a background class, split 75% for training and 25% for validation.The patches were extracted from ground-truth WSIs, averaging 1,150 per class.Source: VISILAB · pp. 41–42
• GoogLeNet classifies grid-sampled WSI patches, returning a class prediction and confidence; slide-level scoring uses the percentage of patches in each class.The model was selected from several CNNs and trained using stochastic gradient descent with a 0.01 base learning rate over 50 epochs.Source: VISILAB · p. 42
• A WSI score is selected by scanning patch classes from 3+ to 1+ for the first class representing at least 10% of patches, with background excluded.A separate expert rule relates class proportions to the percentage of complete cell-membrane staining.Source: VISILAB · p. 42

UCCSSE

• The curves distinguish scores: 1+ curves start above 10%, while 3+ curves lie above 30%, indicating intense, uniform membrane staining.Figure 1 shows the characteristics curves and their corresponding predicted Her2 scores.Source: UCCSSE · p. 43, Figure 1 · p. 43

RumRocks

• RumRocks selects tissue patches with a DCNN and CNN1, then uses CNN2 to predict Her2 score and PCMS from multiple patches from the same WSI.CNN2 reshapes the patch tensor and processes it through a 1D convolution rather than combining individual patch predictions by averaging or aggregating metrics.Source: RumRocks · p. 44
• The RumRocks pipeline selects patches containing at least 50% area of interest at 0.3125×, then translates their coordinates to 10× for further processing.CNN1 accepts or rejects the subsampled patches.Source: RumRocks · p. 44
• CNN models were trained with mean squared error loss and Adam stochastic gradient descent, starting at learning rate 10^-3; predictions from each network were averaged to form an ensemble-based score.The learning rate was reduced every 15,000 iterations by a factor of 1.5, and each network was trained for 200,000–300,000 iterations.Source: RumRocks · p. 45

FSUJena

• FSUJena computes bilinear features by summing spatial dyadic products of AlexNet activation vectors into a Gramian matrix, then uses multiclass logistic regression for four scoring classes.The method uses a pretrained ImageNet AlexNet; random crop predictions are averaged for the WSI score, while PCMS is estimated from the training-set mean tumour-cell percentage for that class.Source: FSUJena · p. 45
• In preliminary results, FSUJena’s bilinear-feature approach outperformed conventional CNN activations.The authors also note that a pretrained network was beneficial for avoiding overfitting.Source: FSUJena · pp. 45–46

Huang’s Method (Huangch)

• Huang’s method extracts handcrafted features from IHC slides and uses multiclass AdaBoosted decision trees to predict Her2 score and PCMS from patch feature vectors.The feature set includes Differential Box-Counting, multi-wavelet, histogram-statistic, and grey-level co-occurrence methods.Source: Huang’s Method (Huangch) · p. 46
• The SAMME model assigns weights to decision trees, selecting the best tree at each iteration from a generated pool before assembling trees for testing.The method samples patches at local maximal points representing the strongest Her2-stained over-expression.Source: Huang’s Method (Huangch) · pp. 46–47