Source-linked AI summary
PanNuke Dataset Extension, Insights and Baselines
Jevgenij Gamper, Navid Alemi Koohbanani, Ksenija Benes, Simon Graham, Mostafa Jahanifar, Syed Ali Khurram, Ayesha Azam, Katherine Hewitt, Nasir Rajpoot
TL;DR
Nuclei-level computational pathology models need datasets and context robust enough for challenging clinical images. This work extends PanNuke, evaluates segmentation and classification models, and finds strong segmentation generalization to unseen brain tissue while nuclear categorization remains difficult.
Problem
Nuclear classification from small image patches can omit the surrounding tissue context pathologists use, while nuclei vary substantially across tissue types.
Method
The paper develops PanNuke using semi-automated, pathologist-quality-controlled annotations across diverse tissues, accelerates mask generation with NuClick, and benchmarks nuclei models.
Results
Models generally perform best with HoVer-Net across tissue types, and PanNuke-trained segmentation generalizes to unseen brain tissue, although nuclear category prediction is challenging.
Takeaways & Limitations
PanNuke provides a broader benchmark and supports evaluating nuclei segmentation beyond tissues represented during training.
Takeaways & Limitations
Whole-slide context remains important because patch-based algorithms lack the contextual information available when reviewing the original WSI.
Abstract
from arXiv · showhide
The emerging area of computational pathology (CPath) is ripe ground for the application of deep learning (DL) methods to healthcare due to the sheer volume of raw pixel data in whole-slide images (WSIs) of cancerous tissue slides. However, it is imperative for the DL algorithms relying on nuclei-level details to be able to cope with data from `the clinical wild', which tends to be quite challenging. We study, and extend recently released PanNuke dataset consisting of ~200,000 nuclei categorized into 5 clinically important classes for the challenging tasks of segmenting and classifying nuclei in WSIs. Previous pan-cancer datasets consisted of only up to 9 different tissues and up to 21,000 unlabeled nuclei and just over 24,000 labeled nuclei with segmentation masks. PanNuke consists of 19 different tissue types that have been semi-automatically annotated and quality controlled by clinical pathologists, leading to a dataset with statistics similar to the clinical wild and with minimal selection bias. We study the performance of segmentation and classification models when applied to the proposed dataset and demonstrate the application of models trained on PanNuke to whole-slide images. We provide comprehensive statistics about the dataset and outline recommendations and research directions to address the limitations of existing DL tools when applied to real-world CPath applications.
I. INTRODUCTION
PanNuke addresses the mismatch between current computational-pathology datasets and the artifacts and diversity of clinical images. It contributes a large, clinically quality-controlled dataset and evaluates models for segmentation, classification, and broader tissue generalization.
- Motivation: Clinical-pathology models can overfit surface statistical regularities, while existing nucleus datasets often fail to represent artifacts found in real clinical images.Detection models trained on narrower datasets produced false detections in burnt tissue, blur, and processing or scanning artifacts.
- Contributions: PanNuke is presented as the largest and most diverse nucleus segmentation and classification dataset, with semi-automated annotation and clinical quality control.The dataset is intended to better support clinically relevant model development than limited-diversity challenge datasets.
- Contributions: NuClick accelerates clinical verification by generating segmentation masks from a single verified nucleus point rather than requiring pathologists to draw complete masks.This procedure is incorporated during segmentation-mask generation and quality control.
- Generalization and impact: The dataset includes a schema intended for use beyond its sampled tissues, and models trained on it are reported to generalize to tissues such as brain.The work also encourages broader development of deep-learning models for clinically relevant research.
A. Related Work
Prior computational-pathology studies demonstrated uses for nuclear features but were often small, tissue-specific, or limited by coarse annotations. PanNuke provides pixel-level boundaries for individual nuclei to support more granular analysis and semi-automatic labeling.
- Prior studies: Nuclear features have been used for cancer scoring, biomarker discovery, recurrence prediction, and treatment-effectiveness prediction, but studies were often small and tissue-specific.Commonly studied tissues included lung, breast, prostate, and colon because data for other tissues were limited.
- Annotation gaps: Large computational-pathology studies have been limited by a lack of granular annotations, including work based only on patch-level labels.Patch annotations do not provide the individual-nucleus boundaries supplied by PanNuke.
- PanNuke contribution: PanNuke provides pixel-level boundary annotations for every individual nucleus, enabling models to support semi-automatic labeling of tumor or tissue phenotypes.The paper frames individual nuclei as building blocks of organ tissue.
II. THE PANNUKE DATASET
The paper describes its methodology, mask-quality analysis, dataset statistics, and evaluation of existing nucleus-segmentation and classification approaches.
- Scope: The paper presents methodology, analyzes the quality of automatically generated nucleus-segmentation masks, and reports qualitative and quantitative dataset analysis.It also discusses the dataset’s algorithmic and practical significance.
- Scope: Existing approaches to nucleus segmentation and classification are evaluated as part of the study.The evaluation is introduced alongside the dataset analysis and methodology.
- Scope: The methodology is organized to connect dataset construction, mask quality, statistics, and model-performance evaluation.These components are described as the paper’s successive methodological and analytical focus.
A. Dataset Generation
PanNuke was built through a semi-automatic, pathologist-verified pipeline combining existing datasets, iterative model-assisted labeling, and NuClick mask generation. The process produced 189,744 verified nuclei from diverse clinical sources while retaining real-world artifacts.
- Initialization: Four public datasets and 106 visual fields initialized semi-automatic nucleus detection and classification.The source data covered multiple tissues and cancer types before relabeling nuclei into PanNuke categories.
- Clinical sampling: 2,000 visual fields were randomly sampled from more than 20,000 WSIs spanning 19 tissues to reduce selection bias.The sampling included TCGA and local-hospital data, with common clinical artifacts retained.
- Iterative annotation: Seven iterative rounds of pathologist verification and model retraining produced 481 visual fields containing 189,744 exhaustively annotated nuclei.Each round used newly collected annotations to improve predictions for the next stage.
- Mask generation: NuClick generated final segmentation masks from verified nucleus points, reducing pathologist effort compared with drawing complete masks.Point conditioning helped avoid pigment false positives and improved handling of elongated or indistinct-boundary nuclei.
- Output: The resulting semi-automatic pipeline generated and verified 189,744 nuclei from more than 20,000 WSIs.Examples of the generated ground truth are shown in Figure 6.
B. Dataset Description and Statistics
PanNuke uses a clinically shared nucleus-label schema across 19 tissues and captures substantial variation in tissue and class distributions. Its annotations also address contextual and clinical complexities that can undermine nuclei classification based on limited image patches.
- Classification schema: PanNuke applies a clinically sound nucleus-label schema consistently across all 19 tissues.The schema is intended to support appropriate nucleus subtyping in future studies and aligns with categories from prior tissue-specific work.
- Classification schema: Neoplastic labels include malignant and benign tumors, including carcinomas, sarcomas, melanomas, and lymphomas.All tumorous cells are labeled as neoplastic in PanNuke.
- Classification schema: Non-neoplastic labels include epithelial, connective or soft-tissue, inflammatory, and dead cells, with atypia assigned to non-neoplastic.Dead cells may originate from either neoplastic or non-neoplastic cells but are treated as non-neoplastic here.
- Dataset statistics: Total nuclei counts and class distributions vary across tissue types.Figure 7 compares these distributions, with parenthesized numbers representing nuclei totals by category or tissue.
- Clinical context: Pathologists use lower-resolution WSI context, whereas common CNN approaches classify nuclei from isolated or small image patches.The paper links this mismatch to hidden stratification and questions whether average accuracy adequately reflects clinical applicability.
- Clinical context: Some neoplastic versus non-neoplastic cases are difficult to classify simply, and such cases are more frequent in clinical practice than in AI literature portrayals.The paper identifies these cases as a limitation of simplified classification framing.
III. PERFORMANCE BENCHMARKS
PanNuke standardizes benchmark data preparation by providing pre-extracted patches and randomized train, validation, and test folds. Tissue-wise splitting balances representation of the smallest class within each tissue.
- Benchmark construction: PanNuke pre-extracts patches and uses three randomized training, validation, and testing folds for fair model comparison.Earlier datasets provided visual fields, leaving patch extraction inconsistent across methods.
- Benchmark construction: Each fold divides every tissue into three sections while ensuring equal representation of its smallest class.The benchmark applies recent and well-known nucleus segmentation models to PanNuke.
A. Baseline Models
The study adapts or evaluates four leading models for joint nuclear segmentation and classification on PanNuke, alongside a detection-based U-Net. HoVer-Net natively performs both tasks, while the other segmentation models require classification adaptations.
- Application: HoVer-Net predictions were overlaid on brain tissue visual fields to demonstrate application beyond the baseline tissue settings.
- Baseline architectures: Four models—DIST, Mask-RCNN, Micro-Net, and HoVer-Net—were evaluated for nuclear segmentation and classification on PanNuke.DIST received an added semantic pixel-wise classification branch; HoVer-Net already performs both tasks inherently.
- Baseline architectures: DIST uses instance distance maps with an added semantic classification branch, whereas HoVer-Net separates clustered nuclei using horizontal and vertical distance maps.
- Baseline architectures: Mask-RCNN is a state-of-the-art natural-image instance segmentation network, while Micro-Net was proposed for nuclear and gland segmentation.
- Detection baseline: A detection-based U-Net was trained using class-specific detection maps represented as 2D Gaussians centered on true nuclear centroids.
B. Evaluation
Evaluation uses panoptic quality for segmentation and F1, precision, and recall for detection, with results broken down across tissues and nuclear categories. HoVer-Net performs best across most tissue types, while small and imbalanced classes remain difficult.
- Instance segmentation: Multi-class PQ and binary PQ quantify instance segmentation, with mPQ evaluating positive classes independently and bPQ treating all nuclei as one class.
- Instance segmentation: HoVer-Net achieves the best performance for most tissue types and the highest average mPQ and bPQ across tissues.
- Nuclear categories: Dead cells obtain low PQ for all models because their small size makes satisfying the IoU>0.5 true-positive criterion difficult.
- Nuclear categories: Class imbalance may contribute to poor performance on dead-cell and non-neoplastic classes, while distinguishing neoplastic from non-neoplastic nuclei can also challenge pathologists.
- Evaluation design: The evaluation reports mPQ and bPQ for all 19 tissue types and average PQ for each nuclear category across three dataset splits.
- Detection: Segmentation models generally outperform the detection model on F1, precision, and recall, possibly because segmentation incorporates boundary information.
C. Generalisation to other tissues
The best PanNuke-trained model was applied to brain tissue from an unseen source and tissue type. It segmented nuclei favorably but had difficulty assigning the correct nuclear categories.
- Unseen tissue evaluation: On four brain images from an unseen German source, the PanNuke-trained model achieved DICE 0.796, mPQ 0.28, and bPQ 0.51 for nuclear segmentation.
- Qualitative findings: The model successfully segmented all nuclei in brain tissue but found correct nuclear-category prediction challenging.
- Generalisation: The brain-tissue application tests the authors’ speculation that PanNuke-trained models could generalize to other tissues.
IV. CONCLUDING REMARKS
The paper presents a semi-annotated, quality-controlled, multi-tissue dataset with detailed nuclear boundaries and five class labels. It frames this contribution as an important but limited step toward safe and robust computational pathology.
- Dataset contribution: PanNuke provides detailed boundaries and class labels for five main nuclear types across multiple cancerous tissue types.
- Dataset contribution: The dataset is semi-annotated and quality-controlled, addressing concerns that challenge-dataset results may have limited validity because of restricted dataset diversity.
- Scope and outlook: Despite advances in modeling and dataset size, the work is described as only a small step toward safe and robust computer-vision applications in computational pathology.
- Scope and outlook: The paper discusses real-world task complexities and offers label-treatment guidance and schemas for advancing nuclei-classification research.
APPENDIX
PanNuke’s appendix details its five-class labeling schema and illustrates why classifying nuclei requires tissue context beyond local appearance. It also shows substantial variation in nuclear size across cell categories and tissue types.
- Labelling schema: PanNuke labels nuclei as neoplastic, non-neoplastic epithelial, connective tissue, inflammatory, or dead cells.These categories can also be grouped into neoplastic and non-neoplastic cell types.
- Labelling schema: Dysplastic colon epithelial cells are labeled non-neoplastic epithelial because dysplasia is described as pre-neoplastic rather than tumorous.Neoplastic labels specifically correspond to benign and malignant tumor cells.
- Contextual ambiguity: Classifying visual fields may require returning to the original whole-slide image because local patches can resemble either artifact or neoplasm.An ambiguous region was identified as pheochromocytoma only after viewing the WSI, illustrating the contextual limitation of patch-based algorithms.
- Nuclear-size variation: Nuclear size does not directly determine class: connective-tissue nuclei are larger, while neoplastic cells are larger on average than other categories.The appendix presents nuclei-size distributions by class and tissue type, but notes that shape and size alone remain insufficient for classification.
- Nuclear-size variation: Epithelial nuclei vary substantially in size across tissues, emphasizing the need for labeled datasets spanning different tissue types.Both within-tissue category variability and between-tissue distribution differences complicate classification.