Source-linked AI summary
A Petri Dish for Histopathology Image Analysis
Jerry Wei, Arief Suriawinata, Bing Ren, Xiaoying Liu, Mikhail Lisovsky, Louis Vaickus, Charles Brown, Michael Baker, Naofumi Tomita, Lorenzo Torresani, Jason Wei, Saeed Hassanpour
TL;DR
Histopathology image analysis faces barriers from limited data, costly annotation, and high-resolution, variable-size images. This paper introduces MHIST, a compact, expert-labeled dataset for testing models; pretrained ResNet-18 outperforms random initialization on the dataset.
Problem
Limited data, costly annotation, and high-resolution, variable-size images make it difficult to quickly iterate on histopathology image-analysis models.
Source: §1 · p. 2Method
Results
With the full training set, ImageNet-pretrained ResNet-18 outperforms random-initialized ResNet-18 by 8.2%, measured by mean AUC.
Source: §4.3 · p. 5Takeaways & Limitations
MHIST is intended to let researchers test models on a smaller scale before implementing them in large-scale applications.
Source: §6 · p. 6MHIST uses fixed-size images rather than whole-slide images and does not include patient demographic information or polyp size and location.
Source: §6 · p. 6
Abstract
from arXiv · showhide
With the rise of deep learning, there has been increased interest in using neural networks for histopathology image analysis, a field that investigates the properties of biopsy or resected specimens traditionally manually examined under a microscope by pathologists. However, challenges such as limited data, costly annotation, and processing high-resolution and variable-size images make it difficult to quickly iterate over model designs. Throughout scientific history, many significant research directions have leveraged small-scale experimental setups as petri dishes to efficiently evaluate exploratory ideas. In this paper, we introduce a minimalist histopathology image analysis dataset (MHIST), an analogous petri dish for histopathology image analysis. MHIST is a binary classification dataset of 3,152 fixed-size images of colorectal polyps, each with a gold-standard label determined by the majority vote of seven board-certified gastrointestinal pathologists and annotator agreement level. MHIST occupies less than 400 MB of disk space, and a ResNet-18 baseline can be trained to convergence on MHIST in just 6 minutes using 3.5 GB of memory on a NVIDIA RTX 3090. As example use cases, we use MHIST to study natural questions such as how dataset size, network depth, transfer learning, and high-disagreement examples affect model performance. By introducing MHIST, we hope to not only help facilitate the work of current histopathology imaging researchers, but also make the field more-accessible to the general community. Our dataset is available at https://bmirds.github.io/MHIST.
1. Introduction
Histopathology research lacks an accessible, easy-to-iterate benchmark amid barriers to collecting, processing, and annotating data. MHIST is introduced as a small-scale experimental dataset for exploring histopathology image classification.
1. Introduction
- Small experimental setups and accessible benchmarks such as MNIST have helped researchers evaluate ideas before translating them to large-scale applications.The introduction presents this iterative approach as the rationale for a histopathology “petri dish.”Source: §1 · pp. 1–2
- Histopathology lacks an accessible classic benchmark because slide scanning, complex processing and annotation, and dataset-release constraints impede dataset creation.The paper links this inaccessibility to histopathology image analysis remaining on the fringes of computer vision research.Source: §1 · p. 2
- MHIST offers a minimalist binary classification dataset of fixed-size colorectal polyp images, with 3,152 images labeled by seven board-certified gastrointestinal pathologists.The gold-standard label is determined by majority vote, and the dataset is publicly available following a simple use agreement.Source: §1 · p. 2
2. Background
Histopathology images and their annotations create practical obstacles for applying mainstream computer vision methods, while limited datasets complicate model comparisons. MHIST uses fixed-size tiles and expert labels as a simpler setting for quickly testing ideas.
2. Background
- Histopathology images are high-resolution and variable-sized, making storage, preprocessing, and fixed-dimension model inputs difficult to manage.The paper notes that CAMELYON17 slides can range from 2 GB to 18 GB, and slide dimensions and aspect ratios vary.Source: §2 · p. 2
- Annotation is costly and complex: it requires pathologists, may reflect individual annotator tendencies, and lacks clear guidance for labeling whole-slide images.Bounding-box annotation also requires training pathologists to use software and increases annotation time and effort.Source: §2 · p. 2
- Limited training data and test sets can make model comparisons difficult and leave models prone to overfitting.The passage attributes this data limitation to slide discard and exclusion of rare classes.Source: §2 · pp. 2–3
- MHIST uses 224 × 224 pixel colorectal-polyp tiles and majority-vote labels from seven board-certified gastrointestinal pathologists for direct use with standard computer-vision models.The dataset is intended as a quickly learned, easy-to-iterate setting for testing inductive biases before large-scale applications.Source: §2 · p. 3
3. MHIST Dataset
MHIST is a tractable dataset for a clinically relevant colorectal-polyp classification task, with fixed-size images and expert majority-vote labels. Its annotation process also records per-image agreement, reflecting variation among pathologists.
3. MHIST Dataset
- MHIST combines a challenging binary task, 2,175 training and 977 testing images, fixed-size inputs, and seven-pathologist majority-vote labels with agreement levels.The dataset is designed to be tractable for standard models while allowing more-specific tuning using annotator agreement.Source: §3 · p. 3
3.1. Colorectal Polyp Classification Task
- The task distinguishes hyperplastic polyps, typically benign, from sessile serrated adenomas, which are precancerous and require sooner follow-up if untreated.The paper describes this as a clinically important task with considerable inter-pathologist variability.Source: §3.1 · p. 3
3.2. Data Collection
- Researchers extracted 3,152 diagnostic 224 × 224 pixel tiles from 328 colorectal-polyp whole-slide images scanned at 40x and compressed to 8x magnification.The images were shuffled, anonymized, checked for quality, and released with institutional review board approval.Source: §3.2 · p. 3
3.3. Data Annotation
- Seven pathologists independently classified each dataset image as HP or SSA using 2019 WHO criteria; the gold-standard label was assigned by majority vote of the seven labels.Average pairwise annotator agreement was 72.9%; approximately 16.7% of images had 4/7 agreement. Per-image agreement levels (4/7, 5/7, 6/7, or 7/7) are provided, and Figure 2 shows their distribution.Source: §3.3 · p. 4, Figure 2 · p. 4
4. Example Use Cases
The section tests how training-set size interacts with network depth, ImageNet pretraining, and annotator agreement in MHIST. Its experiments find a substantial benefit from pretraining, while deeper models do not improve performance and excluding the hardest images can retain comparable performance.
4. Example Use Cases
- The use cases examine network depth, ImageNet pretraining, and training-image selection by annotator agreement, alongside variation in training-set size.The agreement experiment compares very easy images (7/7), easy (6/7), hard (5/7), and very hard (4/7) images across selected combinations.Source: §4 · p. 4, §4.4 · p. 5
4.1. Experimental Setup
- Experiments vary training-set size across n = 100, n = 200, n = 400, and Full, with n denoting images per class and the test set held fixed.Models are trained for 100 epochs; evaluation reports the mean and standard deviation of the five highest test AUCs across 10 random seeds.Source: §4.1 · p. 4
4.2. Network Depth
- Adding layers does not significantly improve performance, and ResNet-152 decreases mean AUC by 5.1% versus ResNet-18 with 100 images per class.The authors suggest that deeper networks may overfit on this relatively small dataset, consistent with findings that deeper networks perform better only with large sample sizes.Source: §4.2 · pp. 4–5, Table 2 · p. 4
4.3. Transfer Learning
- 8.2%: on the full training set, ImageNet-pretrained ResNet-18 outperforms the random-weight baseline in mean AUC.With 100 images per class, the pretrained model performs comparably to the baseline trained on the full dataset.Source: §4.3 · p. 5, Table 3 · p. 5
4.4. High-Disagreement Training Examples
- Excluding hard and very hard images yields performance comparable to training with all images; excluding only very hard images slightly outperforms using all images.Very hard images have 4/7 annotator agreement; using only very easy or only easy images does not match or exceed all-image performance.Source: §4.4 · p. 5, Table 4 · p. 5
5. Related Work
MHIST is positioned as a small-scale resource for rapid model development, with fixed-size images and multi-pathologist annotations distinguishing it from established histopathology datasets. The discussion also places it within broader interest in lower-compute model development.
5. Related Work
- MHIST fits a broader interest in developing models at smaller scale, alongside proposals for surrogate architecture evaluation and MNIST-1D.The paper says MHIST is heavily inspired by MNIST-1D.Source: §5 · pp. 5–6
- CAMELYON17 contains 1,000 labeled whole-slide images, each over 1 GB, creating a substantial barrier to initial model training and preprocessing.Source: §5 · p. 6
- Compared with PCam’s 327,680 96 × 96 images and roughly 7 GB footprint, MHIST uses seven expert labels per image and is an order of magnitude smaller.PCam labels were drawn by a student and checked by a pathologist; MHIST uses the majority vote of seven expert pathologists.Source: §5 · p. 6
- MHIST is smaller than other histopathology datasets and uses seven pathologists, while many datasets use only one or two annotators.The comparison describes MHIST as a faster-to-obtain petri dish with annotations less influenced by any single annotator.Source: §5 · p. 6, Table 5 · p. 6
6. Discussion
MHIST provides a preprocessed binary-classification dataset of fixed-size images, with gold-standard labels and annotator agreement. Its fixed-size image design and image-only data constrain what it represents.
6. Discussion
- MHIST provides 3,152 preprocessed fixed-size images for a straightforward but challenging binary classification task, with gold-standard labels and annotator agreement.Source: §6 · p. 6
- Fixed-size images may be less precise than whole-slide images, which contain more information; whole-slide images can exceed 10 GB and are beyond current computer-vision model training capacity.Source: §6 · p. 6
- MHIST is intended for testing models at smaller scale before their implementation in large-scale applications and for facilitating further histopathology deep-learning research.Source: §6 · p. 6