Source-linked AI summary
Multi-Layer Pseudo-Supervision for Histopathology Tissue Semantic Segmentation using Patch-level Classification Labels
Chu Han, Jiatai Lin, Jinhai Mai, Yi Wang, Qingling Zhang, Bingchao Zhao, Xin Chen, Xipeng Pan, Zhenwei Shi, Xiaowei Xu, Su Yao, Lixu Yan, Huan Lin, Zeyan Xu, Xiaomei Huang, Guoqiang Han, Changhong Liang, Zaiyi Liu
TL;DR
Dense pixel-level annotation of gigapixel histopathology images is costly and time-consuming. The paper uses patch-level classification labels in a two-phase CAM-based and multi-layer pseudo-supervised pipeline, achieving comparable fully supervised performance with an approximately 2% MIoU and FwIoU gap while reducing labeling time.
Problem
Dense pixel-level histopathology annotations are expensive and time-consuming, motivating tissue segmentation from more economical patch-level labels.
Method
A CAM-based classification phase generates pseudo masks, and a segmentation phase uses Multi-Layer Pseudo-Supervision with a classification gate.
Results
The model achieves comparable fully supervised performance with only around a 2% gap for MIoU and FwIoU, while outperforming weakly supervised approaches on two datasets.
Takeaways & Limitations
Patch-level presence or absence labels can support tissue semantic segmentation while reducing annotation time compared with dense manual labeling.
Takeaways & Limitations
Performance is weaker for non-predominant tissue categories because insufficient training samples hinder precise segmentation.
Abstract
from arXiv · showhide
Tissue-level semantic segmentation is a vital step in computational pathology. Fully-supervised models have already achieved outstanding performance with dense pixel-level annotations. However, drawing such labels on the giga-pixel whole slide images is extremely expensive and time-consuming. In this paper, we use only patch-level classification labels to achieve tissue semantic segmentation on histopathology images, finally reducing the annotation efforts. We proposed a two-step model including a classification and a segmentation phases. In the classification phase, we proposed a CAM-based model to generate pseudo masks by patch-level labels. In the segmentation phase, we achieved tissue semantic segmentation by our proposed Multi-Layer Pseudo-Supervision. Several technical novelties have been proposed to reduce the information gap between pixel-level and patch-level annotations. As a part of this paper, we introduced a new weakly-supervised semantic segmentation (WSSS) dataset for lung adenocarcinoma (LUAD-HistoSeg). We conducted several experiments to evaluate our proposed model on two datasets. Our proposed model outperforms two state-of-the-art WSSS approaches. Note that we can achieve comparable quantitative and qualitative results with the fully-supervised model, with only around a 2\% gap for MIoU and FwIoU. By comparing with manual labeling, our model can greatly save the annotation time from hours to minutes. The source code is available at: \url{https://github.com/ChuHan89/WSSS-Tissue}.
I. INTRODUCTION
The paper targets tissue-level histopathology segmentation with patch-level labels instead of costly pixel-level annotations. It proposes pseudo-supervision mechanisms and introduces LUAD-HistoSeg, reporting strong performance against weakly and fully supervised baselines.
- Dense pixel-level histopathology annotations are expensive, labor-intensive, and require clinical expertise because tissue morphology is diverse and heterogeneous.
- The model uses patch-level classification labels to generate pixel-level pseudo masks and then trains a semantic segmentation model.
- Progressive Dropout Attention reduces discriminative-region shrinkage, while Multi-Layer Pseudo-Supervision narrows the information gap between patch- and pixel-level labels.
- A classification gate mechanism reduces false-positive segmentation for non-predominant tissue categories with fewer training samples.
- LUAD-HistoSeg is introduced as a weakly supervised lung adenocarcinoma dataset with four tissue types, 16,678 patch-labeled samples, and 607 pixel-labeled patches.
II. RELATED WORKS
Related work frames histopathology segmentation as an annotation-burden problem addressed through active, semi-supervised, patch-level, and weakly supervised learning. CAM-based methods provide localization cues but struggle to recover complete tissue regions, especially in heterogeneous histopathology images.
- A. Histopathology Image Segmentation: Histopathology segmentation supports computational pathology but traditionally requires expensive, time-consuming pixel-level labeling by pathologists.
- Active Learning: Active learning reduces annotation burden by refining uncertain pseudo-labels or selecting informative samples for annotation.
- Patch-level Classification: Patch-level classification reformulates dense segmentation and has supported diagnostic and survival prediction in prior studies.
- Semi-supervised Learning: Semi-supervised learning leverages limited labeled data with unlabeled samples through consistency or cross-sample relationships.
4) Weakly-supervised Learning:
The paper uses patch-level labels to generate pseudo masks for tissue segmentation, addressing the limited spatial information available from weak annotations. Its two-phase design combines CAM-based classification with segmentation training on generated masks.
- 4) Weakly-supervised Learning:: Weak supervision can use image-level, point, bounding-box, or scribble annotations instead of dense pixel labels.
- 4) Weakly-supervised Learning:: Prior weakly supervised histopathology studies used multiple-instance learning, scribbles, or point annotations to generate pseudo-labels for segmentation.
- 4) Weakly-supervised Learning:: This paper aims to achieve tissue-level semantic segmentation using only patch-level labels.
- 4) Weakly-supervised Learning:: The first phase trains a patch-level multi-label classifier with Progressive Dropout Attention to generate pixel-level pseudo masks.
- 4) Weakly-supervised Learning:: Figure 2 presents PDA-based pseudo-mask generation followed by DeepLab V3+ segmentation guided by MLPS and a classification gate.
- 4) Weakly-supervised Learning:: The second phase trains a semantic segmentation model on the generated pseudo masks, producing final segmentation results.
1) Pseudo Mask Generation:
The classification phase extracts feature maps, applies progressive dropout attention, predicts tissue classes, and generates pixel-level pseudo masks with Grad-CAM for segmentation.
- Given an input patch x, the model extracts deep feature maps from the last layer before classification.
- Progressive Dropout Attention enriches feature representation by preventing excessive focus on the most discriminative region.
- After attention, global average pooling and a fully connected layer calculate each tissue class probability.
- A trained multi-label classifier generates pixel-level pseudo masks with Gradient-weighted Class Activation Mapping for the segmentation model.
2) Progressive Dropout Attention:
Progressive Dropout Attention combats activation-region shrinkage by progressively deactivating highlighted regions, encouraging broader feature learning and more precise pseudo masks.
- Classification models increasingly focus on discriminative regions, causing activated-region shrinkage that harms segmentation, especially in heterogeneous cancer images.
- CAMs are generated for each category from weighted feature maps, then the most highlighted areas are deactivated using a relative cutoff.
- The dropout attention map averages the deactivated CAMs across categories.
- PDA progressively enlarges deactivated areas so the classifier learns from non-predominant regions rather than relying only on the easiest evidence.
- The dropout coefficient decreases with training epochs until a lower bound; training starts with three undisrupted epochs before dropout begins.In practice, σ = 0.985, l = 0.65, and the initial coefficient is 1.
- PDA alleviates discriminative-region shrinkage, produces wider feature representations, and generates more precise pseudo masks.
B. Pseudo-supervised Tissue Semantic Segmentation
The segmentation phase trains a semantic segmentation model with pseudo masks, using multi-layer supervision and a classification gate to reduce the information gap from patch-level labels.
- The segmentation model fseg maps an input patch x to a semantic segmentation result s under supervision from pseudo masks p.
- Multi-layer pseudo-supervision and a classification gate mechanism are introduced to improve semantic segmentation performance.
- Because patch-level labels provide incomplete spatial information, pseudo masks from three CNN layers enrich the segmentation model’s supervision.
- The multi-layer pseudo masks are upsampled to the original image resolution, with loss weights λ1 = 0.2, λ2 = 0.2, and λ3 = 0.6.
2) Classification Gate Mechanism:
The classification gate addresses long-tail errors by suppressing segmentation channels when classification predicts that a tissue category is unlikely, after which patch predictions are stitched into whole-slide results.
- Non-predominant tissues such as necrosis and lymphocytes are more prone to poor pseudo masks and false positives because of long-tail imbalance.
- The gate uses classification confidence because classification is trained on ground-truth labels, whereas segmentation is trained on pseudo masks.
- For each tissue category, the gate zeros its segmentation probability map when the classification probability falls below threshold ϵ.
- The final patch segmentation uses argmax over output probabilities, with ϵ = 0.1 in practice.
- Whole-slide predictions are formed by cropping patches with over 50% overlap, averaging overlapping category probabilities, and applying argmax.
[TE, NEC, LYM, TAS]
LUAD-HistoSeg is a weakly supervised lung adenocarcinoma dataset designed to train with patch-level labels and evaluate semantic segmentation with pixel-level masks. It covers four tissue categories and separates training, validation, and test annotations.
- Annotation format: The training set uses one-hot encoding labels, while validation and test sets use semantic segmentation masks.This separates weak supervision during training from pixel-level evaluation.
- Dataset purpose and categories: LUAD-HistoSeg targets pixel-level semantic segmentation from patch-level annotations for four tissue categories: TE, TAS, NEC, and LYM.The dataset is intended for weakly supervised tissue segmentation in lung adenocarcinoma.
- Data construction: The dataset was built from lung adenocarcinoma whole-slide images by randomly cropping 800 patches per slide at 10× magnification with 224×224 size.Slides came from 29 patients at Guangdong Provincial People’s Hospital and 20 patients from TCGA.
- Annotation format: Patch labels indicate whether each tissue category is present or absent, whereas mask annotations are manually drawn and refined for validation and testing.Junior clinicians performed labeling, and experienced pathologists confirmed it.
B. Breast Cancer Semantic Segmentation (BCSS) Dataset
BCSS supplies pixel-level annotations that were converted into a weakly supervised patch dataset for comparison with existing methods and fully supervised training. Experiments assess quantitative performance, qualitative boundaries, and annotation-effort reduction.
- Dataset construction: BCSS contains 151 breast-cancer ROIs with pixel-level annotations for five tissue categories: TUM, STR, LYM, NEC, and OTR.The ROIs were selected from H&E-stained whole-slide images.
- Dataset construction: The authors generated 31,826 patches from BCSS and split them into patch-labeled training, pixel-labeled validation, and pixel-labeled test sets.The split contains 23,422 training, 3,418 validation, and 4,986 test patches.
- Evaluation: The study evaluates the proposed model against HistoSegNet and SC-CAM using IoU, MIoU, FwIoU, and pixel-level accuracy.HistoSegNet targets histopathology images, whereas SC-CAM was developed for natural images.
- Quantitative comparison: The final model greatly outperforms both existing CAM-based WSSS methods on the two evaluated datasets.Phase 1 pseudo masks already outperform the comparison methods in both datasets on LUAD-HistoSeg.
- Comparison with full supervision: The pseudo-supervised model’s performance gap from fully supervised training is less than 2%, with visually indistinguishable segmentation results reported on BCSS.Both models use DeepLab V3+ with the same training epochs.
B. Ablation Studies
Ablation studies isolate the effects of progressive dropout attention, multi-layer pseudo-supervision, and classification gating. The experiments combine quantitative comparisons with qualitative results on LUAD-HistoSeg and BCSS.
- Ablation design: The ablations compare Phase 1 alone, constant dropout attention, progressive dropout attention, multiple pseudo-mask configurations, and the final model.The study also presents representative qualitative comparisons in Fig. 9.
- Ablation design: The ablation results evaluate whether PDA, MLPS, and the classification gate improve semantic segmentation performance.These components address pseudo-mask quality and category-specific false positives.
- Qualitative analysis: Qualitative ablations use examples from LUAD-HistoSeg and BCSS to compare representative baseline configurations with the final model.The figure directly overlays segmentation results onto the original images.
1) Progressive Dropout Attention:
Progressive dropout attention improves segmentation, particularly for non-predominant categories, while multi-layer pseudo-supervision supplies richer pseudo-mask information. The method remains visually strong on whole-slide images but has lower concordance for rare tissues and ambiguous boundaries.
- Progressive Dropout Attention: PDA continuously improves performance over constant dropout attention, especially for the non-predominant NEC and LYM categories.Progressively increasing training difficulty is reported to smooth the learning process.
- Progressive Dropout Attention: PDA produces lymphocyte regions with higher concordance to ground truth than the model without PDA, although pseudo masks remain imperfect.The comparison is illustrated in Fig. 9.
- Progressive Dropout Attention: PDA reduces classification accuracy by around 1% on LUAD-HistoSeg and less than 0.1% on BCSS.The authors describe this as a trade-off for more than 2% semantic segmentation improvement.
- Multi-Layer Pseudo Supervision: Multi-layer pseudo-supervision outperforms single-layer pseudo-mask training because it provides more information from different classification layers.The pbn7 single-layer model performs best among single-layer variants, but the three-layer model performs better overall.
- Whole-slide results: Whole-slide results show high concordance for predominant tissues and lower, though still visually pleasing, concordance for NEC and LYM.The reported whole-slide examples include lung adenocarcinoma and breast cancer.
- Limitations: Unsmooth boundaries and small isolated regions remain, while tissue borders can be morphologically ambiguous and clinically difficult to define precisely.The paper questions whether smoother boundaries are always meaningful for clinical cancer research.
D. How Can We Reduce Annotation Efforts?
The model reduces annotation demands by replacing pixel-level labels with patch-level annotations while retaining competitive segmentation performance. Its limitations are most apparent for non-predominant tissues and locally ambiguous regions.
- Insufficient training samples remain the greatest barrier to precise segmentation of non-predominant tissue categories.The authors suggest collecting more samples for these categories may alleviate the problem.
- The model can recognize isolated small stroma regions inside lymphocytic infiltrates because it considers morphology only within a local receptive field.The authors identify a global-local design as a possible solution because clinical tissue definition also considers the surrounding microenvironment.
- Patch-level annotations replace pixel-level labels, reducing pathologists’ annotation efforts while supporting tissue-level semantic segmentation.The proposed model uses patch-level presence or absence labels rather than carefully drawn tissue boundaries.
- The model achieves competitive performance with fully-supervised models despite using patch-level annotations.