Source-linked AI summary
Constrained Deep Weak Supervision for Histopathology Image Segmentation
Zhipeng Jia, Xingyi Huang, Eric I-Chao Chang, Yan Xu
TL;DR
The paper targets cancer-region segmentation when only coarse image-level pathology labels are available. It develops DWS-MIL and CDWS-MIL, combining FCNs, deep weak supervision, and area constraints within MIL. The methods achieve state-of-the-art results on large-scale histopathology datasets, including higher F-measures than MIL-Boosting on Dataset B.
Problem
Weakly supervised segmentation seeks pixel-level cancer-region predictions from coarse image-level labels while reducing costly and time-consuming region annotation.
Method
DWS-MIL uses an end-to-end FCN-based MIL framework with intermediate-layer weak supervision and area constraints derived from rough cancer-region size estimates.
Results
The methods attain state-of-the-art results on large-scale histopathology images, with CDWS-MIL reaching F-measures of 0.622 for cancer images and 0.997 for non-cancer images on Dataset B versus 0.449 and 0.993 for MIL-Boosting.
Takeaways & Limitations
The framework provides an efficient image-to-image weak-supervision approach that can use limited additional annotation and is applicable beyond histopathology to MRI, CT, and ultrasound.
Abstract
from arXiv · showhide
In this paper, we develop a new weakly-supervised learning algorithm to learn to segment cancerous regions in histopathology images. Our work is under a multiple instance learning framework (MIL) with a new formulation, deep weak supervision (DWS); we also propose an effective way to introduce constraints to our neural networks to assist the learning process. The contributions of our algorithm are threefold: (1) We build an end-to-end learning system that segments cancerous regions with fully convolutional networks (FCN) in which image-to-image weakly-supervised learning is performed. (2) We develop a deep week supervision formulation to exploit multi-scale learning under weak supervision within fully convolutional networks. (3) Constraints about positive instances are introduced in our approach to effectively explore additional weakly-supervised information that is easy to obtain and enjoys a significant boost to the learning process. The proposed algorithm, abbreviated as DWS-MIL, is easy to implement and can be trained efficiently. Our system demonstrates state-of-the-art results on large-scale histopathology image datasets and can be applied to various applications in medical imaging beyond histopathology images such as MRI, CT, and ultrasound images.
I. INTRODUCTION
The paper addresses cancer-region segmentation from coarse image-level labels by combining multiple instance learning with fully convolutional, image-to-image prediction. It introduces deep weak supervision and area constraints to improve multi-scale learning and segmentation accuracy with little extra annotation effort.
- Weakly supervised learning uses image-level cancer labels to automatically explore pixel-level cancer classification, reducing cancer-region annotation work.
- In MIL, each histopathology image is a bag and each pixel is an instance, while only the bag-level cancer label is provided.Pixel features are computed from each pixel’s surrounding context.
- Existing MIL methods often rely on pre-specified features, motivating an integrated CNN-based framework that learns hierarchical features automatically.
- The paper introduces DWS-MIL, an end-to-end FCN system that performs image-to-image weakly supervised learning and guides intermediate layers with weak supervision for nested multi-scale feature learning.
- Area constraints use pathologists’ rough estimates of cancerous-region size, such as 30%, as additional weak supervision requiring little extra labeling effort.
- The proposed contributions include deep weak supervision, area constraints, and super-pixels, with area constraints reported to significantly enhance segmentation accuracy.
II. RELATED WORK
Related work spans directly related MIL methods, weakly supervised computer vision, and deep supervision with FCNs. The paper positions its approach as a simpler constrained MIL formulation tailored to two-class histopathology segmentation.
- Related work is organized into directly related methods, weakly supervised computer vision, and weakly supervised medical imaging.
- Directly related work: MCIL jointly performs image-level classification, patch-level segmentation, and patch-level clustering but remains space-demanding, time-consuming, and dependent on pre-specified features.
- Directly related work: Earlier FCN-MIL work introduced fully convolutional networks for MIL, while later work optimized linear constraints on the output space.
- Directly related work: The paper’s area constraints are simpler to implement through back-propagation than generalized constrained optimization and are evaluated for two-class classification.
- Directly related work: Deep weak supervision is inspired by HED’s combination of deep supervision and FCNs but applies it to weakly supervised MIL rather than fully supervised edge detection.
B. Weakly supervised learning in computer vision
Computer-vision weak supervision includes patch-based MIL segmentation and detection methods, while medical-imaging approaches use weak labels for local anatomical or identity-related learning. The paper identifies a lack of end-to-end image-to-image MIL learning and presents its framework in response.
- Weakly supervised computer-vision methods include segmentation and detection approaches.
- Segmentation: Patch-based segmentation methods extract pre-specified features from selected patches, creating substantial space and computation requirements.
- Object detection: Object-detection MIL treats varying-size image patches as sliding-window instances, making instance storage enormous and motivating proposal-based selection.
- Medical-imaging weak supervision has been used to discover discriminative local anatomies and to train convolutional neural networks for identification tasks.
- Existing medical-imaging methods lack an end-to-end strategy for image-to-image learning and prediction under MIL, motivating the paper’s FCN-based framework.
A. Our Baseline
The baseline uses an end-to-end FCN within MIL to learn pixel-level cancer segmentation from image-level labels, replacing hard instance aggregation with a differentiable Generalized Mean.
- The baseline performs image-to-image learning and prediction under MIL, using image-level labels to learn pixel-level segmentation.
- Each image is a bag of pixel instances, with Yi=0 denoting a non-cancer image and Yi=1 a cancerous image.
- Hard maximum aggregation is numerically unstable and ignores all nonmaximum instances during learning.
- Generalized Mean provides a softmax-style aggregation whose output approaches maxk bYik as r→∞.
- The FCN produces instance-level cancer probabilities visualized as a heatmap with values ranging from 0 to 1.
B. Constrained Deep Weak Supervision.
Constrained deep weak supervision trains multiple side-output layers under MIL and adds area constraints to limit expansion of predicted positive regions, then fuses multi-scale outputs.
- Deep weak supervision: DWS-MIL applies weak supervision at multiple side-output layers to guide learning across scales.
- Deep weak supervision with constraints: The constrained formulation adds an area constraint because baseline positive instances tend to progressively outgrow true cancerous regions.
- Area constraints: Experts provide rough relative cancer-region sizes ai as additional weak supervision during annotation.
- Area constraints: The area constraint is an L2 loss combined with the MIL loss using a manually specified hyper-parameter ηt.
- Fusion model: A weighted fusion layer combines side-output probability maps to leverage multi-scale predictions.
- Optimization: The overall objective is minimized by stochastic gradient descent through back-propagation.
C. Super-pixels
The super-pixel alternative groups image pixels into small, consistent regions, reducing the number of instances compared with pixel-based MIL.
- Super-pixels define instances as small regions generated by SLIC oversegmentation rather than individual pixels.
- This representation produces fewer, more consistent instances that can be pre-computed.
IV. NETWORK ARCHITECTURE
The network uses the first three stages of a pre-trained 16-layer VGGNet, attaches side outputs at multiple scales, and fuses them into the final prediction.
- The framework uses a 16-layer VGGNet pre-trained on ImageNet and retains only its first three stages.
- Each retained stage connects its last convolutional layer to a one-channel sigmoid side-output layer.
- Different side outputs have distinct strides and receptive fields, producing predictions at different scales before fusion.
- Side outputs are upsampled to input-image size by bilinear interpolation before weighted fusion.
- The fourth and fifth stages are trimmed because their large receptive fields make outputs coarse and nearly image-wide on local tissue textures.
V. EXPERIMENTS
The experiments evaluate the proposed framework on two histopathology image datasets, establishing the empirical setting for the methods.
- Two histopathology image datasets are used to evaluate the proposed methods.
- The evaluation covers the implementation and experimental behavior of the proposed weakly supervised segmentation framework.
- The datasets provide the basis for comparing the proposed methods under histopathology image segmentation tasks.
A. Implementation
The experiments implement and evaluate the FCN-based MIL framework, examining data quantity, area constraints, deep weak supervision, super-pixels, and comparisons with prior methods.
- Implementation: The framework extends Caffe with layers for generalized-mean pixel predictions and area-constraint loss computation.
- Less training data: With less training data, segmentation performance improves as the training set grows, while limited negative images encourage oversized predictions.
- Area constraints: Area constraints restrict overly large positive segmentations and improve accuracy for both cancer and non-cancer images.
- Deep weak supervision: Deep weak supervision yields improving accuracy from lower to higher side-output layers, while final fusion achieves the best result.
- Super-pixels: Super-pixels help DWS-MIL recognize detailed tissue edges and produce more accurate boundaries.
- Advantages of CDWS-MIL: CDWS-MIL provides end-to-end multi-scale segmentation and integrates area constraints more directly than patch-based MIL-Boosting.
C. Experiment B
Dataset B evaluates CDWS-MIL against MIL-Boosting using five-fold cross-validation. CDWS-MIL achieves higher average F-measures on both cancer and non-cancer images.
- Dataset B contains 30 cancer and 30 non-cancer colon histopathology images with rough cancer-region area estimates.
- The comparison uses five-fold cross-validation and the same evaluation metric as Dataset A.
- CDWS-MIL achieves average F-measures of 0.622 on cancer images and 0.997 on non-cancer images, versus 0.449 and 0.993 for MIL-Boosting.
VI. CONCLUSION
The paper presents an end-to-end deep weak supervision framework for histopathology image-to-image segmentation, combining multi-scale learning with area constraints. Experiments report state-of-the-art results on challenging large-scale histopathology images, with broader applicability across medical imaging and computer vision.
- The framework performs image-to-image histopathology segmentation under deep weak supervision.
- Deep weak supervision is developed to learn multi-scale information.
- Area constraints provide additional weakly supervised information during learning.
- Experiments report state-of-the-art results on large-scale challenging histopathology images.
- The methods are presented as broadly applicable to medical imaging and computer vision applications.