PaperlayerView exact revision on arXiv

Source-linked AI summary

A Deep Learning Algorithm for One-step Contour Aware Nuclei Segmentation of Histopathological Images

Yuxin Cui, Guiying Zhang, Zhonghao Liu, Zheng Xiong, Jianjun Hu

arXiv:1803.02786v1cs.CV

TL;DR

Nuclei segmentation in histopathological images must recover individual contours despite variation in color, shape, and texture. The paper proposes a fully convolutional nucleus-boundary model with overlapping-patch processing and simple post-processing; it reports better benchmark performance than prior methods.

  • Problem

    Nuclei segmentation must identify each nucleus’s contour despite variation in color, shape, and texture across histopathological images.

    Source: §I · p. 1

  • Method

    A fully convolutional nucleus-boundary model predicts nuclei and boundaries together, using overlapping patches for large images and simple post-processing for final segmentation.

    Source: §I · p. 2

  • Results

    The method outperforms CNN3 on MOD in F1 score and Dice’s Coefficient, and outperforms a breast-cancer nuclei-segmentation method in precision, recall, and F1 score.

    Source: B. Experiment result · pp. 7–8

  • Takeaways & Limitations

    Individual contours provide nucleus-level features and cluster-distribution information that can be used for breast-cancer grading and classification.

    Source: §I · p. 1

Abstract

from arXiv · show

This paper addresses the task of nuclei segmentation in high-resolution histopathological images. We propose an auto- matic end-to-end deep neural network algorithm for segmenta- tion of individual nuclei. A nucleus-boundary model is introduced to predict nuclei and their boundaries simultaneously using a fully convolutional neural network. Given a color normalized image, the model directly outputs an estimated nuclei map and a boundary map. A simple, fast and parameter-free post-processing procedure is performed on the estimated nuclei map to produce the final segmented nuclei. An overlapped patch extraction and assembling method is also designed for seamless prediction of nuclei in large whole-slide images. We also show the effectiveness of data augmentation methods for nuclei segmentation task. Our experiments showed our method outperforms prior state-of-the- art methods. Moreover, it is efficient that one 1000X1000 image can be segmented in less than 5 seconds. This makes it possible to precisely segment the whole-slide image in acceptable time

I. INTRODUCTION

Nuclei segmentation must recover each nucleus’s contour despite substantial variation in appearance, enabling individual-nucleus analysis. The paper proposes simultaneous nucleus-and-boundary prediction and several supporting contributions for segmentation and evaluation.

I. INTRODUCTION

  • Individual-nucleus contours, unlike center-point detections, permit extraction of each nucleus for analysis of features and cluster distributions used in breast-cancer grading and classification.The paper identifies color, shape, and texture variation as a challenge across organs and cancer grades.Source: §I · p. 1
  • Traditional thresholding [5], clustering [6, 7], and feature-filtering methods [8, 9] are described as image-specific and sensitive to manually set parameters.The paper also contrasts its approach with binary classifiers that distinguish nuclei from background [10, 11, 12].Source: §I · pp. 1–2
  • The nuclei-boundary model predicts nuclei and their contours simultaneously, with boundary detection intended to improve detection accuracy and split touching or overlapping nuclei.The paper contrasts this with prior methods that require complex post-processing to obtain final boundaries.Source: §I · p. 2
  • For extra-large images beyond U-net’s GPU-memory capacity, the proposed seamless patch-wise approach uses a weighted loss map for training and voting to assemble patches.The paper notes that patch borders lack context for accurate prediction.Source: §I · p. 2
  • The paper reports extensive studies of data-augmentation methods for nuclei segmentation.Source: §I · p. 2
  • Four proposed evaluation criteria are missing-detection, false-detection, under-segmentation, and over-segmentation rates.They are intended to give pathologists a more in-depth view of segmentation performance and help select a method for a specific application.Source: §I · p. 2

II. METHOD

The method takes color-normalized images through an end-to-end network and post-processing pipeline. Its preprocessing uses stain normalization, and experiments favor raw RGB input over an H-channel grayscale input.

A. Overview

  • The overview pipeline feeds color-normalized images to an end-to-end network that predicts nuclei and boundaries, then applies fast, parameterless post-processing to obtain nucleus areas.Figure 2 is identified as an overview of nuclei segmentation on histopathological images.Source: A. Overview · pp. 2–3, Figure 2 · p. 3

B. Data Preprocessing

  • The authors normalize H&E images to a target color space using sparse non-negative matrix factorization; experiments found no considerable difference between two tested normalization methods.Their reported experiments use the method in with λ = 0.1.Source: B. Data Preprocessing · p. 3
  • Experiments found the network extracted nuclei better from normalized RGB images than from H-channel grayscale images, so the authors use RGB input.They suggest the H-channel may omit information useful for distinguishing nuclei from cytoplasm.Source: B. Data Preprocessing · p. 3

C. Nucleus-boundary model

The fully convolutional nucleus-boundary model combines nuclei extraction with boundary prediction, producing pixelwise background, boundary, and inside probabilities. Its encoder–decoder architecture builds contextual features and combines them into segmentation maps.

C. Nucleus-boundary model

  • The fully convolutional network predicts all image pixels in one pass, rather than processing one sliding-window patch per pixel.The paper contrasts this with processing one million sliding windows for a 1000×1000 image.Source: C. Nucleus-boundary model · p. 3
  • The model merges nuclei extraction and boundary prediction, outputting probabilities for background, boundary, and inside classes.Boundary pixels lie on or inside an annotated boundary and within 2 pixels of it; inside pixels exclude boundary pixels.Source: C. Nucleus-boundary model · p. 3

1) The architecture of our NB network:

  • Encoding layers extract contextual feature maps, and decoding layers combine them to generate segmentation maps.The experimental input layer is 128×128 because of GPU memory limits; the authors note that larger inputs may improve performance.Source: 1) The architecture of our NB network · p. 3
  • The network uses SELU activations in every convolutional layer and omits normalization layers because SELU is designed for self-normalizing behavior.Convolution filters are 3×3 with same padding and dropout rate 0.2; training uses Adam.Source: 1) The architecture of our NB network · p. 4

2) Data Augmentation:

The authors augment randomly extracted image patches and their targets to address the limited training data available for nuclei segmentation. They apply geometric and elastic transformations to both inputs and targets, then binarize transformed labels.

2) Data Augmentation:

  • The authors motivate augmentation by the small datasets and costly specialist labeling, which make sufficient accurate boundary annotations difficult to obtain.Their datasets often contain only tens of images, and labeling a 1000×1000 image with hundreds of nuclei usually takes at least five hours.Source: 2) Data Augmentation · p. 4
  • Random elastic, rescaling, affine, shift, flip, and rotation transformations are applied to training patches and their corresponding targets.Rescaling factors range from 0.5 to 1.5; elastic transformation uses σ=12 and α randomly set between 100 and 200.Source: 2) Data Augmentation · p. 4
  • After bilinear interpolation, transformed targets are binarized using thresholds and label-priority rules for boundary, inside, and background pixels.Boundary values above 0.5 are set to 1; inside pixels are retained only when positive and not boundary pixels.Source: 2) Data Augmentation · p. 4

3) Weighted loss:

The authors address U-net border-context and cropping issues with a weighted loss and overlapped patch extraction and assembly. Training minimizes categorical softmax cross-entropy, with a proposed weight map assigning position-dependent weights.

3) Weighted loss:

  • Fig. 4 pairs a normalized image patch and its ground truth with an augmented sample derived from that patch and its corresponding ground truth.Source: Figure 4 · p. 5
  • A weighted loss and patch extraction-and-assembly scheme let the network predict equal-size segmentation maps while addressing missing context at patch borders.U-net outputs smaller maps because border pixels lack full context, and its fixed border size may not fit images with different nucleus sizes.Source: 3) Weighted loss · pp. 4–5
  • Training minimizes categorical softmax cross-entropy between predictions and targets, using the proposed weight map.The loss uses each pixel’s softmax probability for its true class.Source: 3) Weighted loss · p. 5
  • The weight map uses pixel position and distances from the border and center, with map height and width also specified.Source: 3) Weighted loss · p. 5

4) Extra-large Image Segmentation Using Overlapped Patch Extraction and Assembling:

Whole-slide images are too large to feed into a U-net at once, and patch-based predictions are inaccurate near patch borders. The proposed overlapped extraction and assembling method uses sliding-window patches and voting to produce each image pixel’s final prediction.

4) Extra-large Image Segmentation Using Overlapped Patch Extraction and Assembling:

  • Overlapped patches address inaccurate predictions near patch borders, a key problem in patch-based U-net segmentation of extra-large images.The paper identifies limited GPU memory as the reason whole-slide images must be processed patch by patch.Source: 4) Extra-large Image Segmentation… · p. 5
  • Patches are extracted with a sliding window and a stride, then assembled using a vote mechanism.The final prediction P(i, j) is for the image pixel at (i, j); k(i, j) denotes that pixel’s position in the kth patch.Source: 4) Extra-large Image Segmentation… · p. 5
  • The method is designed to predict each image pixel from overlapping patch predictions during assembly.The supplied passage describes the vote mechanism but does not specify its weights or outcome.Source: 4) Extra-large Image Segmentation… · p. 5

5) Post-processing:

The post-processing converts the model’s inside-class map into connected nucleus regions and restores their shapes by dilation. The evaluation criteria distinguish detection errors from errors in splitting overlapped nuclei, which precision, recall, and F1 alone may not differentiate.

5) Post-processing:

  • Thresholding the inside-class map at 0.5 makes each connected component one nucleus, and dilation recovers its shape.The procedure is parameter-free and described as taking a negligibly short time.Source: 5) Post-processing · p. 5

A. Evaluation criteria

  • Precision, recall, and F1 may fail to distinguish methods with similar scores but different detection and splitting errors.The paper distinguishes missed detections from under-segmentation, and false detections from over-segmentation.Source: A. Evaluation criteria · p. 6
  • MDR, FDR, USR, and OSR separately evaluate missing detection, false detection, under-segmentation, and over-segmentation.The combinations MDR/FDR and USR/OSR assess nuclei–cytoplasm discrimination and handling of overlapped nuclei areas, respectively.Source: A. Evaluation criteria · p. 6
  • These four criteria can help pathologists select automatic segmentation methods for specific tasks.Recall is negatively correlated with MD and USR, while precision is negatively correlated with FDR and OSR.Source: A. Evaluation criteria · p. 6

1) Datasets:

The evaluation uses three manually annotated nuclei-segmentation datasets: MOD, BCD, and BNS. Their image sources, sizes, and training/test arrangements differ.

1) Datasets:

  • MOD contains 30 1000X1000 images from seven organs, with about 21,000 manually annotated nuclear boundaries.Sixteen images form the training set and 14 the test set; the test set has two images from each organ.Source: 1) Datasets · p. 6
  • BCD has 21 subsetA images and 18 subsetB images, with five cropped 1000*1000 subsetA images manually annotated for training and subsetB used for testing.The annotations were made under specialist supervision.Source: 1) Datasets · p. 6
  • BNS comprises 33 512X512 H&E images from seven triple-negative breast cancer patients, with 2,754 manually annotated nuclei.Source: 1) Datasets · p. 6

B. Experiment result

On MOD, the method outperforms CNN3 on F1 score and Dice’s Coefficient, with qualitative examples showing its staged segmentation. On BCD, it outperforms a state-of-the-art breast-cancer nuclei method in precision, recall, and F1 score.

B. Experiment result

  • Figure 6 shows color-normalized images, raw predictions with visible nuclei areas and boundaries, and final segmentation with each nucleus separately colored.The authors describe the color variety as well controlled after normalization.Source: B. Experiment result · p. 6, Figure 6 · p. 7
  • Five random training/test splits were used for MOD, with 16 images selected for training and 14 for testing.The training set yielded 12,000 patches; the paper notes the dataset’s original train/test assignments were unavailable.Source: B. Experiment result · pp. 6–7
  • The method outperforms CNN3 on MOD in both F1 score and Dice’s Coefficient.Table I presents the MOD comparison; the experiments also include 10-fold cross-validation.Source: B. Experiment result · p. 7, Table I · p. 8
  • On the same hardware and test images, the method predicts a 1000 * 1000 image in about 5 seconds, compared with CNN3 at about 4 minutes; its post-processing takes less than 0.1 seconds.Source: B. Experiment result · p. 7
  • On two images with similar precision and recall, the proposed error criteria distinguish different segmentation-error profiles.The first image’s errors are mainly under-segmentation and false detections; the second’s include over-segmentation, missing detections, and false detections.Source: B. Experiment result · p. 7
  • On BCD, the method outperforms the state-of-the-art breast cancer nuclei segmentation method in precision, recall, and F1 score.Evaluation used leave-one-patient-out cross-validation.Source: B. Experiment result · p. 8

C. Discussion

The discussion examines data augmentation for fully convolutional nuclei segmentation and illustrates how its proposed criteria distinguish error profiles in images with similar precision and recall.

C. Discussion

  • The section frames image transformations as a way to augment limited training samples for segmentation.Common transformations include rotation, flipping, shifting, rescaling, and elastic deformation.Source: 1) Data augmentation for fully… · p. 8
  • On cropped image (a) in Fig. 8, reported values were precision 0.76 and recall 0.83, alongside OSR 0.05, USR 0.15, MDR 0.02, and FDR 0.20.Source: Figure 8 · p. 9
  • For cropped image (b) in Fig. 8, precision was 0.78 and recall was 0.83; OSR was 0.13, USR 0.05, MDR 0.12, and FDR 0.10.Source: Figure 8 · p. 9
  • Without augmentation, validation loss rises rapidly from epoch 5, indicating severe overfitting in the reported experiment.The study compared rotation, flipping, shifting, rescaling, elastic deformation, and their combination.Source: 1) Data augmentation for fully… · p. 9
  • Rotating alone gave the best performance improvement, but only the combination of all tested transforms prevented overfitting and achieved good performance.The combined augmentation includes elastic deformation, flipping, rotation, shifting, and rescaling.Source: 1) Data augmentation for fully… · p. 9

2) Nuclei Segmentation on Extra-large Images:

The paper reports that overlapping-patch assembly alone leaves seams, while a weight map is also needed for seamless results. It further argues that joint inside, boundary, and background prediction captures class relationships, and concludes that this supervised FCN approach may extend to other segmentation tasks.

2) Nuclei Segmentation on Extra-large Images:

  • Using overlapping-patch extraction and assembly without a weight map still leaves noticeable seams; without both techniques, seams and inaccurate border predictions are evident.The comparison holds stride at 64 for the overlapped-patch condition.Source: 2) Nuclei Segmentation on Extra-large… · p. 9

3) NB model versus the mixed nucleus model + boundary model:

  • The NB model learns relationships among inside, boundary, and background classes, including no gaps between inside and boundary and no crossing of the inside class over the boundary.The paper reports more precise inside- and boundary-class predictions than with separately trained binary classifiers.Source: 3) NB model versus the mixed nucleus… · p. 9

IV. CONCLUSION

  • The conclusion describes a supervised FCN pipeline with normalized images, overlapping patches, joint nucleus-boundary detection, seamless reassembly, and fast parameter-free post-processing.The authors state that the model was trained on few images and tested on images with potentially different appearances, and suggest applying the simultaneous identification idea to cell and bacteria segmentation.Source: §IV · p. 9

Related papers

Loading 1803.02786v1…