PaperlayerView exact revision on arXiv

Source-linked AI summary

NuClick: A Deep Learning Framework for Interactive Segmentation of Microscopy Images

Navid Alemi Koohbanani, Mostafa Jahanifar, Neda Zamani Tajadin, Nasir Rajpoot

arXiv:2005.14511v2cs.CVstat.AP

TL;DR

NuClick addresses the costly, expertise-intensive process of collecting dense microscopy annotations. It uses CNN-based interactive segmentation with clicks for nuclei and cells and squiggles for glands, achieving strong results across datasets and supporting released annotation datasets.

  • Problem

    Dense microscopy segmentation labels are expensive and time-consuming to collect, often requiring expert knowledge for computational pathology.

  • Method

    NuClick feeds clicks or squiggles together with RGB image data into a CNN-based segmentation framework, using one click for nuclei and cells and a squiggle for glands.

  • Results

    NuClick obtained state-of-the-art performance across six datasets, including AJI 0.834, Dice 0.912, and PQ 0.838 for nuclear segmentation on MoNuSeg.

  • Takeaways & Limitations

    The framework provides reliable, low-interaction annotations across object scales and domains, and the authors release lymphocyte and white blood cell annotation datasets.

Abstract

from arXiv · show

Object segmentation is an important step in the workflow of computational pathology. Deep learning based models generally require large amount of labeled data for precise and reliable prediction. However, collecting labeled data is expensive because it often requires expert knowledge, particularly in medical imaging domain where labels are the result of a time-consuming analysis made by one or more human experts. As nuclei, cells and glands are fundamental objects for downstream analysis in computational pathology/cytology, in this paper we propose a simple CNN-based approach to speed up collecting annotations for these objects which requires minimum interaction from the annotator. We show that for nuclei and cells in histology and cytology images, one click inside each object is enough for NuClick to yield a precise annotation. For multicellular structures such as glands, we propose a novel approach to provide the NuClick with a squiggle as a guiding signal, enabling it to segment the glandular boundaries. These supervisory signals are fed to the network as auxiliary inputs along with RGB channels. With detailed experiments, we show that NuClick is adaptable to the object scale, robust against variations in the user input, adaptable to new domains, and delivers reliable annotations. An instance segmentation model trained on masks generated by NuClick achieved the first rank in LYON19 challenge. As exemplar outputs of our framework, we are releasing two datasets: 1) a dataset of lymphocyte annotations within IHC images, and 2) a dataset of segmented WBCs in blood smear images.

I. INTRODUCTION

NuClick addresses the high cost of dense microscopy annotation by using minimal user guidance to support annotation collection across nuclei, cells, and glands. The framework targets computational pathology settings where labeled data and expert segmentation are expensive to obtain.

  • Dense segmentation labels are time-consuming, labor-intensive, and may require expert knowledge, while weakly and semi-supervised methods remain behind fully supervised approaches.
  • Annotating 100 nuclei at 10 seconds each takes about 17 minutes, motivating interactive methods that minimize user effort.
  • NuClick proposes a unified CNN-based framework for interactive annotation of nuclei, cells, and glands with minimal user interaction.
  • The framework uses points or squiggles as guiding signals for interactive segmentation, with nuclei and cells shown as point-guided examples and glands as squiggle-guided examples.

B. Interactive segmentation

Prior interactive segmentation methods commonly require multiple points, boxes, or substantial image-level interaction. NuClick instead uses one click for nuclei and cells and a squiggle for glands, while extending across object types and scales.

  • Four extreme points or bounding boxes remain cumbersome for images containing hundreds of nuclei and insufficient for glands with complex shapes and unclear boundaries.
  • Unlike full-image approaches, NuClick avoids requiring background labels and is designed to reduce interference from missing instances and high per-object interaction.
  • NuClick uses one point for nuclei and cells and a squiggle for glands, providing minimal interaction across objects with different complexity.
  • The framework combines guiding signals and multi-scale blocks for precise segmentation across microscopic object scales.
  • The authors report contributions including a weighted hybrid loss, experiments on effectiveness and generalizability, and two released annotation datasets.

III. METHODOLOGY

NuClick is a CNN-based interactive segmentation framework in which user clicks or squiggles guide object delineation. It supports interactions ranging from individual objects to full-image segmentation while using only one click or a simple squiggle.

  • NuClick requires one click inside nuclei or cells and a simple squiggle inside glands as its user input.
  • The supervised CNN framework uses an encoder-decoder architecture and feeds image patches with guiding signals during training and testing.

B. Model architecture & loss

NuClick uses an encoder-decoder design with multi-scale and residual blocks to handle microscopic objects of different sizes. Its loss combines soft Dice loss with weighted cross-entropy to address imbalance and suppress neighboring-object interference.

  • The encoder-decoder architecture uses multi-scale convolutional blocks to capture features needed for both small and large objects.
  • Residual blocks allow a deeper network, which was increased relative to the previous version to better handle complex objects such as glands.
  • NuClick combines soft Dice loss and weighted cross-entropy during training.
  • The Dice component controls class imbalance, while weighted cross-entropy penalizes predictions containing undesired objects.
  • The adaptive weighting scheme emphasizes included objects while avoiding false segmentation of touching undesired objects.

1) Guiding signal for nuclei/cells:

NuClick uses auxiliary guiding signals to segment nuclei, cells, and glands with minimal user interaction. Point-based signals guide nuclei and cells, while gland training varies skeleton-based signals derived from distance-transformed masks.

  • Guiding signal for nuclei/cells:: For nuclei and cells, NuClick concatenates RGB channels with inclusion and exclusion maps to form a five-channel network input.The inclusion map marks clicked target positions, while the exclusion map marks other clicked objects; with one click, the exclusion map is zero.
  • Guiding signal for nuclei/cells:: During training, randomly sampled interior points and centroid-based exclusion maps expose the network to continuously varying guiding signals.Sampled points are kept at least two pixels from object boundaries.
  • Guiding signal for nuclei/cells:: At test time, user clicks generate inclusion and exclusion maps, with information-map sizes of 128 × 128 for nuclei and 256 × 256 for cells.Exclusion maps remain zero when only one nucleus or cell is clicked.
  • Guiding signal for nuclei/cells:: NuClick’s architecture combines convolutional, residual, and multi-scale convolutional blocks.
  • Guiding signal for glands:: Because glands are larger and more complex, the framework uses an internal squiggle rather than a single point to guide gland segmentation.The squiggle indicates the desired gland’s extent and connectivity.
  • Guiding signal for glands:: For gland training, distance-transformed masks are thresholded to create varying masks whose morphological skeletons serve as guiding signals.Changing the threshold changes the skeleton and helps avoid training on one fixed shape.

D. Post-processing

NuClick predictions are assembled into an instance segmentation map through patch localization and post-processing. The experiments use gland, nuclei, and cell datasets, including synthetic touching-WBC images.

  • D. Post-processing: Image patches, inclusion maps, and exclusion maps are fed to NuClick, while stored patch locations enable construction of the final instance segmentation map.
  • D. Post-processing: Post-processing thresholds prediction maps at 0.5, removes objects smaller than 50 pixels, and uses morphological reconstruction to remove undesired objects.The inclusion map acts as the marker and the initial segmentation as the reconstruction mask.
  • Datasets: Gland experiments use the GlaS and GRAG datasets, including 165 GlaS tiles and 213 GRAG images.GlaS contains separate training and test partitions, with TestA and TestB released at different challenge times.
  • Datasets: Nuclei experiments use MonuSeg and CPM, which contain 30 and 32 H&E images, respectively, with 16 images from each dataset used for training.
  • Datasets: Cell experiments use 2,689 synthetic images of touching white blood cells generated from 11,000 manually segmented non-touching WBCs.The source cells cover neutrophils, lymphocytes, eosinophils, monocytes, and basophils.
  • Datasets: Synthetic WBC images contain 10 to 30 touching cells, and 20% of training images are reserved for validation across datasets.Cell augmentation is used during image generation to reduce overfitting to specific WBC shapes.

B. Implementation Details

NuClick was evaluated with standard instance-segmentation metrics across nuclei, cells, and glands, using comparisons with interactive, supervised, and unsupervised methods. It achieved strong results across all three tasks, while annotation remains intensive when images contain many objects.

  • Metrics: AJI, Dice, Hausdorff distance, DQ, and SQ were used for nuclei and cell segmentation, while glands were evaluated with F1-score, DiceObj, and Hausdorff distance.The gland F1-score uses thresholded IoU to define true positives, and DiceObj averages Dice values across objects.
  • Comparative evaluation: The evaluation compared NuClick with region growing, active contour, marker-controlled watershed, GrabCut, Deep GrabCut, DEXTRE, BIFseg, Mask-RCNN, iFCN, and LD methods.Several comparator methods used ground-truth-derived markers, boxes, or boundary points for their evaluations.
  • Nuclei validation: NuClick achieved AJI 0.834, Dice 0.912, and PQ 0.838 on the MoNuSeg nuclear-segmentation validation set, outperforming the compared methods.Compared with Watershed, its AJI was 0.645 higher; compared with iFCN, its margins were 2.8%, 3.4%, and 5.6% for AJI, Dice, and PQ.
  • Cell validation: NuClick achieved AJI 0.954, Dice 0.983, and PQ 0.958 for cell segmentation on synthesized dense blood-smear images.The reported experiment specifically included touching WBCs.
  • Gland validation: For gland segmentation on GlaS testB, NuClick achieved F1-score 1.0, Dice 0.951, and Hausdorff distance 21, outperforming the best supervised comparator BIFseg.The reported improvements over BIFseg were 7.9%, 8.7%, and 17 pixels for these metrics, respectively.
  • Practical considerations: Annotating images remains intensive when they contain many nuclei or cells, because each object may require interaction.The paper also excludes some methods from nucleus and cell comparisons because four boundary clicks or a bounding box can be laborious per object.

V. DISCUSSIONS

The discussion describes a set of evaluation experiments designed to characterize NuClick beyond a single performance comparison. These experiments examine generalizability, adaptation to new domains, output reliability, and sensitivity to guiding-signal variation.

  • The evaluation first assesses the generalizability of the proposed framework.
  • A separate experiment examines whether NuClick can adapt to new domains without further training.
  • The evaluation also studies output reliability and sensitivity to variations in the guiding signals.

A. Generalization study

NuClick generalized across unseen datasets within nuclei and gland segmentation domains. Cross-dataset evaluations and visual examples indicate accurate segmentation when models were trained on one dataset and tested on another.

  • Cross-dataset evaluation: NuClick was evaluated across unseen datasets by training on one dataset and testing on another within the same domain.Nuclei experiments used MoNuSeg and CPM; gland experiments used CRAG and GLaS.
  • Nuclei generalization: 0.908 Dice and 0.821 SQ on CPM were close to 0.912 Dice and 0.839 SQ on MoNuSeg for a model trained on MoNuSeg.The authors interpret this closeness across datasets as evidence of nuclei generalizability.
  • Gland generalization: High DiceObj values and low Hausdorff distances on GlaS test sets supported generalizability for gland segmentation after training on CRAG.GlaS test sets were divided into TestA and TestB for evaluation.
  • Visual evidence: Visual examples showed successful nuclei segmentation on CPM using a MoNuSeg-trained model and gland segmentation on CRAG using a GLaS-trained model.The figure overlays predicted regions, ground-truth boundaries, and point or squiggle guiding signals.

B. Domain adaptation study

NuClick was tested on nuclei from cytology and immunohistochemistry domains after training on histopathology images. Generated annotations were sufficiently reliable for training segmentation models, with performance close to models trained on ground-truth masks.

  • Unseen-domain evaluation: NuClick was trained on MoNuSeg histopathology nuclei and used to segment nuclei in cytology and immunohistochemistry samples.The study evaluated transfer to Pap Smear and IHC images from different data domains.
  • Cytology: Pap Smear images contained overlapping cervical cells and other material, with cervical-cell nuclei manually segmented using cytotechnologist assistance.The cytology dataset comprised 42 fields of view from 10 Pap Smear samples.
  • Immunohistochemistry: The IHC experiment used 441 patches from LYON19 to delineate lymphocytes, although challenge organizers did not release instance segmentation or detection ground truths.The patches came from breast, colon, and prostate samples stained for CD3 or CD8.
  • Annotation reliability: Models trained on NuClick-generated masks achieved metrics with negligible differences from models trained on ground-truth masks.For FCN8 on MoNuSeg, NuClickCPM annotations produced Dice and SQ values 0.01 and 0.006 higher than ground-truth annotations, respectively.
  • Practical utility: NuClick annotations were judged reliable enough for practical training use and more efficient than manually obtained annotations.The reliability comparison used U-Net, SegNet, and FCN8 segmentation networks.

D. Sensitivity to Guiding Signals

NuClick was evaluated for robustness to click and squiggle variation, exclusion-map removal, and difficult objects. Performance remained reliable across many input variations, although large perturbations and missing exclusion maps could reduce performance.

  • Click-position sensitivity: Nuclei and cell segmentation remained robust when click positions were perturbed, with larger drops occurring when jitter exceeded the radius of small objects.For cells, click perturbations were tested up to 50 pixels because white blood cells had diameters of at least 80 pixels.
  • Squiggle sensitivity: NuClick successfully handled diverse squiggle shapes, lengths, placements, and discontinuities in gland segmentation examples.Quantitative sensitivity analysis was unavailable because squiggle changes were not easily measurable or parameterizable.
  • Guiding-signal design: Simple or small squiggles provided enough guiding information for gland-segmentation training to converge, whereas extreme-point and polygon signals failed to converge.The failed alternatives included filled and hollow polygons with many vertices.
  • Exclusion-map sensitivity: Removing the exclusion map significantly reduced nuclei-segmentation performance on MoNuSeg because overlapping nuclei lacked neighboring-object information.Without the exclusion map, the network input contained RGB channels plus an inclusion map.
  • Extreme cases: NuClick produced reasonable annotations for touching nuclei, clustered blood cells, irregular glands, and gland regions containing holes in challenging cancer images.For glands, squiggle coverage determined whether hole regions were excluded or included in the final segmentation.

F. User Correction

NuClick supports iterative user correction when its initial prediction is incorrect. Users can modify clicks or squiggles and resubmit the revised supervisory signal to obtain a new prediction.

  • Correction workflow: Users can correct inaccurate outputs by moving clicks or extending or reshaping squiggles.The revised supervisory signal is fed back to the network to produce a new segmentation.
  • Practical scope: The framework is intended to minimize human effort for dense annotation of nuclei, cells, and glands across datasets and data distributions.The authors propose NuClick as a possible plug-in for whole-slide annotation programs.
Loading 2005.14511v2…