PaperlayerView exact revision on arXiv

Source-linked AI summary

Interactive Medical Image Segmentation using Deep Learning with Image-specific Fine-tuning

Guotai Wang, Wenqi Li, Maria A. Zuluaga, Rosalind Pratt, Premal A. Patel, Michael Aertsen, Tom Doel, Anna L. David, Jan Deprest, Sebastien Ourselin, Tom Vercauteren

arXiv:1710.04043v1cs.CV

TL;DR

Medical-image CNNs lack robust generalization to unseen objects and adaptation to individual test images. BIFSeg combines bounding-box and optional scribble interactions with image-specific CNN fine-tuning and uncertainty-weighted loss, achieving robust unseen-object segmentation and improved accuracy with fewer interactions and less user time.

  • Problem

    Medical-image CNNs have limited clinical robustness, poor generalization to previously unseen objects, and no image-specific adaptation during testing.

  • Method

    BIFSeg uses bounding-box CNN binary segmentation with optional scribble-guided image-specific fine-tuning and a weighted loss based on network and interaction uncertainty.

  • Results

    BIFSeg performs well on previously unseen objects, and image-specific fine-tuning outperforms CRF across fetal-MRI organ and brain-tumor MRI experiments.

  • Takeaways & Limitations

    The framework can be applied to some new organs or segmentation protocols without requiring annotations for all organs and can achieve similar or higher accuracy with fewer interactions in less time.

Abstract

from arXiv · show

Convolutional neural networks (CNNs) have achieved state-of-the-art performance for automatic medical image segmentation. However, they have not demonstrated sufficiently accurate and robust results for clinical use. In addition, they are limited by the lack of image-specific adaptation and the lack of generalizability to previously unseen object classes. To address these problems, we propose a novel deep learning-based framework for interactive segmentation by incorporating CNNs into a bounding box and scribble-based segmentation pipeline. We propose image-specific fine-tuning to make a CNN model adaptive to a specific test image, which can be either unsupervised (without additional user interactions) or supervised (with additional scribbles). We also propose a weighted loss function considering network and interaction-based uncertainty for the fine-tuning. We applied this framework to two applications: 2D segmentation of multiple organs from fetal MR slices, where only two types of these organs were annotated for training; and 3D segmentation of brain tumor core (excluding edema) and whole brain tumor (including edema) from different MR sequences, where only tumor cores in one MR sequence were annotated for training. Experimental results show that 1) our model is more robust to segment previously unseen objects than state-of-the-art CNNs; 2) image-specific fine-tuning with the proposed weighted loss function significantly improves segmentation accuracy; and 3) our method leads to accurate results with fewer user interactions and less user time than traditional interactive segmentation methods.

I. INTRODUCTION

Medical-image CNNs achieve strong automatic segmentation, but clinical robustness is limited by image variability, unseen object classes, and insufficient image-specific adaptation. The paper motivates combining CNNs with user interactions to improve accuracy and robustness while reducing interaction burden.

  • Medical images vary in quality, imaging and segmentation protocols, and patient anatomy, limiting the clinical accuracy and robustness of automatic segmentation.
  • Interactive segmentation incorporates user knowledge to distinguish tissues and remains widely used in commercial surgical planning and navigation.
  • CNN-based interactive segmentation aims to achieve higher accuracy and robustness with fewer user interactions and less user time.
  • CNNs generalize poorly to previously unseen object classes because labeled instances of each class are typically required during training.
  • Current CNNs use fixed test-time parameters and therefore lack image-specific adaptation despite context variation across images.
  • Interactive CNNs must also provide fast inference and memory efficiency, especially for 3D volumes where patch-based and whole-volume approaches impose different costs.

2) Interactive Segmentation Methods:

The paper positions BIFSeg as an image-specific CNN adaptation framework that extends bounding-box and scribble-based interactive segmentation to unseen objects in both 2D and 3D. Its networks preserve resolution with dilated convolutions while balancing 3D context, inference speed, and memory use.

  • Interactive Segmentation Methods: Traditional interactive methods include Graph Cuts, Random Walks, GeoS, GrabCut, and SlicSeg, while earlier CNN methods use interactions mainly as sparse training annotations.
  • Interactive Segmentation Methods: Image-specific model adaptation was previously used with GMMs and ORFs, but BIFSeg proposes this adaptation for CNNs.
  • Interactive Segmentation Methods: BIFSeg adapts a CNN to each test image through alternating segmentation and unsupervised or scribble-guided supervised fine-tuning from a user-provided bounding box.
  • CNN Models: The 2D P-Net uses resolution-preserving dilated convolutions and multiscale features to avoid losing image details.
  • CNN Models: The 3D PC-Net uses an anisotropic receptive field and feature compression to incorporate 3D context while reducing inference memory consumption.

B. Training of CNNs

The CNN training pipeline converts each annotated object instance into a cropped binary segmentation example, then adapts the model to a test image through alternating label and parameter updates constrained by user scribbles.

  • Training of CNNs: Training uses images containing multiple object classes by extracting each instance, calculating its bounding box, and expanding it with a random 0–10 pixel or voxel margin.
  • Training of CNNs: Each annotated object instance is converted into a cropped binary image-label pair, making training independent of object type.The foreground is the target instance and the background is label 0; a cross-entropy loss trains the CNN.
  • Training of CNNs: At testing, the model processes the sub-image inside a user-provided bounding box and predicts its target label map.
  • Training of CNNs: The image-specific objective combines unary CNN probabilities with pairwise terms and enforces scribble labels when user interactions are present.P-Net or PC-Net replaces the GMM used in GrabCut, while the pairwise term links neighboring labels according to image information and distance.
  • Training of CNNs: Optimization alternates segmentation-label updates with CNN-parameter updates; fixed-parameter label updates form a CRF problem solvable by Graph Cuts.The constrained formulation can be converted to an unconstrained equivalent, and the resulting label update preserves user-provided labels.

D. Weighted Loss Function during Network Update Step

During network updates, the method weights pixels according to confidence: scribbles receive greater influence, while pixels considered highly uncertain are excluded from fine-tuning.

  • Weighted Loss Function during Network Update Step: The weighted loss gives user scribbles weight ω ≥1 because their labels are more reliable than other pixels.The modified unary term changes the network update while leaving the label-update objective unchanged.
  • Weighted Loss Function during Network Update Step: Network-based uncertainty identifies pixels whose foreground probability lies between thresholds t0 and t1, indicating low-confidence predictions.
  • Weighted Loss Function during Network Update Step: Scribble-based uncertainty marks non-scribble pixels near foreground or background scribbles when their initial labels conflict with the nearby scribble type.The proximity is measured using geodesic distance, with threshold ϵ.
  • Weighted Loss Function during Network Update Step: The resulting weight map assigns 0 to high-uncertainty pixels, ω to scribbles, and 1 to remaining pixels.
  • Weighted Loss Function during Network Update Step: Zero unary weights are used only during the network update because applying them to label updates would leave those pixels entirely controlled by pairwise potentials.

E. Implementation Details

The implementation uses Caffe-based CNN training and lightweight desktop deployment, with four alternating image-specific fine-tuning iterations during interactive testing.

  • Implementation Details: P-Net and PC-Net were implemented in Caffe and trained with stochastic gradient descent using momentum 0.9, batch size 1, and weight decay 5 × 10−4.Training used a K80 GPU and a maximum of 60k iterations; the initial learning rate was 10−3 and halved every 5k iterations.
  • Implementation Details: Interactive testing ran on a MacBook Pro with 16GB RAM, an Intel Core i7 CPU, and an NVIDIA GeForce GT 750M GPU.Separate Matlab and PyQt interfaces supported 2D and 3D user interactions.
  • Implementation Details: Image-specific fine-tuning alternated updates of the segmentation and network parameters for four iterations.Each network update used learning rate 10−2; the passage begins specifying the corresponding iteration number.

III. EXPERIMENTS AND RESULTS

The framework was evaluated on fetal-MRI organ segmentation and brain-tumor segmentation, including objects absent from training annotations. The fetal-MRI comparison uses a common user-provided bounding box across methods.

  • III. EXPERIMENTS AND RESULTS: The experiments cover 2D segmentation of multiple fetal-MRI organs and 3D segmentation of brain tumors from T1c and FLAIR images.
  • III. EXPERIMENTS AND RESULTS: The fetal-MRI initial-segmentation comparison applies the same user-provided bounding box to every test instance.
  • III. EXPERIMENTS AND RESULTS: Both applications evaluate segmentation of previously unseen objects that were not included in the training set.

A. Comparison Methods and Evaluation Metrics

The study compares CNN-based and traditional interactive segmentation methods under shared bounding-box or scribble conditions, using Dice score and machine time for evaluation.

  • CNN Comparisons: P-Net is compared with FCN and U-Net for 2D images, while PC-Net is compared with DeepMedic and HighRes3DNet for 3D images.The comparisons use the same bounding box; HighRes3DNet runs on a cluster because of laptop GPU-memory limitations.
  • Visual Evaluation: The fetal MRI refinement comparison visualizes foreground probability as a heatmap for P-Net and three unsupervised refinement methods.The figure compares the methods visually rather than reporting a quantitative metric in its caption.
  • Evaluation Metrics: Interactive segmentations are repeatedly updated with user scribbles until accepted, and Dice score measures overlap with the ground truth.Dice is defined as 2|Ra ∩Rb|/(|Ra|+|Rb|), where Ra and Rb are the algorithmic and ground-truth regions.

B. 2D Segmentation of Multiple Organs from Fetal MRI

Fetal MRI experiments evaluate initial CNN segmentation and image-specific refinement across annotated and previously unseen organs, including unsupervised and scribble-guided settings.

  • Data: The fetal MRI dataset contains 18 pregnant women, with 10 volumes used for training and 8 volumes used for testing.The dataset includes 356 training slices and 318 testing slices; large inter-slice spacing and motion motivate interactive 2D segmentation.
  • Initial Segmentation: P-Net performs better than FCN and U-Net on previously unseen fetal lungs and maternal kidneys, achieving the best accuracy with average machine time 0.16s.FCN and U-Net produce substantial under-segmentation for these unseen organs, while the methods perform similarly on placenta and fetal brain.
  • Unsupervised Image-specific Fine-tuning: Unsupervised image-specific fine-tuning with BIFSeg improves accuracy more than CRF or BIFSeg(-w), requiring 0.72s on average for 2D refinement.Fine-tuning moves many uncertain foreground probabilities toward values near 0.0 or 1.0.
  • Supervised Image-specific Fine-tuning: Supervised refinement uses the same initial segmentation and scribbles across P-Net + CRF, BIFSeg(-w), and BIFSeg.All methods improve the segmentation, but BIFSeg achieves better results with the shared scribbles.
  • Supervised Image-specific Fine-tuning: BIFSeg achieves significantly better single-round refinement accuracy for placenta, previously unseen fetal lungs, and maternal kidneys than P-Net + CRF and BIFSeg(-w).The reported comparison uses p-value < 0.05 and a common set of scribbles.

5) Comparison with Other Interactive Methods:

The paper compares BIFSeg with traditional interactive methods and evaluates CNN-based 3D initialization and refinement using shared bounding-box conditions.

  • Comparison with Other Interactive Methods: BIFSeg is compared with SlicSeg, GrabCut, and Random Walks for fetal MRI segmentation using user interaction until the result is accepted.Two users, an Obstetrician and a Radiologist, perform the fetal MRI experiments.
  • Comparison with Other Interactive Methods: BIFSeg takes noticeably less user time with similar or higher accuracy than the other three fetal MRI interactive methods.The comparison reports final accuracy and user time after interactive completion.
  • 3D Initialization: For 3D brain tumor initialization, PC-Net and comparison CNNs use the same bounding box for each test image.The table reports Dice scores, marks previously unseen objects, and identifies statistically best values with p-value < 0.05.
  • Unsupervised Refinement: PC-Net and unsupervised refinement methods are compared without additional scribbles using the same initial PC-Net segmentation.The refinement comparison reports Dice score and machine time, with previously unseen objects explicitly marked.

C. 3D Segmentation of Brain Tumors from T1c and FLAIR

The 3D experiments test tumor-core segmentation and previously unseen whole-tumor segmentation across T1c and FLAIR MRI, then assess unsupervised image-specific refinement.

  • Data and Tasks: The BRATS experiment trains on 249 T1c volumes and tests on 25 remaining T1c volumes, while whole tumors in corresponding FLAIR images are previously unseen during training.The dataset contains 274 cases with multiple MR sequences and expert-delineated ground truth.
  • Initial Segmentation: PC-Net provides stable performance on tumor core and previously unseen whole tumor, while using less complex architecture and lower memory consumption than HighRes3DNet.DeepMedic performs differently across T1c and FLAIR, whereas HighRes3DNet tends to over-fit tumor core.
  • Initial Segmentation: PC-Net requires 3.83s average machine time, compared with 3.87s for 3D GrabCut and 65.31s for DeepMedic on the laptop.HighRes3DNet averages 1.10s on a cluster, so its timing uses different hardware.
  • Unsupervised Image-specific Fine-tuning: Unsupervised BIFSeg refinement raises average Dice from 82.66% to 86.13% for tumor core and from 83.52% to 86.29% for whole tumor.The refinement uses PC-Net’s initial output without additional user interactions.

4) Supervised Image-specific Fine-tuning:

Supervised image-specific fine-tuning improves refinement from the same initial segmentation and scribbles, while BIFSeg achieves strong accuracy on tumor cores and previously unseen whole tumors with less user time.

  • Qualitative refinement: BIFSeg achieved better refined results than CRF and BIFSeg(-w) for tumor cores in T1c and whole tumors in FLAIR.Both tumor categories were refined from the same initial PC-Net segmentation with identical scribbles.
  • Quantitative comparison: 87.49% and 88.11% average Dice scores were achieved for tumor core and previously unseen whole tumor after one refinement round, respectively, outperforming CRF and BIFSeg(-w).The comparison used the same initial segmentation and the same set of scribbles for all refinement methods.
  • User time: BIFSeg required 82.3s for tumor-core segmentation and 68.0s for whole-tumor segmentation, despite similar final Dice scores across interactive methods.Two users—a physician and a radiologist—evaluated GeoS, GrowCut, 3D GrabCut, and BIFSeg.
  • Generalization: The framework segments previously unseen objects, including fetal lungs, maternal kidneys, and whole tumors in FLAIR, without annotated instances for those objects during training.The networks were trained on placenta and fetal brain for 2D images, and tumor cores in T1c for 3D images.
  • Fine-tuning mechanism: Image-specific fine-tuning improves segmentation performance by using a weighted loss that incorporates network uncertainty and interaction-based uncertainty from scribbles.The fine-tuning is described as a post-processing step after initial segmentation and outperforms CRF.
Loading 1710.04043v1…