PaperlayerView exact revision on arXiv

Source-linked AI summary

DeepCut: Object Segmentation from Bounding Box Annotations using Convolutional Neural Networks

Martin Rajchl, Matthew C. H. Lee, Ozan Oktay, Konstantinos Kamnitsas, Jonathan Passerat-Palmbach, Wenjia Bai, Mellisa Damodaram, Mary A. Rutherford, Joseph V. Hajnal, Bernhard Kainz, Daniel Rueckert

arXiv:1605.07866v2cs.CV

TL;DR

DeepCut addresses the difficulty of obtaining large pixelwise-annotated medical image datasets by learning segmentations from bounding boxes. It combines CNN-based target updates with densely connected CRF energy minimisation and performs well on fetal brain and lung MRI, including accuracy comparable to fully supervised learning while reducing annotation effort.

  • Problem

    Large medical image-analysis datasets need extensive annotations, but pixelwise labeling is often infeasible, motivating methods that learn from bounding boxes and other weak annotations.

  • Method

    DeepCut alternates CNN model updates and label updates within an energy-minimisation framework over a densely connected CRF, using CNNs instead of GrabCut's GMM appearance model.

  • Results

    DeepCut segments fetal brain and lungs from highly variable MRI databases and performs well relative to fully supervised and naive weakly supervised approaches.

  • Takeaways & Limitations

    Bounding-box annotations can support accurate medical image segmentation while substantially reducing the annotation effort required for analysis.

Abstract

from arXiv · show

In this paper, we propose DeepCut, a method to obtain pixelwise object segmentations given an image dataset labelled with bounding box annotations. It extends the approach of the well-known GrabCut method to include machine learning by training a neural network classifier from bounding box annotations. We formulate the problem as an energy minimisation problem over a densely-connected conditional random field and iteratively update the training targets to obtain pixelwise object segmentations. Additionally, we propose variants of the DeepCut method and compare those to a naive approach to CNN training under weak supervision. We test its applicability to solve brain and lung segmentation problems on a challenging fetal magnetic resonance dataset and obtain encouraging results in terms of accuracy.

I. INTRODUCTION

DeepCut addresses the annotation burden in medical image segmentation by using bounding boxes and learned shared object information to recover pixelwise segmentations. It combines CNN learning with iterative graphical optimisation, extending GrabCut-style weak supervision.

  • Motivation: Large medical image-analysis datasets require extensive annotations to cover variability, but pixelwise labeling by one rater is often infeasible.Weak annotations such as image-level tags, bounding boxes, and scribbles aim to reduce effort while retaining comparable accuracy to full supervision.
  • Bounding-box annotations: Bounding boxes spatially constrain segmentation and can be placed using two corner coordinates, making them faster to specify than pixelwise annotations.They are used both to initialise segmentation and to constrain the object region.
  • Learning across images: Shared shape and appearance information across an image database can be learned as a common model instead of segmenting each image independently.This is particularly relevant to medical cohorts containing similar organs or regions.
  • Related approaches: Earlier bounding-box methods use image appearance models and smoothness constraints, while weakly supervised learning methods infer pixelwise segmentations from partial labels.Related approaches include GrabCut variants, CRF methods, level sets, graph cuts, EM, and multiple-instance learning.
  • Proposed approach: DeepCut combines a neural network with iterative graphical optimisation to recover pixelwise segmentations from bounding-box annotations.It updates CNN training targets and uses a fully connected CRF to regularise the segmentation, extending the GrabCut idea.

B. Contributions

The paper formulates DeepCut as iterative energy minimisation over a densely connected CRF, using a CNN to provide pixelwise data terms and update segmentation targets. It evaluates variants against supervised and naive weakly supervised baselines on fetal MRI.

  • Contributions: DeepCut learns pixelwise segmentations from image databases with bounding-box annotations by combining a neural network and iterative graphical optimisation.The study compares fully supervised CNN, naive weak supervision, DeepCut variants, and GrabCut on fetal MRI.
  • Energy formulation: The energy comprises unary data consistency and pairwise regularisation terms over pixel labels.The unary term measures label fit to the data, while the pairwise term penalises label differences between pixel locations.
  • CRF regularisation: A densely connected CRF replaces the original pairwise formulation to impose contrast-sensitive smoothness between image locations.The pairwise design includes appearance and smoothness penalties weighted by regularisation parameters.
  • CNN data model: The unary potential is the negative log-likelihood produced by a CNN from an input image or patch.Unlike GrabCut's GMM-based unary term, DeepCut uses CNN parameters Θ to model label probabilities.

A. Segmentation by Iterative Energy Optimisation

DeepCut follows GrabCut's two-stage iterative structure, alternating model estimation with label updates. It replaces GrabCut's GMM and graph-cut solver with a neural network and densely connected CRF, while retaining parameters across iterations.

  • GrabCut structure: GrabCut alternates between estimating appearance-model parameters from current labels and updating pixel labels using the new model.This provides the two-stage structure that DeepCut adapts.
  • DeepCut adaptation: DeepCut replaces GrabCut's GMM with a neural network and its graph-cut solver with inference on a densely connected graph.The method reinitialises the CNN with the previous iteration's parameters using transfer learning.
  • EM interpretation: The two-step procedure resembles an EM algorithm, with label updates as an E-step and model updates as an M-step.The paper contrasts this with an approach that applies regularisation only as post-processing during testing.

B. Convolutional Neural Network Model

DeepCut uses a feed-forward CNN to classify image patches around voxel locations as foreground or background. The architecture combines convolutional feature extraction, max-pooling, and dense classification layers.

  • Architecture: The CNN is a hierarchical feed-forward network with convolutional, max-pooling, and densely connected layers.Convolution extracts features, max-pooling supports shift and scale invariance, and dense layers perform classification.
  • Segmentation model: The model classifies patches centred on voxel locations into foreground and background while restricting the learned objects to their bounding boxes.A CNN with parameters Θ operates on image patches from a database of images and corresponding bounding boxes.
  • Patch representation: Each voxel is represented by a 3D patch whose centre label is binary, with 0 for background and 1 for foreground.The study uses in-plane emphasis because fetal MR slices may contain motion artefacts.

Network Configuration:

The study uses a simple CNN with convolutional, max-pooling, and fully connected layers for foreground/background classification, trained on augmented patches. Dropout and intensity/flip augmentation are used to improve generalisation and reduce over-fitting.

  • Network architecture: The CNN uses two convolutional and max-pooling layer sets followed by fully connected classification layers.Zero-padding preserves tensor size after convolution, while pooling supports representations at different scales.
  • Regularisation: Layers marked with an asterisk apply 50% dropout to their inputs.
  • Training: Training extracts K = 105 patches per epoch, equally distributed between the classes.Weights are Gaussian-initialised and optimised with mini-batch ADAGRAD using η = 0.015 for a fixed number of epochs.
  • Data augmentation: Gaussian intensity offsets and random spatial flips augment patches to increase variation and prevent over-fitting.The intensity offset uses standard deviation σ.
  • Target generation: The naïve classifier assigns patches inside the bounding box to foreground and patches in the halo to background.Because the bounding box can include false-positive locations, this provides only an initial weak-labeling strategy.

D. DeepCut

DeepCut alternates CNN training with inference and densely connected CRF regularisation to update voxel labels. Its iterative optimisation is locally optimal and therefore depends on the initial foreground and background regions.

  • D. DeepCut: DeepCut interrupts CNN training after fixed epochs, updates voxel classes in B through inference and CRF regularisation, then continues training.The CNN is reinitialised with the parameters learned in the preceding iteration.
  • D. DeepCut: The iterative algorithm is limited to local optima, making the resulting segmentation dependent on the initial regions R_FG and R_BG.
  • D. DeepCut: Pre-segmentation within B can initialise foreground and background regions closer to the object and has been associated with large accuracy improvements.The paper examines analogous initialisation for DeepCut.
  • D. DeepCut: Experiments use MR images from 55 fetal subjects, including healthy subjects and subjects with intrauterine growth restriction.The images contain motion artefacts typical of fetal MRI acquisitions.

B. Preprocessing & Generation of Bounding Boxes

Preprocessing corrects bias fields and normalises intensities using bounding-box statistics, while bounding boxes and background halos are generated from manual segmentations. Evaluation uses fixed five-fold cross-validation and Dice overlap, with separate data for parameter tuning.

  • B. Preprocessing & Generation of Bounding Boxes: Bounding boxes are enlarged by 5 voxels per slice from manual segmentation extents, and halo regions extend them by 20 voxels.
  • B. Preprocessing & Generation of Bounding Boxes: Images undergo bias-field correction and intensity normalisation to zero mean and unit standard deviation computed from the bounding box.
  • D. Experimental Setup, Evaluation & Parameter Selection: The comparative setup fixes CNN architecture, preprocessing, and CRF parameters across learning-based methods.CNN_naïve is treated as a lower accuracy bound, while fully supervised CNNFS is treated as an upper bound under model and data constraints.
  • D. Experimental Setup, Evaluation & Parameter Selection: Five-fold cross-validation uses fixed training and testing databases for all compared methods.
  • D. Experimental Setup, Evaluation & Parameter Selection: Segmentations are evaluated by overlap with expert manual segmentations using the Dice Similarity Coefficient.Three datasets are reserved to tune GrabCut and CRF regularisation parameters via random parameter permutations.

E. Implementation Details & Hardware

The implementation uses Lasagne and Theano with a CRF solver on a Tesla K80 system. Results show that DeepCut improves substantially over naïve bounding-box learning, while pre-segmentation further improves agreement and approaches fully supervised performance.

  • E. Implementation Details & Hardware: The CNN is implemented with Lasagne and Theano, and CRF inference uses the implementation described in prior work.
  • E. Implementation Details & Hardware: Experiments run on Ubuntu 14.04 machines with 256 GB memory and a single 12 GB Tesla K80 GPU.
  • B. Initialisation with Pre-segmentations: DCPS shows comparable agreement to fully supervised CNNFS for the brain region.The example figure compares original images, manual segmentations, initial regions, and outputs from all evaluated methods.
  • A. Naïve Learning Approach versus DeepCut: 12.6% and 8.9% average DSC increases are obtained for brain and lungs, respectively, when comparing DeepCut with naïve bounding-box learning.
  • B. Initialisation with Pre-segmentations: Pre-segmentation improves mean DSC by another 3.7% for brain and 4.9% for lungs over bounding-box initialisation.

C. Comparison with GrabCut

GrabCut performs well for fetal brain segmentation but is less robust for lungs, where large outliers lower mean accuracy. DeepCut accuracy improves over iterations toward fully supervised performance and exceeds the naive approach.

  • GrabCut comparison: GrabCut achieves 80.7±4.9% DSC for fetal brain segmentation but only 58.6 ± 19.0% DSC for lungs.Several lung segmentations have DSC < 20%, indicating missed objects or false-positive voxels.
  • Iterative improvement: DeepCut variants increase segmentation accuracy after each iteration and approach the fully supervised CNNFS upper bound.Figure 6 reports mean accuracy in black and standard deviation in gray for CNNna¨ıve and CNNFS reference bounds.
  • Iterative improvement: Both DCBB and DCPS achieve higher average accuracy than the naive CNN approach, while remaining below CNNFS.The comparison includes bounding-box and pre-segmentation initialisations.
  • Methods compared: The compared methods are evaluated for fetal brain and lung segmentation using BB, GC, CNNna¨ıve, DCBB, DCPS, and CNNFS.CNNFS is used as the upper bound for the network architecture.

V. DISCUSSION

DeepCut produces pixelwise segmentations from bounding-box annotations using a generally applicable CNN-based approach. Its weakly supervised DCPS variant reaches accuracy close to fully supervised methods on fetal brain segmentation, while performance differs between brain and lungs.

  • Method and scope: DeepCut obtains pixelwise segmentations from an image database using bounding-box annotations and CNN models without feature engineering.The method is formulated as an iterative energy-minimisation problem over a densely connected CRF.
  • Method and scope: The database includes substantial anatomical variation, including fetal position, gestational ages of 20-38 weeks, and growth restriction.Learning background patches from the halo H avoids modelling the entire image domain and enables faster training.
  • Differences across tasks: Brain segmentation is more accurate than lung segmentation across internally compared methods because brain shape and background contrast are more favorable.The brain’s regular shape is better approximated by bounding boxes, while stronger background contrast benefits intensity-based methods.

D. Effect of Initialisation on DeepCut performance

DeepCut performs better when initialized with a pre-segmentation rather than a bounding box, but its iterative optimization can converge to different local optima. Across fetal brain and lung segmentation, DeepCut variants substantially outperform naive learning and perform well relative to full supervision.

  • Initialization effects: Pre-segmentation initialization increases DeepCut segmentation accuracy compared with bounding-box initialization.The same trend was reported for another EM-based method.
  • Optimization behavior: DeepCut’s iterative optimization is locally optimal, so different initializations can lead to different converged optima.The authors suggest more frequent target updates, while noting that tuning CRF regularization at each iteration may be computationally expensive.
  • Initialization effects: Closer initialization improves performance even when the pre-segmentation is inaccurate, by reducing false-positive targets during initial training.The reported improvement from DCBB to DCPS illustrates this effect.
  • Segmentation results: DeepCut variants report large accuracy improvements over naive learning for both fetal lung and brain segmentation.Learning-based methods also show generally low standard deviations compared with image segmentation methods such as GrabCut.
  • Segmentation results: DeepCut performs well compared with full supervision while substantially reducing the annotation effort required for fetal brain and lung analysis.The method is described as applicable to medical-image problems with large anatomical variation.
Loading 1605.07866v2…