PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Learning with Mixed Supervision for Brain Tumor Segmentation

Pawel Mlynarski, Hervé Delingette, Antonio Criminisi, Nicholas Ayache

arXiv:1812.04571v1cs.CVcs.LGeess.IVstat.ML

TL;DR

Manual tumor delineation is costly and scarce, while image-level tumor labels are cheaper to obtain. The paper jointly trains segmentation and image-level classification using both annotation types, and reports significant segmentation improvement when fully annotated cases are limited.

  • Problem

    Manual tumor segmentations are costly, time-consuming, require medical expertise, and are generally available in limited numbers.

    Source: §1 · p. 2

  • Method

    The model jointly trains segmentation and image-level classification branches using fully annotated images and weakly annotated images labeled for tumor presence or absence.

    Source: §1 · pp. 2–3

  • Results

    The mixed-supervision approach significantly improves segmentation performance when the number of manually segmented images is limited.

    Source: §1 · p. 4, §5 · p. 18

  • Takeaways & Limitations

    Weakly annotated medical images can be used alongside a small number of manually segmented images to improve tumor-segmentation performance in the tested settings.

    Source: §5 · p. 18

    The experiments used approximately 220 weakly annotated MRI, and the authors identify evaluation on datasets with many more such images as an important next step.

    Source: §5 · p. 19

Abstract

from arXiv · show

Most of the current state-of-the-art methods for tumor segmentation are based on machine learning models trained on manually segmented images. This type of training data is particularly costly, as manual delineation of tumors is not only time-consuming but also requires medical expertise. On the other hand, images with a provided global label (indicating presence or absence of a tumor) are less informative but can be obtained at a substantially lower cost. In this paper, we propose to use both types of training data (fully-annotated and weakly-annotated) to train a deep learning model for segmentation. The idea of our approach is to extend segmentation networks with an additional branch performing image-level classification. The model is jointly trained for segmentation and classification tasks in order to exploit information contained in weakly-annotated images while preventing the network to learn features which are irrelevant for the segmentation task. We evaluate our method on the challenging task of brain tumor segmentation in Magnetic Resonance images from BRATS 2018 challenge. We show that the proposed approach provides a significant improvement of segmentation performance compared to the standard supervised learning. The observed improvement is proportional to the ratio between weakly-annotated and fully-annotated images available for training.

1. Introduction

Manual tumor delineation is costly and scarce, motivating segmentation methods that can also use cheaper image-level labels. The paper jointly trains segmentation and classification with both label types, reporting improved accuracy when fully annotated cases are limited.

1. Introduction

  • Different MR sequences highlight tumor subcomponents including edema, necrosis, and the contrast-enhancing core.Medical images support cancer diagnosis, therapy planning, and monitoring.Source: §1 · p. 2
  • Manual tumor segmentation is time-consuming, requires medical expertise, and is generally available in limited numbers.Tumor boundaries can be difficult and subjective to delineate; expert segmentations have interrater Dice overlap of 0.74–0.85.Source: §1 · p. 2
  • Weak annotations indicate tumor presence or absence at image level and are less costly to obtain than full segmentations.The paper calls this training setting mixed supervision.Source: §1 · pp. 2–3
  • The model adds an image-level classification branch to segmentation networks and jointly trains both tasks on fully and weakly annotated images.The stated goal is to learn from weak labels while using fully annotated images to guide learning of segmentation-relevant features.Source: §1 · p. 3
  • Cross-validated BRATS 2018 experiments evaluate binary and multiclass glioma segmentation with variable numbers of training ground-truth segmentations.The study focuses on 2D tumor segmentation in axial MRI slices and assumes slice-level labels for weakly annotated images.Source: §1 · pp. 3–4
  • Mixed supervision significantly improves segmentation accuracy when the number of fully annotated cases is limited.The evaluation uses approximately 220 MRI with slice-level labels alongside varying numbers of fully annotated MRI.Source: §1 · p. 4

2. Related work

Related work spans weakly supervised localization, semi-supervised segmentation, and mixed-supervision approaches. The cited literature includes challenges with weak-only performance, uncertain self-training labels, and methods that use stronger annotations than image-level labels.

2. Related work

  • Weakly supervised localization methods use image-level labels with class heat maps, spatial priors, inconsistency measures, or image-subregion classification.Other approaches localize objects by identifying subregions or by measuring how image modifications affect classification scores.Source: §2 · pp. 4–5
  • Reported weakly supervised segmentation performance is considerably lower than semi-supervised and supervised approaches, and models without pixel labels may learn irrelevant features.The text notes co-occurrences and multicenter image-acquisition differences as examples; it also states that at least a few fully annotated images may be obtainable.Source: §2 · p. 5
  • Semi-supervised segmentation work includes self-training methods, but these rely on uncertain model-generated labels and can be difficult to train.The cited literature also includes graph-based methods whose similarity computations are difficult at image-segmentation scale.Source: §2 · pp. 5–6
  • Prior mixed-supervision work used sequential classification and segmentation networks for natural images, while another approach used segmentations, boxes, and seed points rather than image-level labels.The sequential design may miss image details because its segmentation network does not receive the original image.Source: §2 · p. 6

3. Joint classification and segmentation with Convolutional Neural Networks

The section describes a jointly trained U-Net extension with an image-level classification branch and its multiclass adaptation. It also reports better performance when each tumor subclass has a dedicated classification branch.

3.1. Deep learning model for binary segmentation

  • The binary model extends U-Net with a classification branch that shares features with the segmentation network and is trained jointly for both tasks.The classification branch takes input from U-Net’s second-to-last convolutional layer.Source: §3.1 · pp. 6–7, Figure 2 · p. 7
  • Training batches combine tumor-positive images with segmentations, tumor-negative images, and tumor-positive images without segmentations.The experiments used 4, 2, and 4 images of these types, respectively.Source: §3.1 · p. 8
  • Segmentation uses weighted pixelwise cross-entropy, with batch-set class weights of 0.7 for healthy pixels and 0.3 for tumor pixels.The weights counter class imbalance; tumor pixels receive higher relative weight than in standard unweighted cross-entropy.Source: §3.1 · p. 9
  • Because both losses are normalized, the total loss is a convex combination: segmentation receives weight 0.7 and classification 0.3, with the segmentation weight chosen empirically through cross-validated tests.Fully annotated images are also used to train the classification branch.Source: §3.1 · p. 9

3.2. Extension to the multiclass problem

  • For multiclass segmentation, the model outputs one feature map per class and uses a separate image-level classification output for each tumor subclass.The four segmentation classes in the BRATS setting include non-tumor and three tumor subclasses; image-level labels are not mutually exclusive.Source: §3.2 · p. 10, Figure 3 · p. 11
  • Experiments found better performance when each tumor subclass had its own entire classification branch.The authors suggest that subclass classification is challenging and needs sufficient dedicated parameters.Source: §3.2 · p. 10
  • The multiclass segmentation loss assigns weights of 0.7 to non-tumor and 0.1 to each tumor subclass, with the segmentation-loss coefficient set to 0.3.The authors report choosing the coefficient through cross-validation.Source: §3.2 · p. 10

4. Experiments

The experiments compare mixed supervision with fully supervised U-Net across BRATS 2018 binary and multiclass segmentation, varying the number of manually annotated cases. Mixed supervision helps most when manual annotations are limited.

4.1. Data

  • BRATS 2018 training data comprise 285 patients with four MRI sequences; because all volumes contain tumors, evaluation uses 2D axial-slice segmentation.The benchmark groups labels into whole tumor, tumor core, and enhancing core.Source: §4.1 · pp. 11–12

4.2. Test setting

  • Five-fold tests compared mixed supervision with fully annotated-only U-Net, using 57 test patients per fold and varying fully annotated cases across 5, 15, and 30.The experiments covered binary segmentation of three tumor regions and end-to-end multiclass segmentation.Source: §4.2 · pp. 12–13

4.3. Results

  • With 5 fully annotated scans, mixed supervision adds 8 points of mean Dice for binary whole-tumor segmentation and 9 points of Dice multiclass; at 30 scans, performance is similar.The comparison against fully supervised U-Net is significantly better in the 5- and 15-case scenarios.Source: §4.3 · pp. 13–14, Table 1 · p. 14, Table 3 · p. 17
  • With 5 fully annotated and 223 weakly annotated scans, whole-tumor mean Dice is 78.3 versus 77.9 for standard supervision with 15 fully annotated scans; mixed-supervision gains decline as fully annotated cases increase.Both methods improve as more manual segmentations are available.Source: §4.3 · pp. 14–15, Figure 5 · p. 15, Figure 7 · p. 17

5. Conclusion and future work

The jointly trained segmentation and classification model uses weakly annotated MRI alongside a small number of manually segmented images, improving segmentation when manual cases are limited. Future work includes 3D segmentation and evaluation with many more weakly annotated images.

5. Conclusion and future work

  • Weakly annotated data significantly improves segmentation performance when the number of manually segmented images is limited.The experiments covered binary and multiclass settings.Source: §5 · p. 18
  • The model produces voxelwise and image-level outputs, with subnetworks that share most layers and train jointly on fully and weakly annotated data.The authors describe it as end-to-end and straightforward to implement with common deep learning libraries.Source: §5 · p. 18
  • The study focuses on 2D segmentation because BRATS 2018’s 3D images all contain tumors, while classification training also needs non-tumor images.A 3D extension could incorporate richer spatial context, and volume-level labels would be easier to obtain but less informative than slice-level labels.Source: §5 · pp. 18–19
  • The experiments used approximately 220 weakly annotated MRI, and the authors identify testing on datasets with thousands or millions of such images as future work.Source: §5 · p. 19

Related papers

Loading 1812.04571v1…