PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Cross-modality Medical Image Segmentation with Online Mutual Knowledge Distillation

Kang Li, Lequan Yu, Shujun Wang, Pheng-Ann Heng

arXiv:2010.01532v1eess.IVcs.CV

TL;DR

Medical annotations are costly, and appearance differences make shared knowledge across modalities difficult to exploit for segmentation. The paper aligns assistant-modality images and trains two segmentors through mutual knowledge distillation; it reports improved CT cardiac segmentation using additional MRI data, outperforming other multi-modality methods.

  • Problem

    Medical annotations are expensive and time-consuming, while appearance discrepancies make modality-shared features difficult to learn directly for segmentation.

    Source: Introduction · pp. 1–2

  • Method

    An Image Alignment Module translates assistant-modality images into synthetic target-modality images, while two segmentors learn from synthetic and real target-modality data and mutually distill knowledge.

    Source: Introduction · p. 2

  • Results

    3.06% Dice improvement on CT cardiac segmentation using additional MRI data is reported, outperforming other multi-modality learning methods.

    Source: Introduction · p. 2

  • Takeaways & Limitations

    The experiments support using assistant-modality priors to improve target-modality cardiac segmentation without requiring assistant-modality data at testing.

    Source: Introduction · pp. 1–2

Abstract

from arXiv · show

The success of deep convolutional neural networks is partially attributed to the massive amount of annotated training data. However, in practice, medical data annotations are usually expensive and time-consuming to be obtained. Considering multi-modality data with the same anatomic structures are widely available in clinic routine, in this paper, we aim to exploit the prior knowledge (e.g., shape priors) learned from one modality (aka., assistant modality) to improve the segmentation performance on another modality (aka., target modality) to make up annotation scarcity. To alleviate the learning difficulties caused by modality-specific appearance discrepancy, we first present an Image Alignment Module (IAM) to narrow the appearance gap between assistant and target modality data.We then propose a novel Mutual Knowledge Distillation (MKD) scheme to thoroughly exploit the modality-shared knowledge to facilitate the target-modality segmentation. To be specific, we formulate our framework as an integration of two individual segmentors. Each segmentor not only explicitly extracts one modality knowledge from corresponding annotations, but also implicitly explores another modality knowledge from its counterpart in mutual-guided manner. The ensemble of two segmentors would further integrate the knowledge from both modalities and generate reliable segmentation results on target modality. Experimental results on the public multi-class cardiac segmentation data, i.e., MMWHS 2017, show that our method achieves large improvements on CT segmentation by utilizing additional MRI data and outperforms other state-of-the-art multi-modality learning methods.

Introduction

Because medical annotations are costly and modalities differ in appearance while depicting shared anatomy, the paper investigates using assistant-modality knowledge to improve target-modality segmentation. It proposes image alignment and mutual distillation, reporting improved CT cardiac segmentation with additional MRI data.

Introduction

  • The motivation is to use shared anatomy across modalities to address annotation scarcity, since expert medical annotations are expensive and time-consuming.The paper aims to use assistant-modality priors without requiring assistant-modality data at testing.Source: Introduction · p. 1
  • Joint training and fine-tuning do not fully exploit shared cross-modality information when appearance discrepancies make modality-shared features difficult to learn directly.Existing dual-stream designs also require human intervention to determine parameter sharing and special adjustments for other tasks.Source: Introduction · pp. 1–2
  • 3.06% Dice improvement on CT cardiac segmentation and better performance than other multi-modality learning methods are reported using additional MRI data.The evaluation uses the MM-WHS 2017 Challenge dataset.Source: Introduction · p. 2
  • The framework combines an Image Alignment Module that translates assistant-modality images into synthetic target-modality images with mutual distillation between two segmentors.The segmentors learn from synthetic and real target-modality data, respectively, and exchange output guidance.Source: Introduction · p. 2

Related Work

Prior multi-modality segmentation methods fuse modality features or translate images, while this work targets segmentation using assistant-modality priors with target-only testing. Its mutual-learning approach draws on knowledge distillation but uses two cross-modality segmentors that guide one another.

Multi-modality Learning in Medical Imaging

  • Prior multi-modality methods fuse complementary features, but commonly require paired modalities at testing; this framework instead uses target-modality data at test time.Other work explored modality-specific branches, parameter sharing, and image translation; this framework keeps separate synthetic-data and real-data branches with mutual guidance.Source: Multi-modality Learning in Medical… · p. 2

Knowledge Distillation

  • Classical knowledge distillation uses a static teacher’s output probabilities to train a student, while mutual learning lets networks learn cooperatively without that fixed teacher setup.The paper applies this cooperative idea to cross-modality knowledge and shape priors.Source: Knowledge Distillation · pp. 2–3
  • At testing, both segmentors receive target-modality input and their predictions are ensembled; during training, they are mutually guided.Figure 1 depicts assistant-to-target translation and knowledge transfer between the synthetic-data and real-data segmentors.Source: Figure 1 · p. 3

Methodology

The framework aligns assistant-modality images with the target modality, then trains two segmentors to exchange modality knowledge through online mutual distillation. Their ensemble supplies the target-modality prediction.

Methodology

  • The framework combines an Image Alignment Module with mutual knowledge distillation, training on labeled assistant- and target-modality samples to improve target-modality segmentation.The alignment module reduces appearance differences, while distillation exploits modality-shared knowledge.Source: Methodology · p. 3

Image Alignment Module

  • The Image Alignment Module translates assistant-modality images toward target appearance using adversarial learning and inverse mapping with cycle consistency, while aiming to preserve structures.The cycle constraint makes reconstructed assistant- and target-modality images similar to their respective inputs; the generator is co-optimized with MKD.Source: Image Alignment Module · pp. 3–4

Mutual Knowledge Distillation

  • Mutual Knowledge Distillation trains synthetic and real segmentors with corresponding annotations and peer-output guidance so each learns knowledge from both modalities.The synthetic segmentor learns from translated images and assistant annotations; the real segmentor learns from real target images and target annotations.Source: Mutual Knowledge Distillation · p. 4

Online Mutual Training

  • Both segmentors are updated online using supervised and distillation losses, and their ensemble prediction is used for target-modality testing.The generator is co-optimized with the synthetic segmentor to preserve shape priors in generated images.Source: Online Mutual Training · pp. 4–5

Experiments

The experiments evaluate MRI assistance for CT cardiac segmentation on MM-WHS 2017, finding that the proposed method outperforms comparison methods and benefits from alignment, mutual distillation, and additional assistant-modality data. Experiments also test the reverse modality direction.

Dataset and Implementation Details

  • Across 20 MRI and 20 CT volumes, two-fold cross-validation trains each fold with all 20 MRI volumes and 10 CT volumes.The dataset includes annotations for seven cardiac substructures; images are cropped, resized to 256 × 256, and segmented with a residual-block U-Net backbone.Source: Dataset and Implementation Details · p. 5

Comparison with other methods.

  • Fine-tuning and joint-training improve mean Dice over baseline by only 0.63% and 0.37%, while Zhang et al. (2018b) exceeds Jiang et al. (2018) by 0.85%.The authors attribute the limited result for Jiang et al. (2018) possibly to offline GAN and segmentor training.Source: Comparison with other methods · pp. 5–6
  • The ensemble achieves the highest Dice across all heart substructures, exceeding Zhang et al. (2018b) by 1.62% in mean Dice.It improves mean Dice over the CT-only baseline by 3.06%.Source: Comparison with other methods · p. 6, Table 1 · p. 6

Qualitative comparison.

  • Our predictions have smoother contours and are closer to ground truth than other methods, while CT-only results misidentify substructures.Fine-tuning and X-shape predictions remain scattered and fragmentary.Source: Figure 2 · p. 6, Qualitative comparison · p. 7

Ablation analysis.

  • Removing IAM lowers mean Dice for Ssyn, Sreal, and Sens by 1.19%, 2.02%, and 1.83%, respectively.Without synthetic-to-real distillation, Sreal and Sens decrease by 2.05% and 1.33%; without real-to-synthetic distillation, Ssyn deteriorates and further harms Sens.Source: Ablation analysis · p. 7, Table 2 · p. 7
  • The ensemble balances inconsistent Ssyn and Sreal predictions by retaining uniform predictions and correcting misleading ones.For disagreements, Sens considers both segmentors’ predictions and produces reliable results.Source: Ablation analysis · p. 7, Figure 4 · p. 7

Effect of different amounts of assistant-modality data.

Extensive experiments with CT as assistant modality.

Conclusion

The paper proposes a multi-modality segmentation framework that uses assistant-modality priors to improve target-modality segmentation. Its experiments on a public cardiac dataset report effectiveness and performance superior to other state-of-the-art methods.

Conclusion

  • The framework uses assistant-modality prior knowledge to enhance segmentation performance on a target modality.It is evaluated on a public multi-modality cardiac segmentation dataset.Source: Conclusion · p. 8
  • The Image Alignment Module generates synthetic target-modality images via adversarial learning to reduce appearance discrepancy.This is intended to facilitate learning modality-shared knowledge.Source: Conclusion · p. 8
  • Mutual Knowledge Distillation pairs explicit learning from each modality’s annotations with implicit exploration of the other modality’s knowledge through peer guidance.Experiments report superior performance against other state-of-the-art methods.Source: Conclusion · p. 8
Loading 2010.01532v1…