PaperlayerView exact revision on arXiv

Source-linked AI summary

Multimodal Self-Supervised Learning for Medical Image Analysis

Aiham Taleb, Christoph Lippert, Tassilo Klein, Moin Nabi

arXiv:1912.05396v2cs.CVcs.LG

TL;DR

Medical image analysis faces costly, scarce annotations, while self-supervised methods can miss complementary information across modalities. This paper trains multimodal puzzle solvers, using cross-modal synthetic images for pretraining, and reports competitive benchmark performance and gains over single-modal puzzles.

  • Problem

    Expert annotation of multimodal patient data is costly and difficult to produce at scale, and representations can benefit from complementary information across modalities.

    Source: §I · pp. 1–2

  • Method

    The method mixes patches from multiple modalities to train a modality-agnostic puzzle solver and uses cross-modal generation to augment self-supervised pretraining.

    Source: §I · p. 2

  • Results

    The multimodal puzzles outperform their single-modal counterparts and achieve results competitive with state of the art on three medical-imaging benchmarks.

    Source: §V · p. 12

  • Takeaways & Limitations

    The learned representations support downstream medical-imaging tasks, including settings with unregistered modalities, while reducing required manual annotation.

    Source: B. Puzzle-Solving with Sinkhorn Networks · p. 5, §V · p. 12

Abstract

from arXiv · show

Self-supervised learning approaches leverage unlabeled samples to acquire generic knowledge about different concepts, hence allowing for annotation-efficient downstream task learning. In this paper, we propose a novel self-supervised method that leverages multiple imaging modalities. We introduce the multimodal puzzle task, which facilitates rich representation learning from multiple image modalities. The learned representations allow for subsequent fine-tuning on different downstream tasks. To achieve that, we learn a modality-agnostic feature embedding by confusing image modalities at the data-level. Together with the Sinkhorn operator, with which we formulate the puzzle solving optimization as permutation matrix inference instead of classification, they allow for efficient solving of multimodal puzzles with varying levels of complexity. In addition, we also propose to utilize cross-modal generation techniques for multimodal data augmentation used for training self-supervised tasks. In other words, we exploit synthetic images for self-supervised pretraining, instead of downstream tasks directly, in order to circumvent quality issues associated with synthetic images, while improving data-efficiency and representations quality. Our experimental results, which assess the gains in downstream performance and data-efficiency, show that solving our multimodal puzzles yields better semantic representations, compared to treating each modality independently. Our results also highlight the benefits of exploiting synthetic images for self-supervised pretraining. We showcase our approach on four downstream tasks: Brain tumor segmentation and survival days prediction using four MRI modalities, Prostate segmentation using two MRI modalities, and Liver segmentation using unregistered CT and MRI modalities. We outperform many previous solutions, and achieve results competitive to state-of-the-art.

I. INTRODUCTION

Medical imaging relies on multiple modalities, but annotation is costly and self-supervised methods described here do not account for multimodality. The paper introduces multimodal puzzle solving and cross-modal generation for self-supervised pretraining.

I. INTRODUCTION

  • Expert annotations of multimodal patient data are expensive, time-consuming, difficult to produce at scale, and associated with privacy risks.The introduction identifies annotation scarcity as a major impediment to medical-imaging machine learning.Source: §I · p. 1
  • The multimodal puzzle task mixes patches from multiple modalities to learn modality-agnostic representations, including when modalities differ substantially or are non-registered.The paper also reports rich representations from non-registered modalities.Source: §I · p. 2
  • Cross-modal generation augments self-supervised training with synthetic images rather than using them directly for downstream tasks, addressing synthetic-image quality concerns.The authors report gains particularly in low-data regimes.Source: §I · p. 2
  • The paper extends earlier brain-tumor segmentation experiments with cross-modal generation and evaluations of prostate and liver segmentation.The liver task uses CT and MRI modalities and demonstrates puzzle learning with unregistered inputs.Source: §I · p. 2

II. RELATED WORK

Prior work uses proxy tasks and image context for self-supervised representations, but the reviewed approaches differ in modality handling, computational demands, and data assumptions. This work positions multimodal, data-level puzzle fusion as an alternative.

II. RELATED WORK

  • Self-supervised learning constructs representations through proxy tasks, including spatial-context prediction and other image-derived signals.The paper follows spatial-context supervision but derives context across multiple imaging modalities.Source: §II · pp. 2–3
  • Noroozi et al.’s single-modality Jigsaw method uses a fixed permutation set and substantial compute, while Sinkhorn-based permutation inference supports more complex puzzles.The puzzle objective is framed as permutation-matrix inference rather than classification over a selected set.Source: §II · p. 3
  • Prior domain-adaptation and multimodal approaches fuse modalities at the feature level, whereas this work fuses imaging modalities at the data level to handle large modality differences.The reviewed domain-adaptation methods use joint tasks across domains to encourage domain-agnostic features.Source: §II · p. 3
  • Medical self-supervised methods often make input assumptions, while the proposed puzzles are reported to work across modalities even when spatially unregistered.The reviewed 3D jigsaw approach is limited to 2×2×2 puzzles and is reported as less computationally efficient than the multimodal method.Source: §II · pp. 3–4
  • Cross-modal translation has been used to augment downstream tasks, but this work instead uses generated images for pretraining because synthetic-image quality may be uncertain.The reviewed approach can also address settings with few multimodal samples and imbalanced modality availability.Source: §II · p. 4

III. METHOD

The method mixes patches from multiple imaging modalities to train a modality-agnostic representation for downstream tasks. A Sinkhorn-based solver makes the resulting puzzle task tractable by inferring soft permutations.

A. Multimodal Puzzle Construction

  • Multimodal puzzles mix patches from several modalities, jointly learning organ composition and cross-modal spatial relationships without prior knowledge of modality type.The method assumes modalities may vary across downstream tasks.Source: A. Multimodal Puzzle Construction · p. 4, §III · p. 4
  • The solution space grows as O(C!M), so the method uses Sinkhorn for the permutation factor and a cross-modal network to cancel the modality factor.The paper identifies factorial growth in puzzle permutations and exponential growth in the number of modalities.Source: A. Multimodal Puzzle Construction · p. 4

B. Puzzle-Solving with Sinkhorn Networks

  • A network processes each puzzle patch independently, concatenates its feature vectors into an N × N matrix, and applies Sinkhorn to infer a soft permutation.The permutation is applied to the scrambled patches to reconstruct the image.Source: B. Puzzle-Solving with Sinkhorn Networks · pp. 4–5
  • The solver minimizes mean squared error between the sorted ground truth and reconstructed puzzle, yielding representations for annotation-efficient downstream fine-tuning.The paper depicts this construction using randomly drawn patches from the available modalities.Source: B. Puzzle-Solving with Sinkhorn Networks · p. 5, Figure 2 · p. 5

C. Cross-Modal Generation

Cross-modal generation augments multimodal puzzle training when paired scans are scarce. Experiments report benefits from mixing synthetic and realistic data and examine downstream performance, data efficiency, and puzzle-design variables.

C. Cross-Modal Generation

  • CycleGAN translates between imaging modalities and uses cycle consistency to relax the need for paired, registered scans.The translation step is intended to address settings where large quantities of multimodal data are difficult to obtain.Source: C. Cross-Modal Generation · p. 5
  • Puzzles mix real multimodal images with generated samples from the smaller modality; experiments report better representations than puzzles made from a single modality.The method can also operate on real multimodal images alone; synthetic-image mixtures are evaluated separately.Source: C. Cross-Modal Generation · p. 5

IV. EXPERIMENTAL RESULTS

  • The experiments assess generated-data integration, downstream data efficiency, and the effects of puzzle complexity and permutation-list size.The paper transfers and fine-tunes learned representations on downstream tasks to measure their impact.Source: §IV · p. 5

A. Datasets

The paper evaluates multimodal pretraining on three medical-imaging benchmarks and reports improved BraTS segmentation performance against several baselines. The section also describes the datasets, segmentation targets, and evaluation setup.

A. Datasets

  • BraTS has 285 training and 66 validation cases with four MRI modalities; prostate has 32/16 train/test cases; CHAOS has 20/20 and unregistered CT/MRI, evaluated only for liver.The prostate dataset contains 48 cases, and CHAOS contains 40.Source: A. Datasets · p. 6

1) Brain Tumor Segmentation:

  • BraTS segmentation targets whole tumor, tumor core, and enhanced tumor, with average Dice reported for each region.The paper notes that different tumor regions may appear more clearly in different MRI modalities.Source: 1) Brain Tumor Segmentation · pp. 6–7
  • The comparison includes from-scratch and single-modal baselines; five-fold cross-validation uses matched train/test folds and one-sided two-sample t-tests on Dice scores.The single-modal comparison selects the best modality for each task.Source: 1) Brain Tumor Segmentation · p. 6
  • Multimodal pretraining outperforms scratch, single-modal, JiGen, 2D Models Genesis, 2D Rubik Cube, and 2D Isensee et al. on BraTS segmentation, while achieving results comparable to literature.The authors report outperforming Chang et al. and Li on all reported Dice scores, and outperforming 2D Isensee et al. despite its additional data and augmentation.Source: Table I · p. 7, 1) Brain Tumor Segmentation · p. 7

2) Prostate Segmentation:

For prostate segmentation, the proposed multimodal approach outperforms several pretraining and puzzle-solving baselines. The authors relate its larger performance gap over JiGen to the greater appearance differences between prostate MRI modalities.

2) Prostate Segmentation:

  • Evaluation reports Dice and normalized surface distance for the central and peripheral prostate regions.The study uses five-fold cross-validation and one-sided two-sample t-tests on Dice scores.Source: 2) Prostate Segmentation · p. 7
  • The authors attribute the larger gap over JiGen than in brain tumor segmentation to greater appearance differences between prostate imaging modalities.They contrast prostate modalities with BraTS sequences, which are variants of T1- and T2-weighted scans.Source: 2) Prostate Segmentation · p. 7
  • On prostate segmentation, the proposed method outperforms scratch, single-modal, 2D Models Genesis, JiGen, and 2D Rubik Cube; its Sinkhorn solver also beats the U-Net puzzle solver.The results are reported in Table II.Source: 2) Prostate Segmentation · pp. 7–8, Table II · p. 8

3) Liver Segmentation:

On liver segmentation, multimodal puzzle pretraining outperforms the from-scratch and single-modal baselines and achieves results nearly on par with using registered modalities. The Sinkhorn-based solver also outperforms a U-Net puzzle solver.

3) Liver Segmentation:

  • The liver segmentation results are reported in Table III using average Dice, the metric used by the official challenge.The supplied passages do not provide the table's numerical values.Source: Table III · p. 8, 3) Liver Segmentation · p. 8
  • The multimodal method outperforms both from-scratch and single-modal baselines, and its results are almost on par with the registered-data baseline.The registered baseline uses VoxelMorph to assess the influence of registration.Source: 3) Liver Segmentation · p. 8
  • The Sinkhorn-based puzzle solver outperforms the U-Net puzzle solver, while the method also outperforms 2D Rubik Cube on this task.The U-Net baseline uses pixel-wise image reconstruction instead of the Sinkhorn operator.Source: 3) Liver Segmentation · p. 8

4) Survival Days Prediction (Regression):

The paper fine-tunes its puzzle-solver model for BraTS survival-days regression and reports better results than the compared baselines. The evaluation uses mean squared error.

4) Survival Days Prediction (Regression):

C. Cross-Modal Generation Results

Cross-modal generation augments multimodal puzzle pretraining when paired samples are scarce, and the reported experiments show gains over single-modal puzzle solving. The section also examines low-shot performance and puzzle complexity.

C. Cross-Modal Generation Results

  • Training the puzzle solver on synthetic and realistic multimodal data improves results on all benchmarks over the single-modal puzzle-solving baseline.Even with 1% of total dataset sizes, the generator appears to capture important characteristics; the results also support using synthetic images for self-supervised pretraining rather than direct downstream training.Source: C. Cross-Modal Generation Results · p. 9

D. Low-Shot Learning Results

  • With few labeled samples, the multimodal model outperforms both from-scratch and single-modal baselines, including at 1% of the segmentation training set.The reported low-shot experiments use realistic multimodal data; the gap to the from-scratch baseline appears larger at 1%.Source: D. Low-Shot Learning Results · p. 9

E. Ablation Study

1) Puzzle Complexity:

  • The Sinkhorn solver formulates puzzle solving as permutation-matrix inference rather than permutation classification, a design intended to reduce sensitivity to permutation-list size.Source: 2) Permutation Set Size · p. 9

2) Permutation Set Size:

Increasing puzzle complexity and permutation-set size improves downstream performance up to a point, after which gains become negligible. The conclusion links multimodal puzzles to semantic representations and annotation efficiency.

2) Permutation Set Size:

  • The qualitative generation figure compares scans produced using 1%, 10%, and 50% multimodal subsets across three benchmark-specific modality conversions.The conversions shown are T2 to FLAIR for BraTS, T2 to ADC for Prostate, and CT to MR-T2 for CHAOS liver.Source: Figure 4 · p. 10
  • In the low-shot scenario, the method outperforms both single-modal and from-scratch baselines.Source: Figure 5 · p. 11
  • Downstream performance improves as puzzle complexity and permutation-set size increase, then changes negligibly beyond a limit.The plots show the trends; the saturation points differ across tasks and the model capacity may contribute to saturation.Source: 2) Permutation Set Size · p. 11, Figure 6 · p. 11
  • Small permutation sets, such as 100 or 1000, can harm learned representations, while larger sets help until their influence becomes negligible.The permutation-set experiment uses 3-by-3 puzzles and varies the set size from 10^2 to 3 × 10^5, nearly 9!.Source: 2) Permutation Set Size · p. 11

V. CONCLUSION & FUTURE WORK

  • The conclusion reports competitive results on three medical imaging benchmarks, including one with unregistered modalities, and lower annotation requirements in the low-data regime.The authors identify extending the approach to 3D as future work.Source: §V · p. 12

APPENDIX I MODEL TRAINING FOR ALL TASKS

The appendix describes preprocessing, puzzle-solver training, and network configurations across tasks. It also explains how pretrained inputs are adapted to downstream tasks with different modality and channel requirements.

APPENDIX I MODEL TRAINING FOR ALL TASKS

  • Scans are converted to axial 2D slices, resized to 128 × 128 for BraTS or 256 × 256 for Prostate and CHAOS, then intensity-normalized to [0, 1].Source: Appendix I · p. 14
  • Training uses Adam, with learning rates 0.001 for puzzle solving, 0.0002 for cross-modal generation, and 0.00001 for segmentation and regression; training lasts 500, 200, and 50 epochs, respectively.The appendix also specifies task-dependent Gaussian initialization and an L2 regularizer with λ = 0.1 for puzzle solving and downstream tasks.Source: Appendix I · pp. 14–15
  • The puzzle solver uses five convolutional layers, a fully connected layer, and Sinkhorn; downstream U-Net encoders are pretrained while decoders are randomly initialized.Segmentation combines weighted crossentropy and dice loss equally; cross-modal generators follow CycleGAN, with Johnson et al. generators and 70×70 PatchGAN discriminators.Source: Appendix I · p. 15
  • For downstream inputs with different channel counts, pretrained input-layer weights are duplicated; BraTS survival uses four MRI channels plus a tumor mask from the best segmentation model.Duplicating the input-layer weights adds a few parameters and retains their encoded information for channel fusion.Source: Appendix I · p. 15
  • BraTS segmentation stacks four MRI modalities and prostate segmentation stacks two; unregistered liver CT and MR-T2 are processed as separate single-slice inputs using Joint Learning.Puzzle-pretrained models are described as learning to disregard modality type during training and testing.Source: Appendix I · p. 15
  • A random 5-pixel jitter on each side of sampled patches discourages shortcut solving and is intended to promote better representations.Source: Appendix I · p. 15
  • Puzzle-solver representations capture tissue structures across modalities and can be transferred to downstream tasks by fine-tuning on target domains.The appendix describes this as a consequence of multimodal puzzle solving.Source: Appendix I · p. 15
Loading 1912.05396v2…