PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-Supervised Learning for Medical Image Data with Anatomy-Oriented Imaging Planes

Tianwei Zhang, Dong Wei, Mengmeng Zhu, Shi Gu, Yefeng Zheng

arXiv:2403.16499v2cs.CV

TL;DR

Medical-image pretraining is constrained by limited annotations, and prior self-supervised tasks rarely exploit anatomy-oriented imaging planes. The paper regresses relationships between intersecting and parallel planes, and reports improved transfer performance on heart and knee segmentation and classification compared with recent approaches.

  • Problem

    Medical-image labels are difficult and costly to obtain, while prior self-supervised tasks rarely use anatomy-oriented planes that may be relevant to organ-specific downstream tasks.

    Source: §1 · p. 2

  • Method

    The paper proposes complementary self-supervised tasks that regress intersecting-plane orientation and relative slice locations within parallel-plane stacks.

    Source: §1 · p. 2, §6 · p. 12

  • Results

    The proposed tasks improved transfer performance over other self-supervised approaches on heart and knee imaging for segmentation and classification, especially in low-data settings.

    Source: §1 · p. 2, §6 · pp. 12–13

  • Takeaways & Limitations

    The tasks focus on the anatomy of the imaged structure, which the authors connect to their usefulness for downstream tasks on that structure.

    Source: §6 · p. 13

    Effectiveness was evaluated only for transfer learning between different datasets, because no suitable dataset combined recorded spatial information with annotations for fine-tuning and evaluation.

    Source: §6 · p. 13

Abstract

from arXiv · show

Self-supervised learning has emerged as a powerful tool for pretraining deep networks on unlabeled data, prior to transfer learning of target tasks with limited annotation. The relevance between the pretraining pretext and target tasks is crucial to the success of transfer learning. Various pretext tasks have been proposed to utilize properties of medical image data (e.g., three dimensionality), which are more relevant to medical image analysis than generic ones for natural images. However, previous work rarely paid attention to data with anatomy-oriented imaging planes, e.g., standard cardiac magnetic resonance imaging views. As these imaging planes are defined according to the anatomy of the imaged organ, pretext tasks effectively exploiting this information can pretrain the networks to gain knowledge on the organ of interest. In this work, we propose two complementary pretext tasks for this group of medical image data based on the spatial relationship of the imaging planes. The first is to learn the relative orientation between the imaging planes and implemented as regressing their intersecting lines. The second exploits parallel imaging planes to regress their relative slice locations within a stack. Both pretext tasks are conceptually straightforward and easy to implement, and can be combined in multitask learning for better representation learning. Thorough experiments on two anatomical structures (heart and knee) and representative target tasks (semantic segmentation and classification) demonstrate that the proposed pretext tasks are effective in pretraining deep networks for remarkably boosted performance on the target tasks, and superior to other recent approaches.

1. Introduction

Medical-image pretraining faces limited annotations, while prior self-supervised tasks rarely use anatomy-oriented imaging planes. This work proposes complementary plane-relationship tasks and reports better transfer learning than recent competitors.

1. Introduction

  • Deep networks often need large labeled datasets, which can be difficult and costly to obtain for medical-image analysis.Transfer learning uses pretraining followed by fine-tuning on a target task with limited annotations.Source: §1 · pp. 1–2
  • Anatomy-oriented imaging planes are underused by prior self-supervised tasks, despite their potential relevance to downstream analysis of the imaged organ.These planes are defined relative to organ anatomy, unlike generic pretext tasks that do not exploit that information.Source: §1 · p. 2
  • The proposed tasks use intersecting-plane orientation and parallel-slice location to exploit complementary spatial relationships among anatomy-oriented views.The orientation task regresses intersecting lines; the location task regresses relative slice positions within a stack.Source: §1 · p. 2
  • Experiments on heart and knee imaging and segmentation and classification found the proposed tasks improved transfer learning over recent competitors.The paper also investigated task learnability and combining both tasks in multitask learning.Source: §1 · p. 2

2. Related work

Related work covers transfer learning and self-supervision for natural and medical images, then reviews keypoint detection and multitask self-supervised learning. The paper positions its tasks around anatomy-oriented plane relationships.

2.1. Transfer Learning

  • Transfer learning can address limited target data, but natural-image pretraining may transfer less effectively to medical images because of the domain gap.Large-scale medical-image annotation for supervised pretraining is difficult.Source: §2.1 · p. 2

2.2. SSL for Natural Images

  • Natural-image self-supervised tasks exploit image properties, but the reviewed approaches did not account for anatomy-oriented medical imaging characteristics.Examples include patch-position prediction, jigsaw solving, colorization, rotation prediction, contrastive learning, and masked reconstruction.Source: §2.2 · p. 3

2.3. SSL for Medical Images

  • Prior medical-image tasks used volumetric properties or task-specific cues, while anatomy-oriented intersecting-plane relationships remained comparatively underused.Bai et al. (2019) used square-box segmentation targets along intersecting lines; this work instead regresses a distance-based heatmap.Source: §2.3 · p. 3

2.4. Keypoint Detection

  • The paper extends DNN keypoint detection to line detection by predicting intersecting lines between imaging planes.Keypoint detection generates heatmaps whose peaks indicate keypoint locations.Source: §2.4 · p. 3

2.5. Multi-Task SSL

  • Prior multitask self-supervised learning included natural-image tasks and ColorMe for color scopy images; this work combines orientation and slice-location regression.ColorMe was described as inapplicable to grayscale MRI.Source: §2.5 · p. 3

3. Preliminaries

Anatomy-oriented standard views are prescribed relative to organ anatomy to support comparable measurements. CMR’s SAX and LAX views provide complementary information, and DICOM records spatial attributes that allow their relationships to be computed.

  • Standard views are anatomy-oriented planes used for comparable biometric measurements across populations.Source: §3 · p. 3
  • Fig. 2 depicts straight lines in the mid-ventricular SAX view intersecting the 2C and 4C views, while parallel lines in the 2C and 4C views indicate intersections with the SAX stack.Source: Figure 2 · p. 4
  • CMR’s SAX stack is perpendicular to the LV long axis, while the 2C and 4C LAX views follow it and provide complementary cardiac information.Source: §3 · p. 4
  • DICOM’s IPP and IOP attributes record image position and orientation, enabling computation of spatial relationships between radiographs from the same exam.Source: §3 · p. 4

4. Methods

The methods use spatial information in anatomy-oriented images to define two annotation-free pretext tasks, which can also be combined for multitask learning. The resulting representations are fine-tuned for segmentation or classification.

4.1. Relative Orientation Regression

  • The orientation task trains a network on a SAX image to predict its intersecting lines with the 2C and 4C LAX planes.The target lines are computed from image-plane spatial attributes and represented as Gaussian heatmaps; an L2 loss trains the predictions.Source: §4.1 · pp. 4–5

4.2. Relative Location Regression

  • The location task predicts a slice’s relative position within a parallel stack, using normalized distance or a sine-mapped distance ratio for symmetric anatomy.For the latter case, symmetric slices on opposite sides of the central slice map to similar values; spatial attributes define the target without manual annotation.Source: §4.2 · p. 5

4.3. Multitask SSL

  • Multitask SSL combines the orientation and location losses by summation, with each loss assigned an equal weight of 1.The supervision signals are computed between slices from the same study and scanner.Source: §4.3 · pp. 5–6

4.4. Fine-Tuning on Target Tasks

  • For downstream segmentation, the pretrained encoder and decoder are fine-tuned; for classification, the encoder is fine-tuned with a reinitialized output layer.The evaluation covers semantic segmentation and imaging-based diagnosis with limited annotations.Source: §4.4 · p. 6

5.1. Materials

Experiments use cardiac and knee MRI, representative of medical images with anatomy-oriented imaging planes. Separate datasets provide self-supervised pretraining data and downstream target tasks.

5.1.1. CMR

  • CMR pretraining uses DSBCC exams containing SAX, 2C, and 4C views, with 451 training, 176 validation, and 381 test exams included.The DSBCC data are used only for self-supervised pretraining because they lack segmentation annotations.Source: §5.1.1 · p. 6
  • CMR downstream evaluation uses ACDC for multi-structural segmentation and abnormality diagnosis, with annotations for the LV, RV, and myocardium at end-systolic and end-diastolic phases.The 100 exams are split into 64 training, 16 validation, and 20 test subjects; ACDC lacks spatial information and cannot be used for the proposed pretraining.Source: §5.1.1 · p. 6, §5.1.2 · p. 6

5.1.2. Knee MRI

  • Knee pretraining uses fastMRI sagittal T2-weighted fat-suppressed exams accompanied by axial and coronal views; Stanford data support downstream abnormality and ACL or meniscal tear classification.The Stanford exams are divided into 800 training, 200 validation, and 250 test exams.Source: §5.1.2 · pp. 6–7

5.2. Evaluation Metrics

The section defines separate metrics for pretext-task learning and downstream segmentation and diagnosis. It also notes that the Stanford dataset lacks spatial information needed for the proposed pretraining.

5.2.1. Pretext Tasks

  • Relative-location regression is evaluated with MSE, MAE, Pearson correlation, R2, explained variance, and linear regression analysis; orientation regression is assessed visually against ground-truth heatmaps.Source: §5.2.1 · p. 7

5.2.2. Target Tasks

  • CMR multi-structural segmentation is evaluated volume-wise using Dice and ASSD for the LV, RV, and myocardium, with means across structures for method comparison.Source: §5.2.2 · p. 7
  • Cardiac and knee MRI diagnosis is evaluated with ROC AUC; knee classification covers general abnormalities, ACL tears, and meniscal tears, reporting mean AUC across the three tasks.Source: §5.2.2 · p. 7
  • The Stanford dataset cannot be used for the proposed pretraining because it does not record spatial information.Source: §5.2.2 · p. 7

5.3. Implementation

Implementation uses PyTorch and Adam, with task-specific backbones and training protocols for cardiac and knee MRI. The experiments run on a single NVIDIA TITAN Xp GPU.

  • All experiments use PyTorch, Adam, and a single NVIDIA TITAN Xp GPU for training and inference.Source: §5.3 · p. 7
  • CMR experiments use U-Net; segmentation training runs for 200 epochs with online rotation and scaling, while diagnosis uses a 100-tree random forest on ten features.Source: §5.3 · p. 7
  • Knee MRI diagnosis uses AlexNet with triplicated T2W inputs to use ImageNet-pretrained parameters, followed by pretraining and fine-tuning with online rotation, shift, and horizontal-flip augmentation.Source: §5.3 · p. 7

5.4. Experimental Settings

The experiments assess transfer from self-supervised pretraining to target tasks using varying amounts of target training data. Proposed tasks and established methods are compared under matched backbones and fine-tuning protocols.

  • The evaluation compares training from scratch with fine-tuning after pretraining, using different portions of target-task training data.Source: §5.4 · pp. 7–8
  • The proposed orientation, location, and combined tasks are compared with jigsaw, pairwise slice ordering, Models Genesis, APP, SimCLR, BYOL, and masked autoencoding using matched backbones and protocols.Source: §5.4 · p. 8
  • For the compared methods, pretraining follows the original recipes until convergence, and the same fine-tuning protocols are applied.Source: §5.4 · p. 8
  • ImageNet-pretrained SimCLR and BYOL ResNet-50 and MA ViT-Base models are included as references, using segmentation networks for CMR and pretrained backbones directly for knee diagnosis.Source: §5.4 · p. 8

5.5. Performance on Proposed Pretext Tasks

The proposed orientation task yields predicted heatmaps similar to ground truth across cardiac and knee imaging examples. Relative-location regression performs well on both datasets, and its metrics improve when combined with orientation regression.

5.5.1. Relative Orientation Regression

  • Predicted orientation heatmaps are consistently similar to ground truth across cardiac phases, slice locations, and health conditions, suggesting learned anatomical landmarks that define the imaging planes.Source: §5.5.1 · p. 8, Figure 5 · p. 9

5.5.2. Relative Location Regression

  • Linear regression analyses of predicted and ground-truth slice locations for cardiac and knee MTL are reported in Fig. 6 and accord with the Table 2 results.Source: Figure 6 · p. 9, §5.5.2 · p. 10
  • Combining relative-location and orientation regression improves all evaluated location-regression metrics on both datasets.The reported single-task metrics are described as fairly good: MSE and MAE are around 0.01 and 0.1, Pearson coefficients exceed 0.9, and R2 and explained variance exceed 0.8.Source: §5.5.2 · p. 9, Table 2 · p. 9
  • Fig. 7 visualizes multi-structural SAX CMR segmentation results corresponding to Table 4, with more challenging slices shown when more fine-tuning subjects are available.Rows 1–2 show end-systole and rows 3–5 show end-diastole; red, blue, and green denote RV, LV, and myocardium.Source: Figure 7 · p. 10

5.6. Transfer Learning Performance on Target Tasks

Across cardiac and knee MRI tasks, the proposed pretraining generally improved transfer performance, with multitask learning often strongest; the most effective single pretext task differed by target task.

5.6.1. CMR Semantic Segmentation and Diagnosis

  • With four fine-tuning subjects, MTL reached mean Dice 0.814 and mean ASSD 1.917 mm, comparable to scratch with 16 subjects (0.815; 2.064 mm).With 32 subjects, MTL also outperformed training from scratch with twice as much data.Source: §5.6.1 · p. 10, Table 4 · p. 11
  • MTL produced the greatest improvements in most CMR segmentation settings across both metrics and structures; relative orientation was slightly more effective than relative location.Representative segmentations from MTL were closest to ground truth, consistent with the quantitative evaluation.Source: §5.6.1 · p. 10, Table 4 · p. 11
  • Directly transferred ImageNet-pretrained contrastive and related models generally underperformed their in-domain pretrained counterparts, though the backbones differed.The authors conclude that domain-specific pretraining is more effective for medical images with significant domain gaps from natural images.Source: §5.6.1 · p. 10
  • MTL achieved the highest AUCs across all fine-tuning data amounts for myocardial infarction and other diagnoses on ACDC.Almost all compared pretext tasks improved on scratch, except contrastive approaches with few fine-tuning subjects.Source: Table 3 · p. 10, §5.6.1 · p. 12

5.6.2. Knee MRI Diagnosis

  • ∼0.07 and ∼0.05 higher mean AUC than ImageNet with 80 and 200 knee subjects, respectively; MTL remained comparable at 400 or more.Relative location regression consistently outperformed relative orientation regression for knee diagnosis.Source: §5.6.2 · p. 12

6. Discussion and conclusion

The proposed anatomy-oriented pretext tasks were effective for transfer learning across cardiac and knee MRI tasks, and combining them often improved performance. Their evidence is limited to transfer learning, and their applicability is restricted to multi-view data with anatomy-oriented planes.

6. Discussion and conclusion

  • Across cardiac and knee MRI, the two pretext tasks were feasible to learn and effective for segmentation and diagnosis, outperforming existing self-supervised pretext tasks.The tasks regress relative view orientations and locations.Source: §6 · p. 12
  • The tasks were simple to implement, and the location-regression metrics and stable intersecting-line predictions supported their learnability across varying conditions.The intersecting lines were stable across slice locations, cardiac phases for CMR, and pathological status.Source: §6 · p. 12
  • Combining orientation and location regression improved on either task alone, consistent with the authors’ view that the tasks complement rather than conflict with each other.The authors conjecture that robust representations and reduced overfitting may contribute to the gains.Source: §6 · p. 12
  • Relative orientation worked better for CMR segmentation, whereas relative location worked better for knee classification; the authors conjecture this reflects task-specific pretext–target proximity.They also suggest knee anatomy may make orientation prediction easier, potentially reducing its pretraining efficacy there.Source: §6 · p. 13
  • Pairwise slice ordering can struggle with similar slices symmetric about a center; the proposed sine-based mapping assigns such slices similar regression values.The authors suggest that learning a more general mapping from image data remains possible.Source: §6 · p. 13
  • Evaluation covered transfer learning only, with pretraining and downstream evaluation on different datasets; same-dataset pretraining was not tested.The authors expect greater impact when target data can also support pretraining, provided spatial information is properly recorded.Source: §6 · p. 13
  • The proposed tasks apply specifically to multi-view medical images with anatomy-oriented planes, a scope that includes MRI of various organs and standard mammography views.The authors identify lesion detection as a future application to explore.Source: §6 · p. 13
  • The study focused on pretraining and used straightforward fine-tuning configurations; the authors expect advanced pipelines such as nnU-Net could benefit from the pretrained weights.This focus was intended to emphasize the effect of pretraining.Source: §6 · p. 13
Loading 2403.16499v2…