PaperlayerView exact revision on arXiv

Source-linked AI summary

Disentangled Representation Learning in Cardiac Image Analysis

Agisilaos Chartsias, Thomas Joyce, Giorgos Papanastasiou, Michelle Williams, David Newby, Rohan Dharmakumar, Sotirios A. Tsaftaris

arXiv:1903.09467v4cs.CV

TL;DR

Medical images combine spatial anatomy with imaging-specific characteristics, motivating a representation that separates these factors for cardiac analysis. SDNet learns categorical spatial anatomy and non-spatial modality factors, and demonstrates utility across limited-label segmentation, auxiliary-task learning, multimodal data, and image synthesis.

  • Problem

    Medical-image representations must capture anatomy and imaging characteristics while supporting tasks across modalities and with limited labeled data.

  • Method

    SDNet factorises 2D medical images into a categorical spatial anatomical representation and a VAE-based non-spatial modality factor, combining them for reconstruction and downstream tasks.

  • Results

    Across semi-supervised segmentation, auxiliary anatomical learning, multimodal learning, and synthesis, SDNet maintains useful performance with few labels and can synthesize MR and CT images by factor manipulation.

  • Takeaways & Limitations

    The factorised representation supports modality-independent anatomical tasks, multimodal learning, latent-space manipulation, and interpretable probing of medical-image models.

  • Takeaways & Limitations

    The authors identify extending the method to 3D, hierarchical factors, and higher-fidelity texture modeling as future directions, while preserving spatial semantics remains an open challenge.

Abstract

from arXiv · show

Typically, a medical image offers spatial information on the anatomy (and pathology) modulated by imaging specific characteristics. Many imaging modalities including Magnetic Resonance Imaging (MRI) and Computed Tomography (CT) can be interpreted in this way. We can venture further and consider that a medical image naturally factors into some spatial factors depicting anatomy and factors that denote the imaging characteristics. Here, we explicitly learn this decomposed (disentangled) representation of imaging data, focusing in particular on cardiac images. We propose Spatial Decomposition Network (SDNet), which factorises 2D medical images into spatial anatomical factors and non-spatial modality factors. We demonstrate that this high-level representation is ideally suited for several medical image analysis tasks, such as semi-supervised segmentation, multi-task segmentation and regression, and image-to-image synthesis. Specifically, we show that our model can match the performance of fully supervised segmentation models, using only a fraction of the labelled images. Critically, we show that our factorised representation also benefits from supervision obtained either when we use auxiliary tasks to train the model in a multi-task setting (e.g. regressing to known cardiac indices), or when aggregating multimodal data from different sources (e.g. pooling together MRI and CT data). To explore the properties of the learned factorisation, we perform latent-space arithmetic and show that we can synthesise CT from MR and vice versa, by swapping the modality factors. We also demonstrate that the factor holding image specific information can be used to predict the input modality with high accuracy. Code will be made available at https://github.com/agis85/anatomy_modality_decomposition.

1 Introduction

The paper frames medical images as decomposable into spatial anatomy and imaging-specific factors, then proposes SDNet to learn this factorisation for cardiac analysis. Its categorical spatial representation and VAE-based modality representation support segmentation, auxiliary anatomical tasks, multimodal learning, and synthesis.

  • Motivation: Medical-image representations must support useful tasks across anatomical content and multiple imaging modalities.The paper identifies this as a central representation-learning challenge in medical imaging.
  • Proposed approach: SDNet factorises images into a spatial anatomical map and a non-spatial modality vector.The anatomy is represented across categorical channels, while modality-specific intensity information is encoded in a smooth latent space.
  • Proposed approach: Categorical spatial channels encourage modality-agnostic anatomy factors by assigning each pixel to exactly one anatomical region.This restriction also promotes meaningful topological regions and prevents modality information from being encoded spatially.
  • Proposed approach: A VAE-based modality representation enables sampling, latent-space arithmetic, and synthetic image generation.The two factors can be recombined to reconstruct images, while the modality representation supports generation of new images.
  • Evaluation and contributions: The method is evaluated on semi-supervised segmentation, auxiliary anatomical regression, multimodal learning, and synthesis-related capabilities.Reported contributions include performance with few labels, LVV inference and multi-task learning, and use of MR and CT data together.

2 Related work

Related work spans disentangled and style-content representations, semi-supervised medical segmentation, and cardiac segmentation. The paper distinguishes SDNet by making the spatial factor categorical and anatomically semantic, then leveraging it for semi-supervised and auxiliary-task learning.

  • Disentangled representations: Prior disentanglement methods commonly learn independent factors with VAEs, GANs, or factor-mixing techniques, often as latent vectors.Related medical applications include modality-specific and shared features, but not necessarily semantically meaningful spatial factors.
  • Style and content disentanglement: Style-content methods separate spatial content from non-spatial style, but medical imaging requires a categorical spatial factor for interpretable anatomy.SDNet uses this discretisation to prevent the spatial representation from becoming tied to a particular modality.
  • Semi-supervised segmentation: Semi-supervised cardiac segmentation is motivated by costly manual labeling and limited availability of multiply annotated images.Existing semi-supervised methods cited in the paper do not learn disentangled representations of the data.
  • Position of this work: SDNet focuses on meaningful spatial representations for improved semi-supervised segmentation and auxiliary anatomical tasks.Its application differs from conventional fully supervised multi-class cardiac segmentation approaches.

3 Materials and methods

SDNet decomposes cardiac images into a spatial anatomical representation and a non-spatial modality representation, then reconstructs images while supporting auxiliary segmentation supervision. Binary, categorical spatial channels and modality-factor reconstruction are used to encourage disentanglement.

  • Model architecture: SDNet encodes each 2D image into a multi-channel spatial map and a modality latent vector, which a decoder combines to reconstruct the input.The anatomy and modality representations are produced by dedicated encoders, with auxiliary tasks optionally supplying supervision.
  • Anatomy representation: The anatomy encoder uses a fully convolutional U-Net to map images into spatial representations with skip connections.These connections fuse local and non-local information across downsampling and upsampling paths.
  • Anatomy representation: Each spatial-map pixel belongs to exactly one binary channel, while channels represent cardiac substructures or remaining anatomy needed for reconstruction.The representation can expose structures such as the myocardium and left and right ventricles.
  • Anatomy representation: Thresholding converts softmax spatial activations into binary maps during the forward pass while bypassing the step function during back-propagation.The thresholding reduces spatial-factor capacity and encourages anatomical information to occupy distinct channels rather than encode modality variation.
  • Modality representation: The modality encoder models q(z|x, s) with a stochastic convolutional network that predicts Gaussian parameters and samples the latent factor z.The latent distribution follows the VAE formulation, with a unit-Gaussian prior and reconstruction-based training.
  • Auxiliary supervision: A separate fully convolutional segmentor refines the spatial maps into segmentation masks, improving training stability and enabling the same anatomy representation to support multiple tasks.The paper specifically connects this representation to segmentation and calculation of the left-ventricular volume.
  • Modality representation: The z-reconstruction cost discourages posterior collapse by requiring images generated from sampled modality factors to reproduce the same modality factor when re-encoded.This preserves information in latent dimensions that the KL objective alone could otherwise ignore.

4 Experimental setup

The experiments use four cardiac-image datasets and compare SDNet with supervised and semi-supervised segmentation baselines under consistent training procedures. The setup includes multimodal data, modality-factor constraints, and a weighted composite objective.

  • Datasets: Experiments use 2D images from four datasets normalized to the range [-1, 1].The datasets include cardiac MR, CT/CTA, and canine cine-MR and CP-BOLD acquisitions.
  • Datasets: The ACDC dataset provides 1,920 manually segmented images and 23,530 unsegmented images from 100 patients for semi-supervised segmentation and latent-space arithmetic.Images cover end-systolic and end-diastolic phases with LV, myocardium, and RV masks.
  • Datasets: The MM-WHS data contain 40 volumes comprising 20 cardiac CT/CTA and 20 cardiac MRI volumes for multimodal segmentation, modality transformation, and modality estimation.The resulting image collection includes 3,626 MR and 2,580 CT images with manual segmentations.
  • Training objective: The overall training objective combines KL, supervised segmentation, adversarial, image-reconstruction, and modality-factor-reconstruction costs as L = λ1LKL + λ2Lsegm + λ3Ladv + λ4Lrec + λ5Lzrec.The segmentation and adversarial weights are both set to 10, while the KL, reconstruction, and z-reconstruction weights are 0.01, 1, and 1.
  • Model configuration: The modality factor dimension is nz=8, while the spatial factor uses C=8 channels for ACDC and QMRI and C=16 for MM-WHS.The larger MM-WHS spatial factor accommodates more segmented regions and different MR/CT contrasts and viewpoints.

5 Results and discussion

The paper evaluates SDNet across semi-supervised segmentation, multi-task learning, multimodal segmentation, modality estimation, and latent-space arithmetic. These experiments test the factorisation across limited-label, auxiliary-task, and multimodal settings.

  • Evaluation program: SDNet is evaluated in semi-supervised segmentation, regression-assisted multi-task learning, concurrent MR/CT segmentation, modality estimation, and latent-space arithmetic.The experimental program examines both task performance and whether the learned factors capture anatomy and imaging modality.

5.1 Semi-supervised segmentation

The semi-supervised experiments test whether SDNet maintains segmentation accuracy when labels are reduced to images from a small number of subjects. SDNet remains accurate with few labels, while reconstruction further improves performance over adversarial training alone and supervision-heavy baselines.

  • Experimental design: The study varies the number of subject-level labelled images while fixing an additional pool of 1,200 unlabelled images.This sampling design avoids treating individual images as independent labelled examples and excludes ES and ED phases from the unlabelled pool.
  • Datasets and evaluation: The semi-supervised evaluation uses ACDC masks for LV, myocardium, and RV and QMRI masks for LV and myocardium.ACDC inputs receive rotations up to 90°, whereas QMRI inputs retain their canonical orientation without augmentation.
  • Results: 73%, 77%, and 78% Dice are obtained with 1.5%, 3%, and 6% labelled data, respectively, with significant paired-test improvements at p-values 0.0006, 0.02, and 0.002.The reported SDNet results are attributed to adversarial training plus the additional reconstruction cost.
  • Results: Below 12.5% annotations, supervised U-Net and self-training deteriorate, falling below 55% Dice when labels come from one or two subjects.With many annotations, all methods perform similarly, while SDNet has the lowest variance.
  • Comparison with prior work: SDNet significantly outperforms the earlier model across the evaluated ACDC label fractions, including 75 ± 9% at 6%, 66 ± 15% at 3%, and 59 ± 13% at 1.5%.The comparison uses paired t-tests at the 5% significance level.
  • Results: On QMRI, SDNet and GAN retain 75% and 79% performance, respectively, when training uses annotations from one subject, while supervised performance falls by almost 50%.The paper reports no significant difference between SDNet and GAN in this setting.

5.2 Left ventricular volume

The model uses its spatial anatomical representation to predict left ventricular volume as an auxiliary task and to improve segmentation with limited annotations.

  • LVV estimation: Left ventricular volume is computed by summing cavity pixels per slice, converting each sum to area, multiplying by slice thickness, and summing slice volumes.
  • LVV estimation: SDNet predicts the summed left-ventricle pixels from its spatial representation, after which LVV is calculated offline.
  • LVV estimation: 138.57mL predicted LVV closely matched 139.23mL ground truth, with no significant paired-test difference.
  • Multi-task segmentation: 83.2% segmentation accuracy after multi-task fine-tuning exceeded 75.6% before fine-tuning, a statistically significant increase.

5.3 Multimodal learning

SDNet separates shared spatial anatomy from modality-specific information, enabling multimodal MR-CT training and cross-modal image synthesis.

  • Multimodal representation: The spatial factor captures common anatomical information across MR and CT, while the non-spatial factor captures modality-specific intensity information.
  • Multimodal segmentation: MR and CT training together improved segmentation on both modalities, including when only 12.5% of one modality was added to 100% of the other.
  • Multimodal segmentation: With fixed total training data, CT Dice ranged from 77% at 100% data to 65% at 12.5%, whereas CT-only training at 12.5% achieved 23%.
  • Modality transformation: Combining a modality vector with a fixed anatomy factor synthesised CT from MR and MR from CT, producing domain-typical intensity characteristics.

5.4 Modality type estimation

The study tests whether the learned modality factor contains imaging-specific information by classifying MR versus CT and cine-MR versus CP-BOLD MR.

  • MR versus CT: A logistic regressor classified held-out MR versus CT images from the modality factor with 92% accuracy.
  • MR versus CT: The fifth modality-factor dimension alone reached 82% accuracy, while the remaining dimensions ranged from 42% to 66%.
  • Interpretation: The modality factor captures global intensity differences between MR and CT that affect the entire image.
  • Cine-MR versus CP-BOLD MR: The authors also evaluate modality classification between cine-MR and CP-BOLD MR, where contrasts are more subtle and spatially or temporally localized.

5.5 Latent space arithmetic

SDNet’s latent-space arithmetic tests show that spatial channels control anatomical-region intensities, while modality-factor dimensions primarily control global image characteristics with some local effects.

  • Spatial factor arithmetic: Rearranging spatial channels changes the reconstructed intensities associated with anatomical regions while keeping the modality factor fixed.Moving MYO content into the LV channel equalizes their intensities; exchanging the channels reverses them, and random shuffling alters the reconstruction.
  • Modality factor arithmetic: Interpolating each of eight modality-factor dimensions over seven values from −3 to 3 generates controlled reconstruction grids.Each grid row varies one z dimension while holding the others fixed.
  • Modality factor arithmetic: Correlation and difference images measure which pixels change as each modality-factor dimension varies.Pixelwise correlation uses Pearson correlation, while the difference image subtracts the −3 reconstruction from the 3 reconstruction.
  • Modality factor arithmetic: Most z dimensions show broad positive or negative correlations across the image, indicating that z primarily captures global image characteristics.Some dimensions also show localized effects involving the heart, right ventricle, or myocardium.

5.6 Factor sizes

The factor sizes determine model capacity: larger spatial capacity helps multimodal data, whereas overly large modality factors reduce segmentation performance.

  • Modality factor size: Figure 8 varies one z dimension per row across seven values and reports correlation and image-difference effects for each variation.
  • Spatial factor size: When C > 8, many spatial channels become zero, but C = 16 helps multimodal MR/CT experiments capture shared and modality-specific information.The additional channels separate information common to both modalities from information unique to each.
  • Spatial factor size: C < 4 provides insufficient capacity; an SDNet with C = 4 achieved Dice 68.1 ± 8% despite using 100% of the labels.
  • Modality factor size: nz = 4, 32, and 128 produced Dice scores of 84 ± 5%, 83 ± 6%, and 82 ± 6%, respectively, compared with 84% for nz = 8.The nz = 128 result was worse and statistically significant, whereas nz = 4 and nz = 32 were similar to nz = 8.

6 Conclusion

The paper presents SDNet as a generative disentanglement method that separates semantic spatial anatomy from non-spatial modality information. It demonstrates utility for reduced-label segmentation, auxiliary cardiac regression, multimodal synthesis, and interpretable latent-space manipulation, while identifying extensions to 3D, richer factors, improved texture fidelity, and unsupervised learning.

  • Method and contribution: SDNet disentangles medical images into a semantically meaningful spatial anatomical factor and a non-spatial modality factor.Its variational-autoencoder formulation also models intensity variability generatively.
  • Applications: The factorization achieves high-accuracy semi-supervised segmentation with substantially fewer labelled images and supports cardiac functional-index prediction in multi-task training.
  • Applications: SDNet uses factorized representations to synthesize MR and CT images from one another using small fractions of the respective input images.
  • Significance: The meaningful spatial/non-spatial factorization facilitates latent-space manipulation and model probing, supporting interpretability relevant to clinical translation.
  • Limitations and future directions: Future work includes 3D extensions, hierarchical anatomical and modality factors, improved texture fidelity, better semantic–reconstruction trade-offs, and fully unsupervised learning.
Loading 1903.09467v4…