Source-linked AI summary
Joint Learning of Motion Estimation and Segmentation for Cardiac MR Image Sequences
Chen Qin, Wenjia Bai, Jo Schlemper, Steffen E. Petersen, Stefan K. Piechnik, Stefan Neubauer, Daniel Rueckert
TL;DR
Cardiac MR motion estimation and segmentation are important for quantitative cardiac assessment but are often handled separately and rely on limited annotations. The paper jointly trains an unsupervised Siamese recurrent spatial-transformer motion branch and a fully convolutional segmentation branch with shared features, reporting complementary performance improvements on data from 220 subjects.
Problem
Cardiac MR motion estimation and segmentation are important for assessing cardiac function, while existing approaches commonly treat them separately and segmentation often uses only sparsely annotated frames.
Method
A joint network combines an unsupervised Siamese recurrent spatial-transformer motion branch with a fully convolutional segmentation branch sharing a joint feature encoder.
Results
The joint models significantly outperform competing approaches in segmentation and motion-estimation evaluations, with improvements reported across all three cardiac structures and motion-distance metrics.
Takeaways & Limitations
Joint learning enables weakly supervised segmentation from temporally sparse annotations while exploiting motion information from unlabeled cardiac MR sequences.
Abstract
from arXiv · showhide
Cardiac motion estimation and segmentation play important roles in quantitatively assessing cardiac function and diagnosing cardiovascular diseases. In this paper, we propose a novel deep learning method for joint estimation of motion and segmentation from cardiac MR image sequences. The proposed network consists of two branches: a cardiac motion estimation branch which is built on a novel unsupervised Siamese style recurrent spatial transformer network, and a cardiac segmentation branch that is based on a fully convolutional network. In particular, a joint multi-scale feature encoder is learned by optimizing the segmentation branch and the motion estimation branch simultaneously. This enables the weakly-supervised segmentation by taking advantage of features that are unsupervisedly learned in the motion estimation branch from a large amount of unannotated data. Experimental results using cardiac MRI images from 220 subjects show that the joint learning of both tasks is complementary and the proposed models outperform the competing methods significantly in terms of accuracy and speed.
1 Introduction
Cardiac MR segmentation and motion estimation are important for assessing cardiac morphology and function, but they are commonly treated as separate tasks despite their close relationship. The paper proposes jointly learning both tasks to exploit their complementary information.
- Cardiac MR segmentation and motion estimation support quantitative assessment of cardiac function, including ventricular-volume changes and myocardial elasticity and contractility.
- The proposed approach jointly predicts segmentation and motion from cardiac MR sequences using a deep-learning network with complementary task branches.
- Most existing cardiac MR segmentation methods train fully convolutional networks on manually annotated images, typically using only end-diastolic and end-systolic frames.
- Traditional motion-estimation methods extend optical flow or image-registration techniques, while newer deep-learning methods learn similarity metrics or regress simulated transformations.
2 Methods
The method combines unsupervised recurrent multi-scale motion estimation with segmentation in a shared-feature architecture. Training uses image similarity, motion smoothness, and segmentation losses so unlabeled sequence information can support segmentation.
- The unified model contains an unsupervised Siamese recurrent multi-scale spatial-transformer motion branch and a fully convolutional segmentation branch sharing a joint feature encoder.
- The motion branch estimates pixel-wise 2D displacement fields between a target frame and consecutive source frames using Siamese features, multi-scale concatenation, recurrence, and image warping.
- Motion training minimizes transformed-to-target image error while penalizing spatial and temporal flow variation to encourage smooth motion fields.
- Joint learning uses motion-derived features and warped segmentations from unlabeled data to support weakly supervised segmentation and regularize motion estimation near boundaries.
- The composite objective combines motion loss, supervised segmentation loss, and warped-segmentation loss, with staged initialization followed by joint optimization.
3 Experiments and Results
Experiments on 220 cardiac MR sequences evaluated segmentation and motion estimation using complementary accuracy, visualization, and timing analyses. The joint model improved segmentation and motion estimation over the reported baselines while producing temporally smooth predictions.
- Experimental setup: 220 short-axis cardiac MR sequences were used, with 50 frames per sequence and manual LV, Myo, and RV segmentations available at ED and ES.Each short-axis stack typically contained 10 slices.
- Segmentation evaluation: The joint model significantly outperformed Seg only on Dice scores for LV, Myo, and RV segmentation, with p ≪0.001 for all structures.The improvement was especially pronounced for Myo, where motion commonly affects segmentation accuracy.
- Motion evaluation: Both proposed motion-estimation methods outperformed FFD on MCD and HD for all three structures, with p ≪0.001.The joint model also exceeded Motion only, with p ≪0.001 on LV and RV and p < 0.01 on Myo.
- Motion evaluation: The proposed motion-estimation methods were faster than FFD when testing 50 frames of one cardiac-cycle slice.Table 2 reports testing time per slice for 50 frames.
- Qualitative evaluation: The joint model simultaneously predicted motion fields and segmentations, while warped ED segmentations produced accurate, smooth, and temporally consistent labeling and LV-volume estimates.The visualizations cover ED-to-other-time-point myocardial motion and cardiac-cycle temporal views.
4 Conclusion
The paper concludes that a two-branch deep learning model can jointly estimate cardiac motion and segment cardiac MR sequences. Shared feature learning supports weakly supervised segmentation with temporally sparse annotations, while experiments report improvements in accuracy and speed.
- Architecture: The architecture combines an unsupervised Siamese recurrent spatial transformer for motion estimation with a fully convolutional segmentation branch.A joint feature encoder is shared between the branches.
- Joint learning: Shared multi-task feature learning enables weakly supervised segmentation from temporally sparse annotated data.The paper describes this as effective feature learning through joint training.
- Conclusion: Experiments showed significant improvements over baseline approaches in accuracy and speed.The conclusion reports these improvements without specifying additional metric values here.
- Future work: Future work will evaluate the method on a larger-scale dataset and investigate its usefulness for 3D applications.These are the authors’ stated next research directions.