Source-linked AI summary
RMSim: Controlled Respiratory Motion Simulation on Static Patient Scans
Donghoon Lee, Ellen Yorke, Masoud Zarepisheh, Saad Nadeem, Yu-Chi Hu
TL;DR
Respiratory motion complicates diagnosis, image interpretation, and DIR, while existing user-defined synthetic deformations may not capture realistic respiratory motion. RMSim predicts trace-modulated, patient-specific motion phases from a static CT, and the authors report improved DIR performance with RMSim-generated augmentation. The work presents the simulator as a source of motion ground truth for DIR validation and patient-specific training augmentation.
Problem
User-defined synthetic DVFs offer limited transformations that cannot capture realistic respiratory motion, which also complicates diagnosis, image interpretation, and registration.
Source: §1 · p. 2Method
RMSim uses a 3D Seq2Seq model to predict patient-specific, time-varying DVFs from a static CT, with a 1D breathing trace modulating the motion.
Source: §1 · p. 2Results
6.58±6.38 mm for VoxelMorph with augmentation versus 8.12±5.78 mm without augmentation in POPI landmark TRE; the authors report improved DIR performance with patient-specific augmentations.
Source: §3 · p. 11, §4 · p. 12Takeaways & Limitations
The authors present RMSim-generated motion as ground truth for DIR validation and as patient-specific augmentation for training deep-learning DIR algorithms.
Source: §1 · p. 2, §4 · p. 12The approach assumes regular breathing and that regional DVFs scale linearly with diaphragm motion; its input and output dimensions were limited to 128×128×128 by GPU memory constraints.
Source: §4 · p. 13
Abstract
from arXiv · showhide
This work aims to generate realistic anatomical deformations from static patient scans. Specifically, we present a method to generate these deformations/augmentations via deep learning driven respiratory motion simulation that provides the ground truth for validating deformable image registration (DIR) algorithms and driving more accurate deep learning based DIR. We present a novel 3D Seq2Seq deep learning respiratory motion simulator (RMSim) that learns from 4D-CT images and predicts future breathing phases given a static CT image. The predicted respiratory patterns, represented by time-varying displacement vector fields (DVFs) at different breathing phases, are modulated through auxiliary inputs of 1D breathing traces so that a larger amplitude in the trace results in more significant predicted deformation. Stacked 3D-ConvLSTMs are used to capture the spatial-temporal respiration patterns. Training loss includes a smoothness loss in the DVF and mean-squared error between the predicted and ground truth phase images. A spatial transformer deforms the static CT with the predicted DVF to generate the predicted phase image. 10-phase 4D-CTs of 140 internal patients were used to train and test RMSim. The trained RMSim was then used to augment a public DIR challenge dataset for training VoxelMorph to show the effectiveness of RMSim-generated deformation augmentation. We validated our RMSim output with both private and public benchmark datasets (healthy and cancer patients). The proposed approach can be used for validating DIR algorithms as well as for patient-specific augmentations to improve deep learning DIR algorithms. The code, pretrained models, and augmented DIR validation datasets will be released at https://github.com/nadeemlab/SeqX2Y.
1. Introduction
Respiratory motion complicates diagnosis, image interpretation, and registration, while user-defined synthetic deformations cannot capture realistic respiratory motion. RMSim predicts patient-specific, trace-modulated motion from a static 3D CT, with the stated aim of supporting DIR validation and data augmentation.
1. Introduction
- Respiratory motion can impair diagnosis and radiotherapy, create image artifacts, and make image registration across breathing phases challenging.Commercial synthetic-DVF tools use user-defined monotonic transformations that cannot capture realistic respiratory motion.Source: §1 · p. 2
- Earlier deep-learning approaches described here predicted 2D motion or a single future-time-point 3D DVF, rather than multiple 3D breathing phases.These approaches used Seq2Seq or encoder-decoder models, including latent-space methods.Source: §1 · p. 2
- RMSim generates patient-specific time-varying DVFs from a static 3D CT, with breathing traces modulating the simulated motion.The authors describe these simulations as both diverse patient-specific augmentations and ground truth for DIR validation.Source: §1 · p. 2
- RMSim predicts nine future 3D breathing phases simultaneously and uses a 3D Seq2Seq architecture; the authors report that its augmentations improve deep-learning DIR performance.The model uses the VoxelMorph spatial-transform module rather than the full VoxelMorph method.Source: §1 · p. 3
2. Materials and Methods
The study uses internal and public CT datasets to develop and evaluate RMSim, a 3D encoder-decoder that predicts respiratory motion and can generate augmented DIR training pairs. Evaluation compares simulated images with reference phases, measures landmark registration error, and assesses image similarity and lung-contour overlap.
2.1. Datasets
- RMSim was trained and tested on 140 internal NSCLC 10-phase 4D-CTs, with a 100:40 split; external datasets included LUNA, POPI, and Learn2Reg.The internal scans were retrospectively collected and de-identified; datasets were cropped and resampled to 128×128×128 at 2 mm voxel size.Source: §2.1 · p. 4
2.2. Realistic Respiratory Motion Simulation
- RMSim combines 3D convolutional encoding and decoding, stacked 3D ConvLSTM layers for spatial-temporal features, and a spatial transform that warps phase 1 with predicted DVFs.The decoder upsamples each predicted phase output to the input resolution to produce its DVF.Source: §2.2 · pp. 4–6
- A diaphragm-apex displacement trace extracted from LDDMM-derived DVFs multiplicatively modulates each phase's hidden state; training combines phase-image MSE with DVF-gradient smoothness.The reference phase image is phase 1, and the predicted DVF spatially transforms it toward each ground-truth phase.Source: §2.2 · p. 6
2.3. Data augmentation by RMSim
- For Learn2Reg augmentation, RMSim generated nine additional phases per inhalation image, expanding 20 training pairs to 200 pairs for VoxelMorph training.The study compared VoxelMorph trained on original pairs with a model trained on the augmented pairs.Source: §2.3 · p. 7
2.4. Evaluation Metrics
- Evaluation used SSIM for image similarity, Dice for propagated versus reference lung contours, and TRE for transformed versus target landmarks.SSIM was used for simulated-phase and registration-image comparisons; POPI landmark comparison used Euclidean-distance TRE.Source: §2.4 · p. 7
3. Results
RMSim-generated phases achieved high image similarity and patient-specific motion variation, while RMSim augmentation improved VoxelMorph registration results on the POPI and Learn2Reg evaluations.
- 0.92±0.04 average SSIMsim exceeded 0.86±0.08 SSIMgnd between phase 1 and other ground-truth phases (p < 0.01) in internal test cases.Source: §3 · p. 7
- Diaphragm-displacement error increased from inhale to exhale as later-time-point prediction accuracy decreased, but overall displacement error remained within 3 mm.Source: §3 · p. 8, Figure 3 · p. 8
- Applying different breathing traces to the same CT produced different motion, with higher-amplitude BT2 yielding greater diaphragm movement; propagated anatomical contours matched the predicted images.Source: §3 · p. 8, Figure 4 · p. 9
- For respiratory amplitudes beyond the internal dataset’s 0.14–40 mm range, a fivefold input produced larger fluctuation, but the predicted motion was not proportional to the input.Source: §3 · p. 8
- 0.92±0.64 mm average landmark TRE was achieved when internal-trained RMSim predicted remaining phases for one POPI case with a breathing-trace input.Source: §3 · p. 9, Figure 6 · p. 10
- 0.97 ± 0.01 lung Dice with augmentation versus 0.96 ± 0.01 without (p < 0.001), while SSIM was 0.89 ± 0.02 versus 0.88 ± 0.02 (p < 0.001).Source: §3 · pp. 9–10
- 6.58±6.38 mm average TRE with augmentation, versus 8.12±5.78 mm without and 8.05±5.61 mm pre-DIR (p < 3e-48); Figure 8 shows improvement across all six POPI cases.Source: §3 · p. 11, Figure 8 · p. 12
4. Discussion
The discussion reports benchmark validation and improved DIR performance with patient-specific augmentation, then outlines radiotherapy and artifact-simulation applications. It also identifies surrogate and resolution limits and future extensions.
4. Discussion
- RMSim was validated on private and public benchmarks, and patient-specific augmentation improved state-of-the-art deep-learning DIR performance.The work predicts motion for one breathing cycle; multiple-cycle prediction and evaluation on 4D-MRI are future directions.Source: §4 · p. 12
- Simulated 4D-CT can support internal target volume delineation or motion-PDF extraction for robust radiotherapy planning.The internal target volume is the union of target volumes across respiratory phases.Source: §4 · pp. 12–13
- Generated, breathing-trace-modulated 4D-CT phases could feed prior CT/CBCT motion-artifact simulation for artifact-robust translation, segmentation, and motion correction [21, 22, 23].The paper describes this as a future direction extending an earlier physics-based artifact-simulation pipeline.Source: §4 · p. 13
- The simulator assumes regular breathing and linearly scaled regional DVFs from diaphragm displacement, and 1D traces might not represent cardiac motion.GPU memory constraints limited input and output dimensions to 128x128x128.Source: §4 · p. 13