Source-linked AI summary
Deep Embedding Convolutional Neural Network for Synthesizing CT Image from T1-Weighted MR Image
Lei Xiang, Qian Wang, Xiyao Jin, Dong Nie, Yu Qiao, Dinggang Shen
TL;DR
MR-to-CT synthesis is difficult because the modalities differ substantially in appearance, while existing approaches can be computationally expensive or produce 3D discontinuities. The paper proposes DECNN, which repeatedly embeds tentative CT estimates into feature maps, and reports superior synthesis results and running time on brain and prostate data. Training remains time-consuming and can require 2–3 days to obtain a model.
Problem
MR-to-CT synthesis is important but challenging because MR and CT have substantially different appearances and complex mappings.
Method
DECNN repeatedly embeds tentative CT estimates into deep feature maps and transforms them forward to synthesize a final CT image directly from MR.
Results
DECNN achieves the best synthesis results and is several times or orders of magnitude faster at testing than traditional learning-based methods on brain and prostate data.
Takeaways & Limitations
Repeated embedding supports high-quality, efficient end-to-end MR-to-CT synthesis across brain and prostate datasets.
Takeaways & Limitations
Training DECNN takes 2–3 days to obtain a model, and the authors identify reducing network capacity without sacrificing accuracy as an efficiency challenge.
Abstract
from arXiv · showhide
Recently, more and more attention is drawn to the field of medical image synthesis across modalities. Among them, the synthesis of computed tomography (CT) image from T1-weighted magnetic resonance (MR) image is of great importance, although the mapping between them is highly complex due to large gaps of appearances of the two modalities. In this work, we aim to tackle this MR-to-CT synthesis by a novel deep embedding convolutional neural network (DECNN). Specifically, we generate the feature maps from MR images, and then transform these feature maps forward through convolutional layers in the network. We can further compute a tentative CT synthesis from the midway of the flow of feature maps, and then embed this tentative CT synthesis back to the feature maps. This embedding operation results in better feature maps, which are further transformed forward in DECNN. After repeat-ing this embedding procedure for several times in the network, we can eventually synthesize a final CT image in the end of the DECNN. We have validated our proposed method on both brain and prostate datasets, by also compar-ing with the state-of-the-art methods. Experimental results suggest that our DECNN (with repeated embedding op-erations) demonstrates its superior performances, in terms of both the perceptive quality of the synthesized CT image and the run-time cost for synthesizing a CT image.
1 Introduction
MR-to-CT synthesis is challenging because the modalities have complex, nonlinear and sometimes inconsistent intensity mappings. DECNN addresses this with repeated embedding of tentative CT estimates into deep feature maps, supporting direct, efficient synthesis across brain and prostate data.
- Motivation: MR and CT are important medical imaging modalities, but their intensity mapping is highly complex, nonlinear, and context-dependent.Air appears dark in both modalities, whereas bone is dark in MR but bright in CT.
- Prior methods: Atlas-based methods depend on registration accuracy, can vary with atlas selection and deformation estimates, and may require costly registration of all images.These limitations can affect the synthesized CT image for a new subject.
- Prior methods: Sparse-coding methods are computationally expensive because they perform patch extraction, sparse optimization, and dictionary-based prediction across image locations.Large global dictionaries further increase the cost of solving sparse representation coefficients.
- Prior methods: Existing CNN-based MR-to-CT methods can require independent processing of multiple slices, causing discontinuities and artifacts in 3D synthesized CT.Patch-based learning methods may also require decomposing images into overlapping patches and integrating predicted patches.
- Proposed method: DECNN directly estimates CT from T1-weighted MR using a deep architecture with transform and reconstruction stages, without patch-level pre- or post-processing.The network is designed for modality mapping across different medical applications.
- Proposed method: Repeated embedding blocks synthesize tentative CT images midway through the network, feed them back into feature maps, and refine the final CT estimate.The embedding strategy also supports gradient back-propagation and end-to-end training.
- Results: DECNN strengthens MR-to-CT mapping, supports efficient end-to-end testing, and outperforms state-of-the-art methods in CT accuracy and synthesis speed.The authors report improved mapping capability, training convergence, and test-time efficiency.
2 Method
The proposed DECNN extends a CNN for T1-weighted MR-to-CT synthesis by repeatedly embedding tentative CT estimates into evolving feature maps. This design supports progressive refinement of the modality mapping and efficient full-field CT synthesis.
- CNN stages: The transform stage forwards MR-derived feature maps through convolutional layers, while the reconstruction stage synthesizes CT from the final transformed maps.The transform stage produces 128 feature maps, and the reconstruction stage uses a single convolutional layer to generate a 64×64×3 CT patch.
- Embedding block: The embedding block feeds tentative CT estimates back into feature maps so subsequent layers can refine the CT synthesis across repeated stages.The authors describe this feedback as addressing the large appearance gap between MR and CT.
- Embedding block: Each embedding block tentatively synthesizes CT from current feature maps, concatenates the result with transformed feature maps, and forwards the combined maps through another convolutional layer.The tentative CT is treated as three additional feature maps, producing 131 concatenated maps before reduction to 128.
- DECNN architecture: DECNN inserts multiple embedding blocks into the late transform stage of a CNN and appends a convolutional refinement layer after each block.The final feature maps are passed to the reconstruction stage for CT synthesis.
- Optimization: Auxiliary losses from tentative CT estimates are included to strengthen parameter learning and improve robustness against vanishing gradients.The overall loss combines losses from tentative CT estimations with parameter regularization.
3 Experimental Result
Experiments on brain and prostate datasets evaluate DECNN across embedding depth, quasi-3D inputs, visual quality, feature evolution, and comparisons with existing methods. DECNN achieves strong synthesis quality and efficient inference, while the best embedding depth depends on the dataset.
- Embedding depth: The highest average PSNR occurs with “Ebd 2” on brain and “Ebd 4” on prostate, while performance drops when the network becomes too deep.The decline is attributed partly to the small size of each dataset.
- Comparisons with state-of-the-art methods: 27.3 dB with embedding versus 26.7 dB without embedding on brain, and 33.5 dB versus 33.1 dB on prostate, with statistically significant improvements.The paired t-test p-values are 0.0022 and 0.0086, respectively.
- Embedding depth: Embedding improves synthesis quality progressively on brain images, with “Ebd 2” producing the most satisfying result compared with the ground truth.Without embedding, the model produces more artifacts; one block improves the result further.
- Quasi-3D mapping: Using more than one axial slice improves intra-slice quality and inter-slice continuity, with fewer artifacts across axial planes.The model therefore represents MR-to-CT mapping in a quasi-3D way.
- Quasi-3D mapping: 27.4 dB for “×5 slices” versus 27.3 dB for “×3 slices” on brain makes the settings comparable, motivating the lower-complexity “×3 slices” choice.The “×3 slices” setting is also better than “×1 slice” in the reported visual comparison.
- Comparisons with state-of-the-art methods: DECNN outperforms the compared CNN and other methods, exceeds FCN on brain, matches FCN on prostate, and achieves the highest overall performance among state-of-the-art methods.It produces higher PSNRs and lower MAEs than a same-depth CNN without embedding.
- Efficiency: DECNN synthesizes a brain subject in 46 seconds and a prostate subject in 28 seconds, while RF+ACM takes about 20 minutes for a test subject.DECNN also avoids the FCN testing-stage post-processing requirement because its output size equals its input size.
4 Discussion
The method was evaluated on brain and prostate data, where it achieved better synthesis results and faster testing than traditional learning-based methods. The authors also identify training cost, dataset noise, misalignment, and restricted anatomical or intensity coverage as limitations and future-work targets.
- DECNN achieved the best synthesis results and was several times or even orders of magnitude faster than traditional learning-based methods during testing.
- Training required 2–3 days, whereas traditional methods mostly required less than a day.
- Noise in the training dataset could affect testing performance, producing bone-intensity noise in predicted results.
- Slight misalignment in paired MR-CT training images could cause inaccurate CT predictions.
- The brain experiments focused on bone-window intensities, while prostate experiments focused mainly on the prostate, bladder, and rectum.The authors plan to extend brain synthesis to soft-tissue contrast and prostate synthesis to the whole pelvic image.
- Whole-image clinical use may require linear registration, region cropping, and additional post-processing before synthesis.
5 Conclusion
The paper presents DECNN for synthesizing CT from T1-weighted MR images by repeatedly embedding tentative CT estimates into feature maps. Experiments support more accurate modeling, faster convergence, and overall superiority to state-of-the-art methods in image quality and running time.
- DECNN synthesizes CT images from T1-weighted MR images using a deep embedding convolutional neural network.
- The embedding block inserts tentative CT estimates into deep feature-map flows, enabling forward transformation and reconstruction of better CT synthesis results.
- The embedding block contributes to more accurate MR-to-CT modeling and faster convergence during deep-network training.
- DECNN outperformed state-of-the-art methods in both synthesized CT image quality and running time.
- The authors describe DECNN as a generalized deep-learning solution that is not restricted to MR-to-CT synthesis.
- End-to-end mapping produces CT images the same size as input MR images and is intended to improve test-image processing efficiency.
- Future work will investigate DECNN on other image-synthesis tasks and broader clinical applications.