Source-linked AI summary
Multi-Pass, Multi-View Blended Learning for High-Fidelity Volumetric CT Synthesis from Chest X-Rays
Ozer Can Devecioglu, Serkan Kiranyaz, Rashid Mazhar, Tahir Hamid, Muhammad Chowdhury, Moncef Gabbouj
TL;DR
The paper addresses volumetric CT reconstruction from single CXRs despite scarce paired clinical data and a DRR-to-real-CXR domain gap. It proposes a two-stage blended-learning framework with three CT-synthesis passes, achieving improvements over prior methods on LIDC-IDRI while generating high-fidelity volumes from real CXRs. Real-CXR quantitative clinical validation remains limited.
Problem
Single-CXR-to-CT reconstruction is ill-posed, while scarce paired CXR–CT data forces reliance on DRRs whose domain gap limits clinical generalization.
Method
A two-stage framework performs unsupervised CXR-to-DRR translation, then supervised DRR-to-CT synthesis, unsupervised multi-view refinement, and Progressive Transfer Learning.
Results
The final output achieves the best PSNR across axial, coronal, and sagittal views and the best SSIM in coronal and sagittal views across the evaluated pipeline stages.
Takeaways & Limitations
The framework produces structurally consistent, anatomically realistic CT volumes from real CXRs and qualitatively generalizes to CheXpert clinical inputs.
Takeaways & Limitations
Quantitative evaluation is limited to LIDC-IDRI, while evaluation on real CXRs remains qualitative.
Abstract
from arXiv · showhide
Reconstructing volumetric Computed Tomography (CT) from a single 2D chest radiograph (CXR) is an ill-posed inverse problem, further complicated by the scarcity of paired CXR-CT training data. Prior approaches address this by training on Digitally Reconstructed Radiographs (DRRs), which are synthetic projections derived from CT volumes. However, the domain gap between DRRs and real CXRs limits generalization, often resulting in coarse or anatomically inconsistent reconstructions when applied to clinical images. To address this challenging problem, this study introduces a Multi-Pass Multi-View Blended Learning framework for synthesizing high-fidelity volumetric CT directly from real chest X-ray (CXR) images. The proposed approach progressively decomposes the synthesis task into two distinct, complementary learning stages. Stage 1 is an unsupervised CXR-to-DRR Domain Adaptation, while Stage 2 includes three passes, namely, (a) supervised DRR-to-CT Transformation, (b) unsupervised Multi-View Slice Refinement, followed by (c) Progressive Transfer Learning (PTL). With such a blended learning paradigm, the proposed approach mitigates the synthetic-to-real domain gap while enhancing both the structural integrity and anatomical detail of the final output. On the LIDC-IDRI dataset, where paired DRR-CT ground truth is available for quantitative evaluation, the proposed method improves upon prior methods by up to 14% in PSNR and 7.6% in SSIM. The framework successfully generates structurally consistent and anatomically realistic high-fidelity CT volumes from real CXRs, marking a significant advancement toward clinical viability of CT reconstruction from standard radiographic images.
I. INTRODUCTION Computed Tomography (CT) is one of the most widely used
CT offers detailed volumetric anatomy but exposes patients to substantially more radiation than chest X-rays. Existing CXR-to-CT approaches face limited paired data, synthetic-to-real gaps, and anatomically coarse reconstructions; the proposed framework addresses these challenges through staged, multi-pass learning.
- Motivation: Chest CT provides 3D anatomical information for diagnosis, surgical planning, and treatment monitoring, but can expose patients to 70–180 times the radiation of a chest X-ray.Reported chest CT exposure ranges from 1–1.5 mSv for low-dose screening to 7–18 mSv for high-resolution contrast CT, versus about 0.1 mSv for a chest X-ray.
- Prior work: Most recent deep-learning approaches train on synthetic DRRs because directly paired clinical CXR–CT datasets are scarce.DRRs are derived from target CT volumes, which are unavailable when reconstructing CT from real clinical CXRs.
- Problem: DRR-to-CT reconstruction remains difficult because a small number of 2D views must provide missing depth information for a 3D volume.Recent models often produce coarse, over-smoothed slices with weak fine-grained spatial consistency and anatomical detail.
- Proposed approach: The proposed framework uses two stages: unsupervised real-CXR-to-DRR translation followed by three CT-synthesis passes over the generated DRRs.The second stage combines supervised DRR-to-CT mapping, multi-view slice refinement, and Progressive Transfer Learning.
- Proposed approach: Multi-view refinement independently enhances axial, coronal, and sagittal outputs, while Progressive Transfer Learning continues optimization beyond standard convergence.The design targets cross-view structural consistency and sharper, more realistic reconstructions.
II. METHODOLOGY
The framework uses two consecutive stages: unsupervised CXR-to-DRR domain translation followed by three-pass CT synthesis, incorporating adaptable neural operators and progressive refinement. The first CT pass combines paired DRR–CT supervision, multi-view structural constraints, and a 2D-to-3D reconstruction architecture to produce a structurally consistent but low-fidelity volume.
- Overall framework: The framework first translates real CXR images into the DRR domain, enabling Stage 2 reconstruction models trained on paired DRR–CT data to process clinical inputs.The domain translation is unsupervised because the CXR and DRR datasets lack one-to-one correspondence.
- Self-Organized Operational Neural Networks: Self-ONNs replace fixed convolutional operations with generative neurons whose learned nodal operators support adaptable nonlinear transformations.Their nodal operators are approximated with a Taylor or Maclaurin expansion whose coefficients are learned during backpropagation.
- Progressive Transfer Learning: PTL decomposes a difficult one-step transformation into sequential refinements, reusing the previous best model to improve residual errors and training stability.Each subsequent model is initialized from the previous best-performing model, and the final output is obtained through sequential transformations.
- Overall framework: Stage 2 synthesizes CT over translated DRRs through three successive passes: supervised baseline transformation, unsupervised multi-view refinement, and Progressive Transfer Learning.The first pass captures major anatomy, while later passes target detail and further quality improvement.
- D.1. First pass — DRR-to-CT baseline transformation: The first DRR-to-CT pass uses two 2D encoder streams, feature concatenation, 2D-to-3D projection, and a 3D U-Net-based refinement network.The projection reshapes fused features into a 3D tensor with predefined depth bins before volumetric refinement.
- D.1. First pass — DRR-to-CT baseline transformation: The first-pass output preserves overall anatomical structure but lacks fine detail, while L1 and multi-view SSIM losses encourage voxel accuracy and consistency across anatomical planes.SSIM is computed on axial, coronal, and sagittal slices and averaged.
D.2. Second pass — multi-view slice refinement
The second pass refines low-fidelity CT volumes with an unpaired Op-CycleGAN that learns bidirectional mappings between low- and high-quality CT slice domains. Adversarial and identity objectives seek realistic detail recovery while preserving the original structural content.
- D.2. Second pass — multi-view slice refinement: Op-CycleGAN learns bidirectional low-quality-to-high-quality and high-quality-to-low-quality mappings without paired supervision.Two operational generators and two operational discriminators support the bidirectional refinement process.
- D.2. Second pass — multi-view slice refinement: The refinement targets anatomical details and patterns missed by the supervised first-pass reconstruction while preserving global structural consistency.The method operates on low-quality CT volumes produced in the first pass and learns from unpaired low- and high-quality CT domains.
- D.2. Second pass — multi-view slice refinement: Adversarial losses encourage realistic outputs in both CT domains, while identity loss prevents unnecessary modifications.The cycle structure reconstructs the original low-quality image after high-quality translation, maintaining cycle consistency.
D.3 Third pass — Progressive Transfer Learning
The third pass further refines the CT views produced by the second pass using PTL-initialized Op-CycleGAN models. Refinement is performed independently for axial, coronal, and sagittal views.
- D.3 Third pass — Progressive Transfer Learning: The final pass applies the same Op-CycleGAN architecture independently to axial, coronal, and sagittal CT views.Each view-specific model is initialized with Progressive Transfer Learning from the preceding refinement stage.
III. EXPERIMENTAL RESULTS
Experiments evaluate the framework on synthetic DRR–CT pairs and qualitatively on real CXRs. Across reconstruction passes, synthesis quality improves, while real-CXR outputs show detailed and consistent anatomy; inference remains under two seconds.
- Datasets: The evaluation uses LIDC-IDRI for quantitative DRR-to-CT assessment and CheXpert for real-CXR qualitative evaluation.LIDC-IDRI supplies CT-derived DRRs and paired ground truth, while CheXpert provides unpaired clinical CXRs.
- CT synthesis over DRRs: The first pass captures main anatomical structure but produces blurred boundaries and little fine detail.The first pass is a supervised DRR-to-CT transformation.
- CT synthesis over DRRs: The second pass recovers missing anatomical detail and contrast through multi-view Op-CycleGAN refinement, reducing blurring.Axial, sagittal, and coronal slices are evaluated for three-dimensional consistency.
- CT synthesis over DRRs: PTL further sharpens structural boundaries and textures, improving preservation of fine anatomical detail in the final pass.The qualitative comparison includes three-pass outputs and corresponding ground-truth CT slices.
- Quantitative results: Up to 14% higher PSNR and 7.6% higher SSIM are reported over IRR3CT in the sagittal view.The final output achieves the best PSNR across axial, coronal, and sagittal views and the best SSIM in coronal and sagittal views.
- CT synthesis over real CXRs: Real-CXR samples produce detailed, anatomically consistent structures comparable to the DRR-based evaluation.The results support generalization of the Stage 1 CXR-to-DRR transformation to clinical inputs.
- Computational complexity: The multi-stage framework achieves inference in under two seconds, supporting near real-time clinical applications.The complexity analysis includes network size, parameter count, and inference time for the framework components.
IV. CONCLUSION
The study proposes a two-stage, three-pass framework to reconstruct high-fidelity volumetric CT from real CXRs despite ambiguous 2D-to-3D mapping and scarce paired data. Experiments report improved anatomical detail and cross-view consistency, while quantitative clinical validation remains a future need.
- Conclusion: The framework combines unsupervised CXR-to-DRR adaptation with supervised DRR-to-CT synthesis, unsupervised multi-view refinement, and PTL.The formulation decomposes the reconstruction problem into complementary stages and passes.
- Conclusion: Each pass contributes to synthesis performance, although axial SSIM decreases slightly after the second pass.The subsequent refinement improves local anatomical detail and structural consistency, while PTL further sharpens reconstructions.
- Conclusion: The framework outperforms recent prior methods on LIDC-IDRI and produces high-fidelity volumetric CT views from real CheXpert CXRs qualitatively.The conclusion reports both quantitative and qualitative evidence across the two datasets.
- Limitations and future work: Quantitative clinical validation is limited because current quantitative evaluation uses LIDC-IDRI, while real-CXR evaluation remains qualitative.Future work proposes expert radiologist evaluation and larger, multi-source datasets to assess generalization.
APPENDIX
The appendix provides additional sample CT slice synthesis results across different passes of the proposed approach.
- Appendix: Additional sample CT slice synthesis results are provided for different passes of the proposed approach.