Source-linked AI summary
PhyDiCT: Plug-and-Play CT Reconstruction from Sparse X-Rays via Differentiable Rendering and Strong Priors
Weicheng Dai, Shantanu Ghosh, Kayhan Batmanghelich
TL;DR
Reconstructing 3D lung CT from a few X-ray projections is highly ill-posed, and paired 2D–3D training data are difficult to acquire. PhyDiCT steers a text-conditioned diffusion prior with differentiable physics-based rendering and refinement. It outperforms baselines across reported metrics, including a 7.5% SSIM improvement over X2CT, but is designed specifically for lung CT.
Problem
Reconstructing volumetric lung CT from limited 2D X-ray projections is highly ill-posed, and paired 2D–3D datasets are difficult to acquire.
Source: §1 · p. 1Method
PhyDiCT steers a pretrained text-conditioned diffusion prior using a differentiable Beer–Lambert X-ray renderer and reward-guided sampling, with Extra Compute refinement.
Source: §1 · pp. 2–3, §2 · pp. 3–4Results
Takeaways & Limitations
PhyDiCT combines physical 3D-to-2D rendering with a strong text-guided prior for limited-view lung CT reconstruction.
Source: §4 · p. 9PhyDiCT is specifically designed for 3D lung CT; other organs require corresponding pretrained priors.
Source: §4 · p. 9
Abstract
from arXiv · showhide
Reconstructing 3D Computed Tomography (CT) images from a few X-ray projections is a highly ill-posed inverse problem due to the loss of volumetric information. We propose PhyDiCT, a training-free framework that integrates a differentiable Physics-based forward model, grounded in the Beer-Lambert law, with a text-conditioned Diffusion as a strong prior to reconstruct 3D lung CT images. We refer to our approach as training-free since the prior model is used without fine-tuning, and our goal is to steer the denoising procedure to generate samples consistent with X-ray observations. We guide the diffusion generation using Split Gibbs sampling to jointly optimize for projection fidelity (reward) and consistency with prior knowledge. Also, we introduce a test-time refinement step that enhances image realism and anatomical coherence. We extensively evaluate our method on publicly available 3D CT datasets using both perceptual and semantic metrics, demonstrating that it surpasses existing plug-and-play diffusion and fully trained reconstruction approaches. Our findings highlight that combining a strong generative prior with the underlying physics of image formation substantially improves reconstruction quality, e.g., 7.5\% improvement on SSIM compared to full training methods. Code will be released at https://github.com/batmanlab/PhyDiCT.
1 Introduction
Reconstructing lung CT from sparse X-rays is highly ill-posed, while existing approaches either need many views or paired training data. PhyDiCT combines a text-guided pretrained prior with differentiable physics-based projection to reconstruct volumes without end-to-end training.
1 Introduction
- Sparse-view lung CT reconstruction is highly ill-posed; traditional methods often need many views, while learned methods require difficult-to-acquire paired 2D–3D data.Traditional approaches also weakly exploit anatomical and tissue-specific priors.Source: §1 · pp. 1–2
- PhyDiCT uses a pretrained text-guided generative prior and a differentiable reward to enforce consistency between 3D CT volumes and sparse X-ray observations.The reward models the projection process using the Beer–Lambert law.Source: §1 · p. 2
- Existing plug-and-play methods may use natural-image or unconditional priors and rewards that ignore the physical 3D-to-2D image-formation process.These limitations motivate stronger text-guided priors and physics-grounded rewards.Source: §1 · p. 2
- Split Gibbs sampling alternates likelihood optimization and denoising, while Extra Compute refinement is introduced to suppress reconstruction artifacts.The method uses a diffusion model that directly generates full CT volumes.Source: §1 · pp. 2–3
- PhyDiCT is reported to outperform all baselines across perceptual, semantic, and domain-specific metrics.The introduction describes evaluation across these metric categories.Source: §1 · p. 3
2 Method
PhyDiCT treats reconstruction as posterior sampling from a text-conditionable diffusion prior tilted toward consistency with observed X-rays. It combines differentiable physics-based projection, Split Gibbs guidance, and a final unguided refinement step.
2 Method
- PhyDiCT samples from a posterior that combines the MedSyn prior, optional text condition, and a reward for consistency with observed X-rays.The method uses Split Gibbs Sampling to alternate between prior denoising and likelihood optimization.Source: §2 · pp. 3, 5
- MedSyn starts from noise and iteratively predicts a clean image before applying a deterministic DDIM backward step.It optionally conditions on text and also produces lung segmentation masks to increase anatomical fidelity.Source: §2 · p. 4
- A Beer–Lambert renderer models X-ray formation from the CT attenuation field, and a differentiable trilinear projection maps volumes to 2D images.The total guidance loss adds LPIPS to reconstruction loss to address possible domain shift between simulated projections and observations.Source: §2 · p. 4
- Split Gibbs Sampling alternates MedSyn denoising with likelihood optimization over an auxiliary variable to balance observation fidelity and prior consistency.The likelihood step is approximated with a few gradient steps; experiments use two.Source: §2 · p. 5
- Extra Compute refinement forward-noises the guided output and then denoises it without likelihood guidance to return it to the prior’s data manifold.The authors introduce this step to remove minor artifacts and improve output quality.Source: §2 · p. 5
3 Experiments
Experiments assess fidelity, perceptual realism, and pathology alignment against trained and plug-and-play baselines using synthesized X-rays from CT-Rate cases. PhyDiCT improves perceptual reconstruction while retaining semantic features, with ablations showing trade-offs in view count and guidance.
Experimental Setup
- The evaluation uses 400 CT-Rate cases with synthesized DiffVox and DeepDRR projections, and assesses SSIM, PSNR, FID, and CLIP-based T2I and I2I scores.CT radiology reports serve as proxies for X-ray reports; comparisons include full-training and plug-and-play methods.Source: §3 · pp. 5–6
Quantitative Comparison
- 7.5% improvement in SSIM (PhyDiCT: 0.4268 vs. X2CT: 0.3969) accompanies leading perceptual quality while preserving strong semantic features.X2CT performs slightly better in semantic metrics, while PhyDiCT remains stronger in perceptual quality.Source: Table 1 · p. 7, §3 · p. 7
- Under DeepDRR, all methods lose perceptual metrics and CLIP-I2I, but PhyDiCT's FID and CLIP-I2I remain stable relative to the original source.The authors attribute improvement to LPIPS, which enforces consistency in a high-dimensional feature space.Source: §3 · pp. 7–8
Qualitative Comparison
- PhyDiCT preserves finer structural and pathological details, avoids disease exaggeration, reconstructs fissure lines, and EC further suppresses minor artifacts.Other baselines fail to reconstruct the atelectasis region and fissure lines in the illustrated comparison.Source: Figure 2 · p. 6, §3 · p. 8
Ablation Study
- EC takes under 30 seconds versus 1342 seconds for the main method; δ=0.8 balances SSIM and CLIP-T2I, while increasing views improves both with marginal gains beyond K=8.Larger δ values are generally preferred, but increasing δ up to 0.9 improves SSIM while degrading CLIP-T2I; larger view counts increase computational cost.Source: §3 · p. 8, Table 2 · p. 8, Table 3 · p. 8
- Without report text, SSIM and FID degrade slightly, CLIP-I2I increases, and CLIP-T2I decreases, showing a trade-off between image fidelity and semantic alignment.The authors describe radiological prompts as beneficial.Source: §3 · p. 8
4 Conclusion
PhyDiCT reconstructs 3D lung CT from limited-view X-rays using physical rendering and a text-guided prior, but its stated scope is lung reconstruction.
Conclusion
- PhyDiCT combines physical 3D–2D rendering and a strong text-guided prior for limited-view lung CT, but other organs require corresponding pretrained priors.The conclusion also identifies x0 prediction and LPIPS loss as elements yielding better results.Source: §4 · p. 9