Source-linked AI summary
DiffNR: Diffusion-Enhanced Neural Representation Optimization for Sparse-View 3D Tomographic Reconstruction
Shiyan Su, Ruyi Zha, Danli Shi, Hongdong Li, Xuelian Cheng
TL;DR
Sparse-view CT neural representations suffer artifacts in underconstrained regions, motivating a more effective diffusion-enhanced optimization strategy. DiffNR uses SliceFixer, a conditioned single-step diffusion model that periodically creates pseudo-reference volumes for perceptual supervision. It improves reconstruction quality by 3.99 dB, generalizes across domains, and maintains efficient optimization.
Problem
Neural representations model CT volumes effectively but suffer severe artifacts in underconstrained regions under sparse-view acquisition.
Method
DiffNR finetunes SliceFixer, a conditioned single-step diffusion model, and periodically uses its repaired slices to create pseudo-reference volumes for NR optimization.
Results
3.99 dB improvement in NR reconstruction quality is reported, with generalization across domains and reasonable runtime.
Takeaways & Limitations
DiffNR combines diffusion-based artifact correction with neural-representation optimization while avoiding frequent diffusion queries.
Takeaways & Limitations
Real-world projections contain noise from physical effects and hardware imperfections.
Abstract
from arXiv · showhide
Neural representations (NRs), such as neural fields and 3D Gaussians, effectively model volumetric data in computed tomography (CT) but suffer from severe artifacts under sparse-view settings. To address this, we propose DiffNR, a novel framework that enhances NR optimization with diffusion priors. At its core is SliceFixer, a single-step diffusion model designed to correct artifacts in degraded slices. We integrate specialized conditioning layers into the network and develop tailored data curation strategies to support model finetuning. During reconstruction, SliceFixer periodically generates pseudo-reference volumes, providing auxiliary 3D perceptual supervision to fix underconstrained regions. Compared to prior methods that embed CT solvers into time-consuming iterative denoising, our repair-and-augment strategy avoids frequent diffusion model queries, leading to better runtime performance. Extensive experiments show that DiffNR improves PSNR by 3.99 dB on average, generalizes well across domains, and maintains efficient optimization.
Introduction
Sparse-view CT reduces radiation exposure but makes reconstruction underconstrained and artifact-prone. DiffNR addresses this by augmenting neural-representation optimization with conditioned diffusion-based slice repair, improving quality while retaining efficiency.
- Sparse-view CT seeks high-quality 3D recovery from only a few dozen projections to reduce radiation exposure.
- Neural representations model CT volumes with learnable fields or 3D Gaussians but produce artifacts in underconstrained regions.
- DiffNR enhances a global neural representation with conditioned diffusion rather than embedding local solvers into unconditional denoising.The design promotes unified volumetric consistency and enables finetuning 2D foundation models.
- SliceFixer is finetuned as a single-step diffusion model on curated clean and corrupted slice pairs, using biplanar projections as additional conditioning.During reconstruction, it periodically generates pseudo-reference volumes for guiding underconstrained regions with perceptual supervision.
- 3.99 dB improvement in NR reconstruction quality is reported alongside cross-domain generalization and reasonable runtime.
- DiffNR combines neural representations with diffusion priors and reports gains in accuracy, generalization, and efficiency over existing methods.
Related Work
Learning-based CT reconstruction increasingly favors optimization frameworks that enforce measurement consistency, with neural representations and neural priors forming the main categories. DiffNR differs from prior diffusion-NR hybrids by globally enhancing an NR with diffusion-based augmentation rather than using diffusion as a local solver or frequent scorer.
- Learning-based CT methods are broadly divided into direct feedforward reconstruction and iterative optimization frameworks.Optimization approaches receive increasing attention because direct regression lacks physical constraints.
- Neural representations learn neural fields or 3D Gaussians through optimization, while neural-prior methods combine solvers with pretrained networks.
- State-of-the-art neural-prior approaches plug traditional local solvers into unconditional diffusion models, while early diffusion-NR hybrids adapt NRs as local solvers.
- Some diffusion-enhanced methods query diffusion models at every optimization step, significantly compromising efficiency.
- DiffNR follows a repair-and-augment strategy that uses finetuned diffusion models to repair NR outputs and generate pseudo-observations without frequent diffusion queries.
Background
The paper formulates cone-beam CT reconstruction as recovering a density field from multi-angle projections and reviews neural fields, 3D Gaussians, and diffusion models as relevant representations and priors. SliceFixer supplies conditioned single-step slice refinement within this reconstruction setting.
- Cone-beam CT models X-rays traveling through a 3D density field, with reconstruction recovering that field from multi-angle projections.The method can also be adapted to parallel-beam geometry.
- Logarithmic projections are used as inputs, and real-world projections contain noise from physical effects and hardware imperfections.
- SliceFixer: SliceFixer takes an NR-queried CT slice with biplanar projections and a text prompt, then outputs a refined artifact-free slice.It uses SD-Turbo with trainable LoRA layers and zero convolutions.
- Neural Representations: Neural representations optimize a 3D model through differentiable rendering; neural fields query an MLP to produce density at spatial locations.
- Neural Representations: R2-Gaussian represents the density field as a mixture of learnable 3D Gaussians and offers faster reconstruction than neural-field methods.Each Gaussian has learnable density, center, and covariance parameters.
- Diffusion Models: Diffusion models learn data distributions through denoising, while SD-Turbo distills multi-step denoising into single-step generation.
Proposed Method
DiffNR integrates a conditioned, single-step diffusion model into neural-representation optimization for sparse-view CT. SliceFixer repairs queried slices into pseudo-reference volumes, which provide periodic 3D perceptual supervision while preserving optimization efficiency.
- Overview: DiffNR combines neural fields or 3D Gaussians with diffusion-based augmentation to reconstruct sparse-view CT volumes.The framework targets underconstrained regions prone to artifacts.
- SliceFixer: SliceFixer predicts refined CT slices from NR-queried slices using a single-step diffusion model built on SD-Turbo.The model is adapted for artifact correction in volumetric reconstruction.
- SliceFixer: SliceFixer is conditioned on a text prompt and two orthogonal X-ray projections to preserve anatomical structure and provide global structural cues.RAD-DINO encodes radiograph features, which are aggregated with text embeddings through cross-attention.
- SliceFixer: Training uses L2, LPIPS, CLIP alignment, adversarial, and SSIM losses to guide artifact correction and perceptual quality.The objective combines standard diffusion losses with structural similarity supervision.
- Data Curation: DiffNR curates paired training slices by synthesizing sparse-view projections, varying view distributions, underfitting neural representations, and mixing neural fields with 3D Gaussians.These strategies create diverse artifact patterns and discourage representation-specific overfitting.
- Optimization: Every ℓ iterations, repaired slices are stacked into a pseudo-reference volume and periodically compared with queried volumes using 3D SSIM regularization.The repair-and-augment strategy reduces SliceFixer query frequency while promoting structural coherence across axial, sagittal, and coronal planes.
Experiments
DiffNR is evaluated on in-distribution and out-of-distribution datasets, downstream lung segmentation, and ablations of SliceFixer and DiffNR design choices. Results show improved reconstruction quality, generalization, and practical utility while retaining efficiency relative to prior diffusion-based methods.
- Experimental Setup: Experiments use ToothFairy and LUNA16, with 36-, 24-, and 12-view sparse-view settings and separate SliceFixer models trained per dataset.The datasets contain 443 dental scans and 888 chest scans, with 256^3 volumes and 256^2 projections.
- Experimental Setup: DiffNR is compared with traditional iterative, self-supervised neural-representation, and diffusion-based iterative reconstruction methods using PSNR and SSIM.The comparison includes SART, ASD-POCS, SAX-NeRF, NAF, R2-Gaussian, DDS, and DiffusionMBIR.
- In-Distribution Performance: +2.19 dB in PSNR for NAF and +5.79 dB for R2-Gaussian are the average in-distribution improvements reported for DiffNR.DiffusionMBIR requires 11 hours per case, while DiffNR remains substantially faster despite additional optimization time.
- Out-of-Distribution Performance: DiffNR suppresses hallucinations and artifacts on an OOD dataset spanning 18 diverse real-world cases, indicating that SliceFixer learns generalizable artifact patterns.The OOD cases include human organs, biological specimens, and artificial objects.
- Downstream Application: Gaussian-based DiffNR produces lung segmentation masks more consistent with ground-truth masks, demonstrating utility for downstream medical tasks.Segmentation is evaluated with Dice and average surface distance on LUNA16 reconstructions.
- Ablation Study: A 0.3 dB PSNR gain comes from SSIM loss, while biplanar conditioning adds 0.6 dB; 512^2 finetuning resolution and up-downsampling also improve quality.The DiffNR ablation finds slice supervision more stable than novel-view augmentation, while standalone SliceFixer causes slice jitter and hallucinations and voxel-wise L1 loss reduces performance.
- Ablation Study: λdiff = 0.5 and τ = 10 provide the best reported balance between 3D guidance, projection fitting, hallucination avoidance, structural regularization, and computation.More frequent supervision increases computational cost, whereas sparse supervision weakens structural regularization and degrades performance.
Conclusion
DiffNR uses SliceFixer, a single-step diffusion model, to correct artifacts in reconstructed CT slices and generate pseudo-reference volumes for augmented perceptual regularization. This repair-and-augment strategy improves reconstruction quality, generalization, and optimization efficiency.
- SliceFixer is a single-step diffusion model finetuned on curated datasets to correct artifacts in reconstructed CT slices.
- During reconstruction, SliceFixer generates pseudo-reference volumes that provide augmented perceptual regularization.
- DiffNR’s repair-and-augment strategy avoids frequent diffusion model queries, improving reconstruction quality without sacrificing efficiency.
- Experimental results show that DiffNR outperforms prior methods in reconstruction quality, generalization capability, and optimization efficiency.