Source-linked AI summary
Difix3D+: Improving 3D Reconstructions with Single-Step Diffusion Models
Jay Zhangjie Wu, Yuxuan Zhang, Haithem Turki, Xuanchi Ren, Jun Gao, Mike Zheng Shou, Sanja Fidler, Zan Gojcic, Huan Ling
TL;DR
Extreme novel views remain artifact-prone because NeRF and 3DGS are underconstrained, while diffusion-guided reconstruction methods can be slow. Difix3D+ uses a single-step diffusion artifact fixer during progressive 3D refinement and inference-time post-processing, achieving a 2× FID improvement over baselines while maintaining 3D consistency.
Problem
NeRF and 3DGS produce artifacts in underconstrained regions and extreme novel views, while existing diffusion-guided approaches are often time-consuming.
Method
Difix3D+ fine-tunes a single-step diffusion model to clean rendered views, distills enhanced pseudo-views progressively into NeRF or 3DGS, and applies the model during inference.
Results
2× improvement in FID scores over baselines while maintaining 3D consistency.
Takeaways & Limitations
A single Difix model works with both NeRF and 3DGS, improving 3D representation quality and enabling near-real-time artifact removal during inference.
Takeaways & Limitations
Direct inference-time Difix enhancement can produce pose and frame inconsistencies in under-observed or noisy regions.
Abstract
from arXiv · showhide
Neural Radiance Fields and 3D Gaussian Splatting have revolutionized 3D reconstruction and novel-view synthesis task. However, achieving photorealistic rendering from extreme novel viewpoints remains challenging, as artifacts persist across representations. In this work, we introduce Difix3D+, a novel pipeline designed to enhance 3D reconstruction and novel-view synthesis through single-step diffusion models. At the core of our approach is Difix, a single-step image diffusion model trained to enhance and remove artifacts in rendered novel views caused by underconstrained regions of the 3D representation. Difix serves two critical roles in our pipeline. First, it is used during the reconstruction phase to clean up pseudo-training views that are rendered from the reconstruction and then distilled back into 3D. This greatly enhances underconstrained regions and improves the overall 3D representation quality. More importantly, Difix also acts as a neural enhancer during inference, effectively removing residual artifacts arising from imperfect 3D supervision and the limited capacity of current reconstruction models. Difix3D+ is a general solution, a single model compatible with both NeRF and 3DGS representations, and it achieves an average 2$\times$ improvement in FID score over baselines while maintaining 3D consistency.
1. Introduction
NeRF and 3DGS enable novel-view synthesis but remain artifact-prone in underconstrained or extreme viewpoints. Difix3D+ adapts single-step diffusion to refine 3D representations and renderings, improving quality efficiently across both representations.
- Motivation: NeRF and 3DGS still produce spurious geometry and missing regions in less observed or extreme novel views.These artifacts persist even with dense captures, varying lighting, or imperfect camera poses and calibration.
- Motivation: Per-scene optimization struggles with shape-radiance ambiguity and lacks data priors for plausible geometry and appearance in underconstrained regions.Such methods can reproduce training images with geometrically incorrect representations and otherwise rely on representation smoothness.
- Approach: Difix adapts a 2D diffusion prior with minimal fine-tuning to remove artifacts from rendered NeRF and 3DGS views.The model is used during reconstruction to create pseudo-training views that are distilled back into 3D.
- Approach: Difix3D+ progressively refines the 3D representation and applies Difix during inference as a near-real-time post-processing enhancer.The pipeline includes 3D optimization, real-time post-rendering, and an updated 3D representation.
- Results: >10× faster than methods querying diffusion at every training step.The approach also supports both implicit NeRF and explicit 3DGS representations.
2. Related Work
Related work addresses reconstruction artifacts, under-observed regions, and generative priors through camera, lighting, geometric, feedforward, GAN, and diffusion-based methods. Diffusion-guided 3D optimization can provide strong priors but is often slow because the model is queried at every optimization step.
- Improving 3D reconstruction discrepancies: NeRF robustness methods optimize camera poses, address lighting variation, or mitigate transient occlusions in noisy captures.These approaches target inconsistencies that produce artifacts or blurriness when data is distilled into 3D.
- Priors for novel view synthesis: Geometric priors use regularization, depth, or normal supervision to improve novel-view synthesis in sparse-view settings.They are sensitive to noise, difficult to balance with data terms, and provide only marginal gains in denser captures.
- Priors for novel view synthesis: Feedforward neural networks learn from posed multi-view data collected across numerous scenes to address limitations in under-observed regions.The passage introduces these methods as an alternative line of work alongside geometric priors.
- Generative priors for novel view synthesis: Generative priors include per-scene GAN enhancement, direct diffusion-based novel-view generation, and diffusion-guided 3D optimization.Diffusion models learn generalizable priors from internet-scale datasets.
- Generative priors for novel view synthesis: Diffusion-guided optimization often queries the diffusion model at every optimization step, significantly slowing training.This computational cost motivates approaches that avoid repeated diffusion queries.
3. Background
NeRF represents scenes as continuous neural fields rendered through volume accumulation, while 3DGS uses parameterized volumetric particles rendered with the same formulation. Diffusion models learn data distributions by denoising progressively noised examples.
- Neural Radiance Fields: NeRF models a scene with an MLP that maps spatial locations to view-dependent radiance and volume density.Rays are sampled through the field and their radiance is accumulated by volume rendering.
- Neural Radiance Fields: NeRF volume rendering accumulates N sampled points along each ray using opacity terms and quadrature step sizes δ_i.The formulation defines α_i from density and step size, with N denoting the number of ray samples.
- 3D Gaussian Splatting: 3DGS represents scenes as volumetric particles parameterized by position, rotation, scale, opacity, and color.Novel views use the same volume-rendering formulation, with contributing Gaussians selected through tile-based rasterization.
- Diffusion Models: Diffusion models learn pdata(x) through iterative denoising and denoising score matching.Training examples are progressively perturbed by Gaussian noise before optimizing the denoiser.
- Diffusion Models: Noised data is formed as x_τ = α_τx + σ_τϵ, where ϵ is Gaussian noise and τ indexes diffusion time.The diffusion-time distribution is often uniform over a discretized range up to τ = 1000.
4. Boosting 3D Reconstruction with DM priors
DIFIX adapts a single-step diffusion model to enhance artifact-ridden NeRF and 3DGS renderings, then progressively distills improved pseudo-views back into the 3D representation. A final render-time enhancement step further improves perceptual quality while maintaining consistency.
- 4.2. DIFIX3D+: NVS with Diffusion Priors: DIFIX-enhanced pseudo-views are progressively distilled into the 3D representation, increasing overlap between reference and target-view 3D cues.This progressive update improves 3D consistency and supports artifact-free target-view renderings.
- 4.1. DIFIX: From a pretrained diffusion model to a 3D Artifact Fixer: DIFIX refines rendered novel views using a single-step SD-Turbo-based image-to-image model conditioned on clean reference views.The model uses a U-Net with cross-view reference mixing, a frozen VAE encoder, and a LoRA fine-tuned decoder.
- 4.1.1 Data Curation: The model is trained with reconstruction, perceptual LPIPS, and Gram-matrix style losses on paired degraded and clean images.The paired data includes artifacts generated through sparse reconstruction, cycle reconstruction, cross-reference training, and model underfitting.
- 4.1. DIFIX: From a pretrained diffusion model to a 3D Artifact Fixer: τ = 200 balances artifact removal and context preservation, whereas τ = 600 alters context and τ = 10 leaves most artifacts intact.The intermediate noise level achieves the highest metrics.
- 4.2. DIFIX3D+: NVS with Diffusion Priors: DIFIX3D+ applies DIFIX again at render time to address residual blur and artifacts while maintaining a high degree of consistency.The single-step post-processing adds 76 ms on an NVIDIA A100 GPU and is over 10× faster than standard multi-step diffusion.
5. Experiments
Experiments evaluate DIFIX3D+ across in-the-wild, automotive, and ablation settings using NeRF and 3DGS backbones. The method consistently improves perceptual and rendering quality, while diagnostics show benefits from progressive updates and post-render enhancement.
- Evaluation setup: The evaluation covers in-the-wild DL3DV and Nerfbusters scenes, automotive RDS scenes, and ablations of the pipeline and DIFIX components.Experiments use held-out target views, compare against base methods and several artifact-removal or enhancement baselines, and report PSNR, SSIM, LPIPS, and FID.
- Main results: Both DIFIX3D+ variants outperform comparison methods across all reported metrics, reducing LPIPS by 0.1 and FID by almost 3× relative to their NeRF and 3DGS backbones.The method also improves PSNR by about 1 dB, indicating stronger pixel-wise fidelity to original views.
- Automotive scenes: RDS experiments show that DIFIX3D+ outperforms its baselines across all metrics and reduces artifacts consistently across views.The RDS DIFIX model was trained on 40 scenes with 100,000 paired samples.
- Pipeline components: Direct DIFIX enhancement improves views near references but performs poorly in less observed regions and causes flickering, whereas incremental 3D updates improve quality significantly.Adding post-render DIFIX further improves outputs, and the complete pipeline yields the metric gains reported in the final ablation row.
- DIFIX components: Reducing diffusion noise, conditioning on reference views, and adding Gram loss each improve DIFIX results on the Nerfbusters dataset.High noise produces more hallucinated pixels that contradict ground truth, while reference conditioning corrects structural inaccuracies and color shifts and Gram loss sharpens details.
6. Conclusion
DIFIX3D+ enhances 3D reconstruction and novel-view synthesis with a near-real-time single-step diffusion model. It improves NeRF and 3DGS representations while maintaining 3D consistency and achieving a 2× FID improvement over baselines.
- Conclusion: DIFIX3D+ combines progressive 3D updates with real-time artifact removal during inference.Its DIFIX enhancer is compatible with both NeRF and 3DGS representations.
- Conclusion: 2× FID improvement over baselines is achieved while maintaining 3D consistency.The conclusion presents this as evidence of improved artifact removal and photorealistic rendering.
Supplementary Material
The supplementary material specifies DIFIX’s 2D supervision losses and points to implementation details, pseudocode, additional results, and limitations discussed elsewhere.
- A.1. Loss Functions: DIFIX is supervised using RGB-space 2D losses rather than requiring 3D supervision.The losses include reconstruction, perceptual, and style terms based on VGG-16 features and Gram matrices.
- A.1. Loss Functions: The final training objective is the weighted sum L = LRecon + LLPIPS + 0.5LGram.The reconstruction and Gram terms use L2-based formulations, while the perceptual term is LPIPS-based.
- Supplementary sections: Additional implementation details, pseudocode, results, and limitations are provided in the referenced supplementary sections.Algorithm 1 is cited for further training details, while Sections B and C contain further results and limitations or future work.
A.3. Evaluation Metrics
The pipeline evaluates novel-view synthesis with image-quality metrics while progressively refining a 3D representation through rendered, diffusion-refined views added back to training. DIFIX is trained on paired corrupted and clean images that simulate artifacts in underconstrained views.
- Evaluation Metrics: PSNR, SSIM, LPIPS, and FID quantify novel-view synthesis quality, with invisible regions masked using a visibility map.PSNR, SSIM, and LPIPS assess image fidelity or perceptual similarity, while FID measures distributional similarity.
- Progressive 3D Updates: Camera poses are perturbed from their nearest reference poses before novel views are rendered and refined.The algorithm identifies the nearest camera pose, applies a pose perturbation, renders a novel view, and refines it with diffusion.
- Progressive 3D Updates: Refined novel views are added to the training set, and the pipeline returns renderings at the target views.This update loop progressively improves the representation before producing final target-view renderings.
- DIFIX Training Data: DIFIX training uses paired corrupted and clean images generated through sparse reconstruction, cycle reconstruction, cross-referencing, and intentional underfitting.The simulated corruptions include blurred details, missing regions, ghosting structures, and spurious geometry.
B.1. Ablation Study of DIFIX
The ablation study attributes DIFIX’s improvements to lower diffusion noise, Gram loss, and reference-view conditioning, which respectively reduce artifacts, sharpen details, and correct structural or color errors.
- DIFIX Design Choices: Lowering the noise level significantly removes artifacts and improves overall visual quality compared with a high-noise setting.The comparison uses a lower noise level against pix2pix-Turbo’s higher-noise configuration.
- DIFIX Design Choices: Gram loss enhances fine details and sharpens the image.
- DIFIX Design Choices: Reference-view conditioning corrects structural inaccuracies and alleviates color shifts.
B.2. Evaluation of Multi-View Consistency
DIFIX3D+ improves multi-view consistency in novel-view synthesis while its final post-processing step enhances image sharpness without compromising 3D coherence.
- Consistency Evaluation: Higher TSED scores indicate better multi-view consistency, and DIFIX3D+ achieves higher scores than reconstruction-based methods and other baselines.TSED quantifies the number of consistent frame pairs in a sequence.
- Consistency Evaluation: The final DIFIX3D+ post-processing step enhances image sharpness without compromising 3D coherence.
C. Limitation and Future Work
DIFIX3D+ depends on the quality of the initial 3D reconstruction and currently struggles when that reconstruction has entirely failed. Future work includes stronger diffusion priors and single-step video diffusion for long-context consistency.
- Limitation: DIFIX3D+ is inherently limited by the quality of the initial 3D reconstruction and struggles to enhance views where reconstruction has entirely failed.
- Future Work: Future directions include integrating modern diffusion priors and scaling DIFIX to single-step video diffusion for enhanced long-context 3D consistency.