Source-linked AI summary
DualDiff3D: Dual Structure-Appearance Diffusion Priors for Reliability-Enhanced 3D Gaussian Splatting
Qian Wang, Yu Wang, Weiqi Li, Xinhua Cheng, Xiandong Meng, Ronggang Wang, Jian Zhang
TL;DR
Sparse-view 3DGS produces unreliable reconstructions and artifacts, while single-network diffusion refinement conflates structural and appearance information. DualDiff separates those roles with dual diffusion branches and SAA, and DualDiff3D integrates refinement into a reliability-enhanced RRO loop. The framework improves reconstruction and NVS quality even without training, with further gains from fine-tuning.
Problem
Sparse input views degrade 3DGS reconstruction quality, and existing single-network reference fusion overlooks the different structural and appearance roles of novel and reference views.
Method
DualDiff uses separate structure and appearance diffusion branches linked by SAA, while DualDiff3D incorporates refined views through a reliability-enhanced RRO loop.
Results
DualDiff3D improves objective and subjective reconstruction and NVS quality in inference-only use, with further performance gains attainable through fine-tuning.
Takeaways & Limitations
The framework provides a reference-guided diffusion solution that progressively and robustly refines 3DGS through increasingly high-quality novel-view constraints.
Takeaways & Limitations
Despite fast one-step inference, DualDiff3D still requires substantial computational resources and may degrade significantly in extremely sparse-view scenarios.
Abstract
from arXiv · showhide
While 3D Gaussian Splatting (3DGS) has revolutionized 3D reconstruction and novel-view synthesis, scenarios with limited input views often lead to poor reconstruction quality and artifacts in rendered novel views. Recent efforts attempt to utilize powerful diffusion priors, yet they typically process rendered and reference views concatenated along an additional dimension in a single network. These methods overlook an inherent nature that different views should maintain appearance similarity but differ in structure due to view shifts, leading to blur caused by conflicts between the two properties. In this paper, we propose DualDiff, a novel pipeline that leverages dual diffusion priors with a Structure-Appearance Attention (SAA) module to introduce reference guidance for refining low-quality novel views rendered from flawed 3D representations. Specifically, we retain one diffusion branch to focus on extracting structural information from the low-quality novel views, while introducing another branch to ensure appearance consistency with reference views. Furthermore, we present a 3D reconstruction framework named DualDiff3D, which integrates a reliability-enhanced Render-Refine-Optimize (RRO) loop to progressively and robustly incorporate the refined novel views, yielding more accurate 3DGS. Extensive experiments demonstrate that our approach outperforms state-of-the-art methods even in the inference-only setting, with further performance gains achievable through training. Our code and pre-trained weights are available at https://github.com/Akaneqwq/DualDiff3D.
1 Introduction
Sparse input views make 3DGS novel-view reconstruction unreliable, while single-network diffusion refinement can conflict between structural and appearance consistency. DualDiff separates these roles with dual diffusion branches and SAA, and DualDiff3D incorporates refinements through a reliability-enhanced RRO loop.
- Sparse viewpoints leave 3DGS reconstructions vulnerable to insufficient information, especially for novel views far from the inputs.
- Single-network reference fusion can conflict with the distinct structural and appearance requirements of rendered novel and reference views.Novel views should preserve perspective structure, whereas reference views provide warped appearance content for artifact regions.
- DualDiff uses separate diffusion branches for extracting novel-view structure and maintaining appearance consistency with reference views, connected by SAA.The appearance branch contributes hidden states to the structure branch’s attention computation.
- DualDiff3D adds a reliability-enhanced Render-Refine-Optimize loop with Progressive Sampling and Filtering and Confidence-Driven Weighting.The loop progressively incorporates increasingly high-quality novel-view constraints into the 3D representation.
- DualDiff3D improves objective and subjective reconstruction and NVS quality in inference-only use, with further gains attainable through fine-tuning.
2 Related Works
Diffusion priors address sparse-view and geometric-consistency challenges in novel-view synthesis, but existing approaches trade computational efficiency against cross-view refinement capabilities.
- Diffusion research also includes general image-generation advances such as latent-space diffusion and text-conditioned models.
- Diffusion priors have become important for improving novel-view synthesis under sparse inputs and geometric inconsistency.
- DIFIX3D+ trains a single-step diffusion model to remove artifacts caused by suboptimal 3D representations.
3 Preliminary
3DGS represents scenes with anisotropic Gaussians, projects them into image space for efficient rendering, and builds on diffusion models that learn denoising transformations.
- 3.1 3D Gaussian Splatting: 3DGS represents a scene with anisotropic Gaussians parameterized by position, covariance, scale, rotation, spherical-harmonic color, and opacity.
- 3.1 3D Gaussian Splatting: An efficient 3D-to-2D mapping projects each Gaussian into image space using transformed means and covariances.The projected mean and covariance are computed from projective and viewing transformations and an affine Jacobian.
- 3.1 3D Gaussian Splatting: Pixel colors are rendered by compositing the colors and opacities of Gaussians overlapping each image-plane pixel.
- 3.2 Diffusion Model: Diffusion models define a forward noising process and learn an inverse denoising process that predicts the added noise or reverse-process mean.
- 3.2 Diffusion Model: DualDiff uses dual diffusion branches and SAA to inject reference guidance into refinement of low-quality views rendered from defective 3D representations.
4 DualDiff
DualDiff separates structural and appearance guidance across two diffusion branches, using SAA to exchange information while training with reconstruction, perceptual, and detail-sharpening losses.
- 4 DualDiff: DualDiff uses dual branches to decouple structural alignment from appearance alignment, drawing on architectures used in virtual try-on and image animation.The design parallels separate alignment of model structure and garment appearance in virtual try-on.
- 4 DualDiff: The structure and appearance branches use SD-Turbo-like denoising U-Nets initialized with DIFIX weights for low-noise, one-step refinement.Rendered novel views replace Gaussian-noise inputs, with timestep t = 200.
- 4 DualDiff: SAA injects appearance-branch information into structure-branch attention by combining hidden states before producing keys and values.Structure-branch queries preserve the novel-view stream while appearance features provide reference guidance.
- 4 DualDiff: Training combines L2 reconstruction, perceptual LPIPS, and VGG-16 Gram-Matrix losses to improve fidelity, visual quality, and sharp details.The Gram-Matrix term is weighted by λ0 = 0.1.
5 DualDiff3D
DualDiff3D uses a reliability-enhanced RRO loop to add refined novel views progressively, admitting and weighting them through confidence checks and stabilizing optimization with validation rollback.
- 5 DualDiff3D: The RRO loop renders novel views from an initial low-quality 3DGS representation, refines them with DualDiff, and incorporates them into subsequent optimization.This augments the 3DGS training set for sparse-view reconstruction.
- 5 DualDiff3D: Progressive Sampling and Filtering expands the training set through camera interpolation and slight extrapolation, retaining only views with strong geometric and appearance consistency.Sampled views are evaluated at both pixel and image levels using a composite confidence mask.
- 5 DualDiff3D: Confidence-Driven Weighting incorporates only high-confidence regions from refined novel views using a composite mask of three complementary confidence measures.The mask is defined over image pixels and combines confidence cues used during 3DGS training.
- 5 DualDiff3D: Refinement variance, pixel discrepancy, and reprojection consistency provide confidence cues for weighting novel-view regions.Variance is estimated across repeated refinements, while discrepancy compares refined output with the original rendering and reprojection compares against reference images.
- 5 DualDiff3D: Optimization mixes original and refined views and uses held-out validation with rollback to discard newly added views when reconstruction quality deteriorates.The refined-view loss balances pixel-wise and perceptual terms.
6 Experiments
Experiments evaluate DualDiff and DualDiff3D across datasets, sparse-view settings, baselines, ablations, and runtime. The results show strong refinement and reconstruction quality, while the reliability-enhanced pipeline improves optimization stability and efficiency.
- Implementation Details: DualDiff is trained on DL3DV scenes using rendered views, ground truths, and closest training views across multiple sparse-view counts.DualDiff-s uses 24-view training results, whereas DualDiff-m uses 3-, 6-, 9-, and 24-view results.
- Refinement Results: DualDiff is evaluated on DL3DV and LLFF against vanilla 3DGS and DIFIX for novel-view refinement.Testing covers 3, 6, 9, and 24 sparse input views on DL3DV.
- Refinement Results: DualDiff outperforms vanilla 3DGS and DIFIX by a large margin across all experimental settings.Subjective results show fewer artifacts, finer details, and stronger appearance consistency with reference views.
- Reconstruction Results: DualDiff3D achieves the highest objective metric values under all view-count conditions on DL3DV and LLFF.Visual results indicate a balance between photorealism and geometric coherence, including on LLFF despite no LLFF refinement training.
- Ablation Study: The full DualDiff3D model performs best in component ablations, while K = 3 provides a quality/cost trade-off and three CDW terms perform best together.The ablations remove or replace the refinement model, PSF strategy, and CDW process under the LLFF 3-view setting.
- Runtime Analysis: Despite its larger backbone, DualDiff3D has lower average single-refinement runtime than DIFIX3D.The comparison attributes this to DIFIX3D processing only one input view at a time through its batch-dimension design.
7 Conclusion
DualDiff3D combines dual-branch diffusion refinement with a reliability-enhanced Render-Refine-Optimize loop for 3D reconstruction and novel-view synthesis. It delivers superior visual quality, but remains computationally demanding and can degrade in extremely sparse-view scenarios.
- Conclusion: DualDiff3D uses DualDiff, an SAA-connected dual-branch diffusion pipeline, within a Render-Refine-Optimize loop with sampling and confidence-guided masks.The framework progressively refines 3DGS while reducing artifacts in rendered novel views.
- Conclusion: Extensive experiments validate DualDiff3D’s effectiveness and show superior visual quality compared with baselines.The conclusion presents the framework as a solution for diffusion-based novel-view synthesis and 3D reconstruction.
- Limitations: DualDiff3D still imposes computational-resource overhead despite its one-step diffusion architecture and fast inference speed.The authors identify model distillation as a future direction.
- Limitations: Performance may suffer significant degradation in extremely sparse-view scenarios.The authors propose extending the method to extremely sparse input settings as future work.
A More Subjective Results
Additional subjective comparisons evaluate DualDiff refinement and DualDiff3D reconstruction across DL3DV and LLFF. The reported visual comparisons emphasize improved image quality, edge sharpness, and artifact removal.
- Subjective Refinement Results: DualDiff outperforms DIFIX in subjective image quality across indoor and outdoor scenes on DL3DV and LLFF.The comparison covers both datasets and varied scene types.
- Subjective Reconstruction Results: DualDiff3D enhances reconstruction quality on both training and out-of-distribution datasets.The reported improvements are particularly visible in edge sharpness and artifact removal.
B Generalization Ability
Generalization experiments test DualDiff3D under different resolutions, view counts, initializations, and base reconstruction methods. The results indicate robust performance with poor initialization and stable plug-and-play improvements when reliability filtering is used.
- Experimental Settings: Generalization is evaluated on LLFF at one-quarter resolution with 3, 6, and 9 input views.The experiments test standard 3DGS initialization, random initialization, and integration with FSGS and DropGaussian.
- Initialization Quality: DualDiff3D achieves strong performance without reliable geometric initialization from a random point cloud.Using 3DGS initialization can produce worse final metrics because sparse-view artifacts create a large rectification cost.
- Integration with Base Models: DualDiff3D stably and significantly improves reconstruction quality when integrated into FSGS and DropGaussian.DIFIX3D is unstable and degrades performance in some cases.
- Reliability Filtering: PSF and CDW suppress hallucinations and misalignment by ensuring that only reliable refined information enters 3DGS optimization.This reliability mechanism supports DualDiff3D as a plug-and-play module across different base models.
C User Study
A 30-participant user study evaluated the method and baselines on image quality and cross-view consistency. Participants selected the best image for each criterion, and selection proportions were calculated.
- 30 participants evaluated 10 sets of reference and novel views from the proposed method and baseline approaches.
- The study assessed two criteria: image quality and cross-view consistency.
- Participants selected the best image for each criterion, and the researchers calculated each method’s selection proportion.
D Multi-view Consistency
DualDiff3D achieves better multi-view consistency than 3DGS and DIFIX3D across all tested view-count settings on LLFF and DL3DV. The evaluation uses MEt3R, where lower scores indicate better geometric and appearance consistency.
- DualDiff3D consistently outperforms 3DGS and DIFIX3D under all view-count settings on both LLFF and DL3DV.
- MEt3R measures multi-view consistency, with lower scores indicating better geometric and appearance consistency across views.
- Fig. D.2 compares subjective results of DualDiff3D and baselines on the DL3DV and LLFF datasets.