Source-linked AI summary
CoR-GS: Sparse-View 3D Gaussian Splatting via Co-Regularization
Jiawei Zhang, Jiahe Li, Xiaohan Yu, Lei Huang, Lin Gu, Jin Zheng, Xiao Bai
TL;DR
Sparse-view 3DGS can overfit and produce unrealistic novel views, while existing approaches may rely on external supervision. CoR-GS trains two Gaussian radiance fields, measures their point and rendering disagreements, and suppresses inaccurate reconstruction through co-pruning and pseudo-view co-regularization. It achieves state-of-the-art sparse-view novel view synthesis across multiple datasets while reconstructing coherent, compact geometry.
Problem
Sparse training views make 3DGS prone to overfitting and unrealistic novel-view synthesis, while sparse-view behavior remains less investigated.
Method
CoR-GS trains two radiance fields and suppresses point and rendering disagreement using co-pruning and pseudo-view co-regularization.
Results
CoR-GS achieves state-of-the-art sparse-view novel view synthesis across LLFF, Mip-NeRF360, DTU, and Blender while reconstructing coherent, compact geometry.
Takeaways & Limitations
Point and rendering disagreement can identify inaccurate reconstruction without ground-truth information, supporting co-regularization of sparse-view 3DGS.
Abstract
from arXiv · showhide
3D Gaussian Splatting (3DGS) creates a radiance field consisting of 3D Gaussians to represent a scene. With sparse training views, 3DGS easily suffers from overfitting, negatively impacting rendering. This paper introduces a new co-regularization perspective for improving sparse-view 3DGS. When training two 3D Gaussian radiance fields, we observe that the two radiance fields exhibit point disagreement and rendering disagreement that can unsupervisedly predict reconstruction quality, stemming from the randomness of densification implementation. We further quantify the two disagreements and demonstrate the negative correlation between them and accurate reconstruction, which allows us to identify inaccurate reconstruction without accessing ground-truth information. Based on the study, we propose CoR-GS, which identifies and suppresses inaccurate reconstruction based on the two disagreements: (1) Co-pruning considers Gaussians that exhibit high point disagreement in inaccurate positions and prunes them. (2) Pseudo-view co-regularization considers pixels that exhibit high rendering disagreement are inaccurate and suppress the disagreement. Results on LLFF, Mip-NeRF360, DTU, and Blender demonstrate that CoR-GS effectively regularizes the scene geometry, reconstructs the compact representations, and achieves state-of-the-art novel view synthesis quality under sparse training views.
1 Introduction
Sparse-view 3DGS can overfit because limited training constraints permit unrealistic novel-view reconstructions. CoR-GS uses disagreement between two radiance fields to identify inaccurate geometry and rendering, then suppresses those disagreements during training.
- Sparse training views reduce constraints, making 3DGS prone to overfitting and unrealistic novel-view synthesis.
- Two radiance fields trained on the same sparse views differ in Gaussian positions and rendered pixels, with differences increasing during random densification.Densification creates new Gaussians by sampling positions from a normal distribution.
- CoR-GS trains two radiance fields jointly and identifies inaccurate reconstruction using point disagreement and rendering disagreement.
- Co-pruning matches the two Gaussian point clouds and removes Gaussians without nearby counterparts, while pseudo-view co-regularization penalizes disagreeing rendered pixels.Pseudo views are sampled by interpolating training views, and pixel differences are combined with the training-view loss.
- CoR-GS reconstructs coherent, compact geometry and achieves state-of-the-art sparse-view rendering performance across LLFF, Mip-NeRF360, DTU, and Blender.
- Point disagreement and rendering disagreement provide ground-truth-free indicators of reconstruction quality because both are negatively correlated with accurate reconstruction.
2 Related Work
Prior work established radiance-field reconstruction and addressed sparse-view degradation with regularization, generative models, external knowledge, and depth supervision. CoR-GS instead introduces co-regularization between two Gaussian radiance fields for sparse-view 3DGS.
- Radiance Fields: 3DGS represents scenes with anisotropic 3D Gaussians and renders complex scenes in real time, but its sparse-view behavior remains less investigated.
- Sparse-View Reconstruction: Sparse-view novel view synthesis often produces degraded scenes and unrealistic views, motivating regularization methods for NeRFs and 3DGS.
- Sparse-View Reconstruction: Existing sparse-view 3DGS methods use predictions from pretrained depth estimators to regularize geometry, but external supervision can introduce additional noise.
- Agreement-Based Regularization: Agreement between independently trained networks has been used to pseudo-label unlabeled data and clean noisy annotations, providing precedent for agreement-based regularization.
3 Point Disagreement and Rendering Disagreement
Two 3D Gaussian radiance fields trained on the same sparse views can diverge in both Gaussian positions and rendered pixels. These disagreements increase during densification and correlate negatively with reconstruction accuracy, especially in regions where the fields disagree.
- 3.1 Preliminary and Definition: Each Gaussian primitive is parameterized by its center, scale, rotation, opacity, and color feature.Its spatial influence is modeled with a covariance determined by scale and rotation, while rendering uses alpha-blending.
- 3.2 Empirical Study: Point disagreement and rendering disagreement increase during training, especially during densification.Densification creates new Gaussians and initializes their positions by sampling from a normal distribution.
- 3.2 Empirical Study: Two radiance fields trained on the same sparse views can exhibit different Gaussian positions and rendered pixels.The fields are simultaneously trained and their rendering disagreement is evaluated at unseen views.
- 3.2 Empirical Study: Higher disagreement is negatively correlated with accurate reconstruction of the scene.Masking regions with the highest disagreement scores improves the reconstruction quality averaged over the remaining regions.
- 3.2 Empirical Study: Disagreed rendered regions tend to be inaccurate compared with ground truth.This relationship is observed in both rendered images and depth maps with corresponding error maps.
4 Method
CoR-GS trains two Gaussian radiance fields and suppresses their point and rendering disagreements through co-pruning and pseudo-view co-regularization.
- CoR-GS simultaneously trains two 3D Gaussian radiance fields and regularizes them using point and rendering disagreement.
- 4.1 Co-pruning: Co-pruning matches Gaussians across the two fields and removes points without nearby counterparts, treating them as inaccurately positioned outliers.Matching is based on nearest Gaussians, with a permissible maximum distance τ = 5.
- 4.2 Pseudo-view Co-regularization: Pseudo-view co-regularization samples views between nearby training cameras and suppresses disagreement between the two rendered images.The pseudo-view uses interpolated camera locations with random normal noise and an averaged rotation.
- 4.2 Pseudo-view Co-regularization: The pseudo-view color regularizer combines L1 and D-SSIM losses with balance weight λ = 0.2.
- The final training loss combines supervised reconstruction at training views with pseudo-view color co-regularization, using λp = 1.0.
5 Experiments
Across four datasets, CoR-GS improves sparse-view reconstruction quality and produces coherent, compact geometry, while adding training cost but reducing inference cost.
- LLFF: CoR-GS achieves the best reported performance across LLFF PSNR, SSIM, LPIPS, and AVGE for 3, 6, and 9 training views.
- Qualitative comparison: Qualitative comparisons show CoR-GS reconstructs compact scene geometry on LLFF, clearer non-central structures on Mip-NeRF360, and more complete objects on DTU.
- Mip-NeRF360: On Mip-NeRF360, CoR-GS achieves the best PSNR, SSIM, and AVGE with 12 and 24 training views, while vanilla 3DGS leads LPIPS at 12 views.
- Efficiency: CoR-GS reduces the number of Gaussians by 33% and achieves the most efficient inference in the LLFF 3-view efficiency comparison.Training remains more expensive than vanilla 3DGS because CoR-GS trains two fields and renders pseudo views, but it remains more efficient than NeRFs.
- Ablation: Ablations show that suppressing either disagreement improves sparse-view 3DGS, while combining both achieves the best performance.The visualization attributes complementary effects to co-pruning and pseudo-view co-regularization.
6 Conclusion
The paper presents co-regularization as a way to identify inaccurate sparse-view reconstructions from disagreement between two Gaussian radiance fields and improve novel-view synthesis.
- CoR-GS uses point and rendering disagreement to identify and suppress inaccurate reconstruction without ground-truth information.
- Experiments across LLFF, Mip-NeRF360, DTU, and Blender report state-of-the-art sparse-view novel view synthesis.
Overview
The paper organizes additional analyses, method details, comparisons, visualizations, future work, and experiment setup into sections A through F.
- Section A presents additional disagreement results, while Section B discusses hyperparameters and method design.
- Sections C and D provide additional comparison and visualization results, respectively.
- Section E discusses future work, and Section F provides further experiment setup details.
A Additional Results of Disagreement
Additional experiments show that disagreement between radiance fields increases during densification and is negatively correlated with reconstruction quality. Suppressing these disagreements improves reconstruction quality and yields more compact, dense-view-like geometry.
- Disagreement trends: Disagreements between two radiance fields increase significantly during densification across varied scenes.CoR-GS effectively suppresses both point and rendering disagreement.
- Quality correlation: Reducing regions with high disagreement continuously improves the average reconstruction quality of the remaining regions.This relationship is reported for both disagreement measures on the 3-view LLFF setting.
- Quality correlation: Suppressing both disagreements improves reconstruction quality over vanilla 3DGS across regions with different disagreement scores.The improvement is reported across additional scenes.
- Geometry: CoR-GS reconstructs more compact point-cloud representations that more closely resemble dense-view representations.Dense-view representations are obtained by training radiance fields with dense views.
B Additional Ablation Results
Ablations support the combined co-regularization design: moderate co-pruning thresholds work best, color regularization largely subsumes depth regularization, and the method extends across depth-regularized and generalized settings.
- Co-pruning threshold: τ = 5 and τ = 10 achieve the best co-pruning results, while τ = 3 is too strict and τ = 30 imposes loose constraints.The threshold controls when Gaussians are treated as having no matching point.
- Color and depth regularization: Depth co-regularization alone helps, but its additional impact becomes very weak when combined with color co-regularization.Color co-regularization also constrains depth information through the sorting process used for rendering, so depth-map co-regularization is omitted.
- Multiple radiance fields: Using more than two radiance fields makes disagreement reflect inaccurate reconstruction more accurately and produces further improvements.Co-pruning and pseudo-view co-regularization compare each field with the others.
- Depth-regularized 3DGS: CoR-GS works with depth regularization and shows a particular advantage on image structural SSIM.The integration is evaluated with the depth-regularized method FSGS.
- Qualitative comparisons: Compared with FreeNeRF, CoR-GS reconstructs more realistic high-frequency details and better geometry for thin structures.The comparison covers 3-view LLFF, 3-view DTU, and 8-view Blender.
F.1 Dataset Split
The evaluation uses sparse-view splits across LLFF, DTU, Mip-NeRF360, and Blender, with dataset-specific novel-view settings and input-view sampling procedures.
- LLFF: LLFF contains 8 forward-facing scenes, using every eighth image as novel views and evenly sampling input views from the remainder.Images are downsampled 8× to 378 × 504 resolution.
- Qualitative settings: The qualitative results include 3-view DTU, 3-view LLFF, 24-view Mip-NeRF360, and 8-view Blender settings.These settings are visualized in Figs. VII–X.