Source-linked AI summary
PGSR: Planar-based Gaussian Splatting for Efficient and High-Fidelity Surface Reconstruction
Danpeng Chen, Hai Li, Weicai Ye, Yifan Wang, Weijian Xie, Shangjin Zhai, Nan Wang, Haomin Liu, Hujun Bao, Guofeng Zhang
TL;DR
3DGS renders efficiently but struggles to produce geometrically accurate, multi-view-consistent surfaces because its Gaussian representation is irregular. PGSR introduces unbiased plane-based depth rendering, geometric regularization, and exposure compensation. Across indoor and outdoor evaluations, it achieves high geometric reconstruction accuracy with competitive rendering quality while retaining efficient training and rendering.
Problem
3DGS image reconstruction loss alone does not reliably ensure geometric accuracy and multi-view consistency because Gaussian point clouds are unstructured and irregular.
Method
PGSR compresses Gaussians into planes, derives unbiased depth from plane distance and normal maps, and applies single-view, multi-view, and exposure-compensation constraints.
Results
PGSR achieves the highest geometric reconstruction accuracy and competitive rendering quality compared with state-of-the-art methods on MipNeRF360, DTU, and TnT.
Takeaways & Limitations
The planar representation preserves high-quality Gaussian rendering while supporting precise global geometric reconstruction and fast training and rendering.
Takeaways & Limitations
PGSR produces incomplete or less accurate geometry with missing or limited viewpoints and does not address reflective surfaces or mirrors.
Abstract
from arXiv · showhide
Recently, 3D Gaussian Splatting (3DGS) has attracted widespread attention due to its high-quality rendering, and ultra-fast training and rendering speed. However, due to the unstructured and irregular nature of Gaussian point clouds, it is difficult to guarantee geometric reconstruction accuracy and multi-view consistency simply by relying on image reconstruction loss. Although many studies on surface reconstruction based on 3DGS have emerged recently, the quality of their meshes is generally unsatisfactory. To address this problem, we propose a fast planar-based Gaussian splatting reconstruction representation (PGSR) to achieve high-fidelity surface reconstruction while ensuring high-quality rendering. Specifically, we first introduce an unbiased depth rendering method, which directly renders the distance from the camera origin to the Gaussian plane and the corresponding normal map based on the Gaussian distribution of the point cloud, and divides the two to obtain the unbiased depth. We then introduce single-view geometric, multi-view photometric, and geometric regularization to preserve global geometric accuracy. We also propose a camera exposure compensation model to cope with scenes with large illumination variations. Experiments on indoor and outdoor scenes show that our method achieves fast training and rendering while maintaining high-fidelity rendering and geometric reconstruction, outperforming 3DGS-based and NeRF-based methods.
I. INTRODUCTION
The paper addresses the difficulty of obtaining geometrically accurate, multi-view-consistent surfaces from 3DGS while retaining its rendering and speed advantages. PGSR combines unbiased plane-based depth rendering with geometric regularization and exposure compensation, yielding accurate reconstruction and efficient training.
- 3DGS provides high-fidelity rendering and fast training and rendering, but its irregular Gaussians often produce geometrically inaccurate surfaces.
- PGSR compresses 3D Gaussians into planes and derives depth from rendered normal and camera-to-plane distance maps.The plane-ray intersection defines the depth for each pixel.
- Two photometric coefficients compensate for overall image-brightness changes, improving reconstruction quality in scenes with illumination variation.
- Single-view and multi-view geometric regularization optimize pixel-plane parameters and promote global geometric consistency.The method also reduces regularization influence near geometric edges, where local planar assumptions fail.
- PGSR preserves high-quality rendering and speed while achieving state-of-the-art reconstruction accuracy, with training near 100 times faster than NeRF-based reconstruction methods.The reported comparison retains the original Gaussian rendering accuracy and speed.
B. Neural Surface Reconstruction
Neural and Gaussian-splatting surface-reconstruction methods trade off fidelity, inference efficiency, or geometric reliability. PGSR uses planar Gaussian representations and explicit depth and regularization mechanisms to improve surface reconstruction while retaining efficient rendering.
- End-to-end neural surface reconstruction methods can require substantial inference computation and extensively labeled 3D training models.
- NeRF predicts density and color along rays for realistic rendering, but its representation does not capture high-fidelity surfaces directly.
- NeRF-derived surface frameworks improve reconstruction with occupancy or signed-distance fields, but stacked MLPs constrain inference time and representation ability.
- SuGaR extracts meshes from Gaussian-derived point clouds using Poisson reconstruction, but its surface quality depends on biased depth quality.
- Gaussian Splatting based Surface Reconstruction: PGSR compresses Gaussians into flat planes, renders plane distance and normal maps, and transforms them into unbiased depth maps.Its overview also includes single-view and multi-view geometric regularization and exposure-compensated RGB loss.
- Gaussian Splatting based Surface Reconstruction: Unlike methods relying only on image reconstruction constraints, PGSR uses plane rendering and geometric regularization to fit Gaussians more closely to actual surfaces.
IV. METHOD
PGSR represents 3D Gaussians as flat planes to render plane distance and normals, which are converted into unbiased depth maps for efficient, high-fidelity geometry reconstruction.
- Method Overview: The resulting representation supports global-consistency constraints while maintaining rendering quality and speed.The method targets efficient, high-fidelity geometry reconstruction and rendering from multi-view RGB images.
- Unbiased Depth Rendering: The method renders plane-to-camera distance and normal maps, then intersects camera rays with the planes to obtain depth.The image position is represented by p, with camera intrinsics K used in the ray-plane intersection.
- Planar-based Gaussian Splatting Representation: PGSR compresses 3D Gaussians into 2D flat planes that approximate local scene geometry.The ellipsoids are flattened along their minimum-scale direction, whose corresponding axis represents the Gaussian normal.
- Unbiased Depth Rendering: The viewing direction resolves the shortest-axis normal ambiguity before alpha-blending produces the viewpoint-specific normal and distance maps.The camera-to-plane distance uses the camera pose and Gaussian center.
- Unbiased Depth Rendering: PGSR depth lies on the flattened Gaussian plane, unlike prior alpha-blended depth that forms curved surfaces inconsistent with the Gaussian shape.Dividing plane distance by the rendered normal removes the influence of weight accumulation coefficients.
B. Geometric Regularization
PGSR adds geometric constraints because image-only optimization can produce Gaussian shapes that do not conform to real surfaces.
- Motivation: Image reconstruction loss alone can lead to local overfitting and Gaussian shapes inconsistent with actual scene surfaces.The resulting geometry may have poor reconstruction accuracy.
- Local Plane Assumption: PGSR adopts local planarity to constrain consistency between depth and normals.The assumption treats neighboring pixels as an approximate plane.
1) Single-View Regularization:
PGSR combines single-view depth-normal consistency with multi-view geometric and photometric constraints to improve global geometric consistency.
- Single-View Regularization: Single-view regularization estimates local plane normals from four neighboring depth samples and aligns them with rendered normals.The sampled depths are projected into camera-coordinate 3D points before computing each local plane normal.
- Single-View Regularization: Image gradients weight the single-view normal loss so edge regions receive weaker local-planarity enforcement.The image gradient is normalized to the range 0 to 1.
- Multi-View Regularization: Single-view geometry provides accurate initial information, but multi-view regularization is needed because Gaussian optimization can yield inconsistent geometry across views.Image noise, blur, and weak textures can make photometric geometric solutions unreliable.
- Multi-View Regularization: Multi-view geometric consistency renders normals and plane distances in neighboring frames and compares forward-backward homography projections.Pixels with projection error above a threshold are excluded as occluded or geometrically unreliable, while larger errors receive smaller weights.
- Multi-View Regularization: Multi-view photometric consistency compares grayscale 7 × 7 patches mapped between frames using plane-induced homographies.The patches are centered on corresponding reference and neighboring-frame pixels.
- Geometric Regularization Loss: The geometric regularization loss combines single-view geometric, multi-view photometric, and multi-view geometric consistency terms.The complete loss is described as containing these three constraints.
C. Exposure Compensation Image Loss
PGSR models per-image exposure differences with two coefficients and uses exposure-adjusted image losses to improve consistency under changing illumination.
- Exposure Compensation: Different exposure times can create overall brightness variations across images and floating artifacts in practical scenes.Original 3DGS does not model these brightness changes.
- Exposure Compensation: PGSR assigns two exposure coefficients, a and b, to each image to model overall brightness variation.Exposure-adjusted images are computed using these coefficients.
- Exposure Compensation Image Loss: The exposure-adjusted image is constrained with L1 and SSIM losses against the ground-truth image.L1 enforces image consistency, while SSIM encourages structural similarity.
- Exposure Compensation Image Loss: Exposure estimation is performed after rendered and ground-truth images have sufficient structural similarity to improve coefficient robustness.The rendered image remains globally consistent, while the adjusted image matches ground-truth brightness.
- Overall Training Loss: The final training objective combines image reconstruction, flattened-Gaussian, and geometric losses.The reported weights are λ1 = 100, λ = 0.2, λ2 = 0.015, λ3 = 0.15, and λ4 = 0.03.
V. EXPERIMENTS
Experiments evaluate PGSR on diverse datasets using standard rendering and surface-quality metrics, showing strong reconstruction accuracy, fast training, and competitive novel-view synthesis. Comparisons with neural and 3DGS-based methods support these outcomes.
- Datasets: PGSR is evaluated on MiP-NeRF360 for novel-view synthesis and on TnT and DTU for reconstruction quality.The experiments cover objects plus indoor and outdoor environments.
- Evaluation Criterion: The evaluation uses PSNR, SSIM, and LPIPS for rendering, plus F1 score and chamfer distance for surface quality.
- Novel View Synthesis: PGSR achieves rendering results close to 3DGS and outperforms the similar reconstruction method SuGaR on MiP-NeRF360.
- Comparisons: PGSR is compared with NeRF, Deep Blending, INGP, Mip-NeRF360, NeuS, 3DGS, SuGaR, 2DGS, and GOF across rendering and reconstruction evaluations.
- Surface Reconstruction: On DTU, PGSR achieves the highest reconstruction accuracy with relatively fast training and significantly outperforms other 3DGS-based reconstruction methods.Its reconstructed surfaces are described as smoother and more detailed.
- Surface Reconstruction: On TnT, PGSR's F1 score is similar to NeuralAngelo and better than other current reconstruction methods, while training is over 100 times faster than NeuralAngelo.PGSR also reconstructs more surface details than NeuralAngelo.
C. Ablations
Ablations indicate that unbiased depth, single-view and multi-view regularization, and exposure compensation improve geometric reconstruction, while geometric constraints can slightly reduce rendering quality. The method is also demonstrated in a virtual-reality composition, with limitations under insufficient viewpoints and reflective surfaces.
- Unbiased Depth: Unbiased depth produces smoother and more precise geometric structures, especially in flat regions, while improving reconstruction and rendering accuracy.
- Single-View and Multi-View Regularization: Single-view regularization provides initial geometric accuracy without multi-view information, and removing it decreases reconstruction accuracy.
- Single-View and Multi-View Regularization: Multi-view regularization improves cross-view geometric consistency and overall reconstruction accuracy, with both photometric and geometric consistency contributing.
- Single-View and Multi-View Regularization: Without potential occlusion estimation, multi-view regularization negatively affects surface reconstruction and rendering accuracy.
- Ablation Findings: Geometric constraints slightly degrade rendering quality, possibly because an incomplete image-rendering model forces a balance between image and geometry losses.
- Exposure Compensation: Exposure compensation enhances reconstruction and rendering quality.
- Virtual Reality Application: PGSR supports immersive virtual-reality effects by separately rendering objects and scenes and using rendered depth to determine occlusion relationships.
- Limitations and Future Work: PGSR cannot reconstruct regions with missing or limited viewpoints accurately and does not address reflective surfaces or mirrors.
I. ADDITIONAL IMPLEMENT DETAILS
The additional experiments describe training-frame selection, geometric and photometric regularization, exposure compensation, depth filtering, and comparisons with 2DGS. Ablations report improved reconstruction accuracy, with some regularizers slightly reducing rendering quality.
- Implementation: Training iterations randomly select a neighboring frame from sets built using relative camera angle and position constraints.The neighboring-frame set is computed from training camera poses and dataset-specific thresholds.
- Depth Filtering: Normal-based depth filtering removes edge noise before TSDF fusion and slightly improves surface reconstruction accuracy on Tanks and Temples.Filtering is applied after scene training, and only on the Tanks and Temples dataset.
- Single-View Regularization: Single-view constraints improve reconstruction accuracy but slightly reduce rendering quality, while edge awareness helps preserve more details.The edge-aware variant provides only a slight reconstruction improvement.
- Multi-View Geometric Consistency: Multi-view geometric consistency improves surface smoothness, completeness, and reconstruction accuracy, especially for highly specular metallic objects, but slightly reduces rendering quality.Single-view regularization alone can leave holes on specular objects; multi-view consistency reduces this issue.
- Multi-View Photometric Consistency: NCC-based multi-view photometric consistency improves surface quality in highly specular scenes by reducing sensitivity to brightness changes.The ablation reports smoother, detail-rich surfaces, with a slight rendering-quality reduction overall.
- Depth and Exposure: Unbiased depth rendering is essential for improving reconstruction quality across scenes and slightly enhances rendering quality.Exposure compensation also brings brightness closer to ground truth when frame illumination varies.
B. More Results
Additional qualitative comparisons evaluate PGSR against 2DGS and GOF across indoor and outdoor scenes. These results further support PGSR’s high-fidelity geometric reconstruction capability.
- Comparisons: PGSR, 2DGS, and GOF are qualitatively compared on surface reconstruction across multiple DTU scenes.The comparisons are presented in Figs. II6–II9.
- Additional Results: Additional scenes and objects further confirm PGSR’s capability for high-fidelity geometric reconstruction.The broader qualitative results are shown in Fig. II10.