Source-linked AI summary
Differentiable Surface Splatting for Point-based Geometry Processing
Wang Yifan, Felice Serena, Shihao Wu, Cengiz Öztireli, Olga Sorkine-Hornung
TL;DR
Differentiable rendering for point clouds must provide useful gradients despite discontinuities and underconstrained point configurations. DSS uses surface splatting with carefully designed location and normal gradients plus regularization, and achieves high-quality geometry processing across synthesis and denoising tasks. Its reported advantages include handling substantial topology changes and preserving geometric detail, while prior point renderers have insufficient gradients for changing point-cloud shape.
Problem
Differentiable point-cloud rendering needs gradients that remain useful around occlusion and edge discontinuities while supporting geometry changes without explicit connectivity.
Method
DSS is a high-fidelity point-based differentiable renderer using surface splats, gradients for point locations and normals, and regularization that promotes plausible, uniformly distributed surface points.
Results
DSS yields significantly better results than previous differentiable-rendering methods, especially for substantial topological changes and geometric-detail preservation, and Pix2Pix-DSS outperforms compared denoising methods quantitatively and qualitatively.
Takeaways & Limitations
DSS supports image-based geometry synthesis, local shape modification, and point-cloud denoising while propagating gradients to point positions and normals.
Takeaways & Limitations
Prior point-set depth renderers generally lack gradients sufficient to alter point-cloud shape, and DSS’s point-position gradient is undefined at ellipse edges.
Abstract
from arXiv · showhide
We propose Differentiable Surface Splatting (DSS), a high-fidelity differentiable renderer for point clouds. Gradients for point locations and normals are carefully designed to handle discontinuities of the rendering function. Regularization terms are introduced to ensure uniform distribution of the points on the underlying surface. We demonstrate applications of DSS to inverse rendering for geometry synthesis and denoising, where large scale topological changes, as well as small scale detail modifications, are accurately and robustly handled without requiring explicit connectivity, outperforming state-of-the-art techniques. The data and code are at https://github.com/yifita/DSS.
1 INTRODUCTION
Differentiable rendering enables image-based manipulation of scene parameters but must address difficult gradients and point-cloud surface representation. DSS introduces a high-fidelity point-based renderer with regularized gradients and demonstrates strong geometry-processing results.
- Motivation: Differentiable renderers map scene information to synthesized images, allowing image losses to update scene parameters through renderer gradients.The scene may include geometry, lighting, materials, and camera position.
- Motivation: Existing differentiable renderers use voxel, mesh, or point representations, while point-based methods must capture details, handle gaps and occlusions, and form continuous surfaces.Voxel methods have high memory requirements even for relatively coarse geometries.
- DSS: DSS represents points as projected disks or ellipses, interpolates splats for hole-free antialiased renderings, and designs gradients for point locations and normals.Its regularization drives points toward plausible configurations that remain on local geometric structures with uniform distribution.
- Applications: DSS renders multi-view color images and auxiliary maps, then propagates processed image information to produce high-quality geometries.The method focuses particularly on point-cloud denoising and reports significantly better results than previous differentiable-rendering methods.
2 RELATED WORK
Related work spans differentiable rendering, learning-based 3D vision, point-set processing, and surface splatting. DSS builds on surface splatting while targeting point-cloud gradients and geometry processing beyond prior depth-rendering or mesh-based limitations.
- Differentiable rendering: Differentiable-rendering research includes general frameworks and task-specific methods for learning-based 3D vision, reconstruction, completion, and image synthesis.Prior methods use voxel, mesh, point, and ray-based representations.
- Differentiable rendering: Prior point-set depth renderers commonly provide 2D supervision but do not define sufficient gradients for updating point positions or normals to alter point-cloud shape.Their gradients may be restricted to depth values or small local neighborhoods.
- Point-based processing: Point-cloud processing methods address noise, outliers, incompleteness, and misalignment using optimization-based, learning-based, unsupervised, or projection-based techniques.Some approaches rely on smooth-surface priors, learned local properties, or edge annotations.
- Renderer comparison: Compared with generic mesh renderers, DSS jointly updates point positions and normals, considers occlusion and visibility changes, and enables topology-changing deformation.The table contrasts these capabilities with limitations involving depth gradients, silhouettes, normal fields, and mesh topology.
- Surface splatting: Surface splatting supplies the foundation for DSS, which is presented as the first high-fidelity differentiable implementation of this point-based rendering approach.Surface splatting was developed for efficient point-set rendering and processing.
3 METHOD
DSS optimizes point-cloud scene parameters through an EWA-based renderer with occlusion-aware gradients and surface regularization. Its design supports faithful rendering, large deformations, and cleaner point distributions without explicit connectivity.
- Differentiable rendering: DSS formulates inverse rendering by optimizing scene parameters so rendered images match reference images under an image loss.The parameters can include point coordinates, normals, colors, camera pose, and lighting.
- Differentiable rendering: Rendering discontinuities arise because occlusion events and edges make gradients with respect to point coordinates and normals undefined everywhere.This discontinuity motivates the specialized backward-pass design.
- Forward pass: DSS uses screen-space EWA surface splatting, representing each point as a tangent-plane disk or ellipse projected into an elliptical Gaussian splat.The final pixel value is the normalized sum of filtered point attributes, with bounded support and occlusion handling.
- Backward pass: The backward pass approximates visibility-switch gradients using point movement and pixel-intensity changes, while propagating gradients jointly through all dimensions of point position.Unlike image-plane-only alternatives, the formulation includes depth and considers occluded and occluding ellipses.
- Evaluation: Compared with RBF-derived gradients, the proposed approximation avoids sensitivity to σ_k and is less prone to local minima in multi-point optimization.The method also produces faithful geometry renderings rather than relying on blurred edges or transparent surfaces.
- Surface regularization: DSS adds repulsion and projection regularizers to distribute points uniformly and preserve clean surfaces during optimization.Repulsion separates neighboring points on a local projection plane, while projection minimizes distance to the surface tangent plane.
4 IMPLEMENTATION DETAILS.
DSS combines image-loss optimization with alternating normal and point updates, view sampling, and regularization to improve convergence and surface reconstruction. Error-aware views focus optimization on difficult regions, while normal updates and regularization support clean, detailed geometry.
- 4.1 Optimization objective: SMAPE is selected as the image loss because it is more stable for high-dynamic-range rendered images.The loss is designed for unbounded image values.
- 4.2 Alternating normal and point update: Normals and point positions are optimized alternately because their gradients are derived while holding the other variable fixed.The method performs Tn normal-optimization steps followed by Tp point-position steps.
- 4.2 Alternating normal and point update: Normal refinement supports clean surfaces, while regularization gradients can update point positions even when positions are otherwise held fixed.This strategy consistently yields satisfying results for local shape-surface modification.
- 4.3 View selection: Camera positions are sampled to cover object viewpoints, with periodic resampling to improve optimization and quick convergence.The baseline scheme uses farthest point sampling on a surrounding sphere, random perturbations, and cameras aimed at the object center.
- 4.3 View selection: For complex topology, error-aware view sampling selects cameras around regions with the largest image errors to improve point positions in small holes.The method finds the highest-error pixel, identifies nearby projected points, and samples a local sphere around their mean 3D position.
5 RESULTS
DSS is evaluated for geometry deformation, image-filter-based editing, and point-cloud denoising, outperforming prior differentiable renderers especially under topology changes, noise, and detail-preservation demands.
- DSS is compared with state-of-the-art differentiable renderers for point-based geometry editing and filtering.
- Comparison of different DRs: DSS recovers large topology-changing structures and elaborate surface details more faithfully than mesh-based renderers.On the teapot, mesh methods struggle with the handle, cover, and spout, whereas DSS recovers these structures and body patterns.
- Comparison of different DRs: Naive depth-based point rendering fails on large deformation and fine-scale denoising because position gradients remain local and normals are unused.
- Shape editing via image filter: DSS transfers image-filter changes to geometry and remains reasonable with 1% input noise.Its independent updates of point normals and positions avoid structural constraints that make mesh-based gradients sensitive to noise.
- Point cloud denoising: Pix2Pix-DSS outperforms the compared denoising methods quantitatively and qualitatively on sparse, relatively noisy inputs.The evaluation uses 20K points with 1% noise and compares reconstructed surfaces using Chamfer and Hausdorff distance.
- Point cloud denoising: On dense inputs with 0.3% noise, Pix2Pix-DSS reconstructs clean surfaces with abundant details despite training on 20K-point data.
- Runtime: DSS takes 160 optimization steps and 11 minutes for a similar L0-smoothing result that requires Paparazzi 30000 steps and 30 minutes.
6 CONCLUSION AND FUTURE WORKS
The paper presents DSS as a high-quality splat-based differentiable renderer that supports accurate geometry and topology processing. It also points toward broader image-to-geometry applications enabled by neural networks.
- DSS propagates gradients to point positions and normals while producing accurate geometries and topologies through gradient handling and regularization.
- Combining DSS with contemporary deep neural networks yielded state-of-the-art results.
- DSS is positioned as a first step toward transferring image-based neural techniques such as stylization, segmentation, and super-resolution to geometry.
- The paper identifies inverse rendering and dynamic scene geometry inference in multimodal capture setups as future applications.
A PARAMETER DISCUSSION
The parameter discussion describes settings for forward and backward rendering, regularization, optimization, and camera sampling, including trade-offs affecting coverage, convergence, clustering, and topology changes.
- Forward rendering: Forward-rendering parameters follow EWA defaults, with a higher cutoff for close views to maintain hole-free splats.
- Backward rendering: A backward-rendering cache size of K = 5 is sufficient for the experiments, while larger K improves occlusion-aware weight accuracy.
- Regularization: Regularization uses D = 4√(D/N) and Θ = π/3, with a higher projection bandwidth helping pull large-deformation outliers toward the nearest surface.
- Optimization: Learning rates of 5 for positions and 5000 for normals generally work well, while higher rates accelerate convergence but increase clustering risk.
- Optimization: Twelve camera views are used because eight or fewer views cause results to degenerate.
- Regularization: Excessive repulsion can move points off the surface, whereas excessive projection can inhibit topology changes.
B DENOISING PIX2PIX
The denoising network is based on Pix2Pix and trained on synthetic noisy point clouds generated from Sketchfab models with Poisson-disk sampling and PCA normals.
- The Pix2Pix model uses a generator and discriminator, with ResNet selected over U-Net because it performs slightly better.
- Pixelwise normalization, added convolutional layers, and removal of final-layer tanh address generator artifacts and unbounded pixel values.
- The network is trained with Pix2Pix defaults and ADAM for about two days on an NVIDIA 1080Ti GPU.
- Training data come from 91 high-resolution Sketchfab models sampled to 20K reference points per model, with white Gaussian noise added to create inputs.
- PCA normals are computed for both reference and noisy point sets before generating training data.