PaperlayerView exact revision on arXiv

Source-linked AI summary

SuGaR: Surface-Aligned Gaussian Splatting for Efficient 3D Mesh Reconstruction and High-Quality Mesh Rendering

Antoine Guédon, Vincent Lepetit

arXiv:2311.12775v3cs.GRcs.CV

TL;DR

SuGaR tackles the difficulty of extracting meshes from the unstructured, tiny Gaussians produced by 3D Gaussian Splatting. It regularizes Gaussians toward scene surfaces, samples level-set points for Poisson reconstruction, and optionally jointly refines the mesh and bound Gaussians. The method produces accurate meshes within minutes and supports high-quality rendering and traditional mesh-based editing.

  • Problem

    Gaussian Splatting produces realistic renderings quickly, but its optimized Gaussians are generally unstructured and poorly aligned with the scene surface, making mesh extraction difficult.

  • Method

    SuGaR regularizes Gaussians toward the scene surface, extracts meshes by sampling visible level-set points for Poisson reconstruction, and optionally jointly optimizes the mesh with bound Gaussians.

  • Results

    SuGaR extracts accurate, detailed meshes within minutes on a single GPU, while bound Gaussians improve rendering quality and reconstruct fine details missing from the mesh.

  • Takeaways & Limitations

    The resulting mesh enables traditional editing, sculpting, animating, and relighting workflows while retaining Gaussian-splatting rendering.

Abstract

from arXiv · show

We propose a method to allow precise and extremely fast mesh extraction from 3D Gaussian Splatting. Gaussian Splatting has recently become very popular as it yields realistic rendering while being significantly faster to train than NeRFs. It is however challenging to extract a mesh from the millions of tiny 3D gaussians as these gaussians tend to be unorganized after optimization and no method has been proposed so far. Our first key contribution is a regularization term that encourages the gaussians to align well with the surface of the scene. We then introduce a method that exploits this alignment to extract a mesh from the Gaussians using Poisson reconstruction, which is fast, scalable, and preserves details, in contrast to the Marching Cubes algorithm usually applied to extract meshes from Neural SDFs. Finally, we introduce an optional refinement strategy that binds gaussians to the surface of the mesh, and jointly optimizes these Gaussians and the mesh through Gaussian splatting rendering. This enables easy editing, sculpting, rigging, animating, compositing and relighting of the Gaussians using traditional softwares by manipulating the mesh instead of the gaussians themselves. Retrieving such an editable mesh for realistic rendering is done within minutes with our method, compared to hours with the state-of-the-art methods on neural SDFs, while providing a better rendering quality. Our project page is the following: https://anttwo.github.io/sugar/

Erratum

SuGaR addresses the difficulty of extracting editable meshes from unstructured Gaussian Splatting representations. It aligns Gaussians with scene surfaces, samples level-set points for Poisson reconstruction, and optionally jointly refines the mesh and Gaussians.

  • Erratum: An erratum clarifies that the regularization uses an ideal distance function associated with density d, which aligns with the true surface when d = d̄.The correction updates Equation 7 and affects the computation of regularization term R in Equation 8.
  • Motivation: 3D Gaussian Splatting renders realistic scenes quickly, but its millions of tiny Gaussians are generally unstructured and do not correspond well to the scene surface.This makes mesh extraction difficult despite the usefulness of mesh representations for editing, sculpting, animating, and relighting.
  • Surface alignment: SuGaR regularizes Gaussians toward the scene surface by deriving a volume density under a flat, well-distributed Gaussian assumption and matching it to the actual density.The regularization is designed to improve the geometric representation captured by the Gaussians.
  • Mesh extraction: Marching Cubes struggles with the sparse density produced by millions of differently scaled and rotated Gaussians, whereas SuGaR samples visible level-set points and applies scalable Poisson reconstruction.The resulting mesh can be reconstructed within minutes on a single GPU.
  • Mesh refinement and editing: SuGaR optionally jointly optimizes the extracted mesh and bound Gaussians through Gaussian-splatting rendering, enabling traditional mesh editing while maintaining high-quality rendering.The contributions also include a regularization term and an efficient mesh-extraction algorithm.

2. Related Work

Related work spans image-, point-, mesh-, volumetric-, and hybrid-rendering representations. SuGaR is positioned against methods that combine meshes with neural or radiance-field representations, particularly those requiring substantially longer training or yielding lower rendering quality.

  • Image-based rendering: Image-based rendering uses two-dimensional scene images to generate novel views, while early light-field methods emphasized efficient traversal of volumetric data.These approaches established volume rendering as a tool for realistic novel-view synthesis.
  • Scene representations: Scene representations include triangle meshes, point clouds, voxel grids, multiplane images, and neural implicit functions.This range motivates comparisons across distinct representation families.
  • Traditional mesh-based IBR: Traditional mesh-based methods use SfM and MVS for surface reconstruction, then render through textured triangles or image warping and blending.Later methods use deep learning-based mesh representations to improve view synthesis.
  • Volumetric IBR: Volumetric methods represent scenes with voxel grids, multiplane images, or neural networks, with NeRF using a continuous density-and-color function rendered by volumetric ray tracing.Subsequent methods improve NeRF performance through sparse features, hierarchical sampling, or low-rank approximations while retaining volumetric ray marching.
  • Hybrid IBR: Hybrid methods combine mesh-based and volumetric representations for surface reconstruction, editability, and view-dependent appearance.BakedSDF requires 48 hours of training, while NeRFMeshing requires approximately an hour on 8 V100 NVIDIA GPUs and produces lower PSNR than SuGaR; SuGaR reports faster mesh retrieval and higher rendering quality than previous mesh-based solutions.
  • Point-based IBR: 3D Gaussian Splatting offers fast, high-quality point-based rendering through α-blending and point rasterization.Its speed and quality motivate using it as the basis for SuGaR’s mesh extraction and rendering pipeline.

3. 3D Gaussian Splatting

3D Gaussian Splatting represents a scene with many parameterized Gaussians whose rendering is optimized to match training images. A rasterizer projects these Gaussians into image-space splats, enabling faster rendering than NeRF’s ray marching.

  • Gaussian representation: Each Gaussian is represented by a mean µ_g and covariance Σ_g, with scaling s_g and quaternion q_g encoding its size and rotation.Gaussians also carry opacity α_g and spherical-harmonics coefficients describing emitted colors.
  • Rasterization: A rasterizer splats 3D Gaussians into 2D Gaussians parallel to the image plane for image rendering.This splatting process is the key component behind Gaussian Splatting’s speed advantage over NeRF ray-marching compositing.
  • Optimization: Given training images, Gaussian means, rotations, scales, opacities, and spherical-harmonics parameters are optimized so rendered images match the inputs.Additional Gaussians are added during optimization to better fit scene geometry.

4. Method

SuGaR aligns 3D Gaussians with scene surfaces, uses depth-map-guided level-set sampling and Poisson reconstruction for mesh extraction, then optionally refines the mesh and surface-bound Gaussians jointly.

  • Aligning the Gaussians with the Surface: SuGaR regularizes Gaussian density and SDF estimates to encourage flat, opaque, well-distributed Gaussians aligned with the scene surface.The method compares an ideal density or SDF derived under aligned-Gaussian assumptions with quantities computed from the current Gaussians.
  • Aligning the Gaussians with the Surface: Depth maps rendered from training viewpoints efficiently estimate the current Gaussian surface SDF at sampled points drawn according to the Gaussian distribution.For visible points, the estimate uses the difference between the point depth and the corresponding rendered depth-map value.
  • Efficient Mesh Extraction: Refining sampled points onto the level set reduces noise and restores details that direct use of splatted depth points can miss.The refinement is motivated by the inexactness of splatted depth maps.
  • Efficient Mesh Extraction: The mesh extractor samples points on a density level set, computes their analytical density-gradient normals, and applies Poisson reconstruction.Points are found by sampling along depth-map rays and interpolating where density crosses the chosen level λ.
  • Binding New 3D Gaussians to the Mesh: After extraction, SuGaR binds new thin Gaussians to mesh triangles and jointly optimizes the mesh and Gaussians with the Gaussian Splatting rasterizer.This preserves high-quality rendering while allowing the scene to be edited through mesh-editing tools.

5. Experiments

Experiments evaluate SuGaR’s rendering, mesh extraction, and surface-Gaussian representations across real scenes and ablations. SuGaR achieves strong rendering quality with meshes while preserving efficient extraction and detail.

  • Implementation: 15,000 iterations comprise unconstrained Gaussian Splatting, opacity-entropy regularization, and final surface regularization.The stages use 7,000, 2,000, and 6,000 iterations, respectively.
  • Implementation: 5–10 minutes is the typical mesh-extraction time using a λ-level set with λ = 0.3, Poisson reconstruction, and mesh simplification.Poisson reconstruction uses depth 10 and quadric-error simplification.
  • Evaluation: Evaluation covers 7 Mip-NeRF360 scenes, 2 Tanks&Temples scenes, and 2 DeepBlending scenes using PSNR, SSIM, and LPIPS.Flowers and Treehill are omitted because of licensing and availability issues.
  • Rendering results: SuGaR significantly outperforms mesh-based state-of-the-art methods for novel-view synthesis and exceeds several rendering-only models.The comparisons include Instant-NGP and Plenoxels.
  • Rendering results: Surface-constrained Gaussians improve rendering by initializing refinement from the extracted mesh and reconstructing fine details absent from the mesh.They act as an efficient texturing tool during Gaussian-splatting refinement.
  • Mesh rendering ablation: Increasing mesh vertices improves rendering quality, while lower-triangle meshes already reach state-of-the-art results with more surface-aligned Gaussians per triangle.The ablation varies mesh resolution and compensates with Gaussian count per triangle.
  • Mesh rendering ablation: Surface-aligned Gaussians outperform traditional UV textures, although UV-textured meshes still produce satisfying rendering results.The comparison uses only the diffuse spherical-harmonics component for SuGaR.

6. Conclusion

The method obtains accurate 3D triangle meshes from Gaussian Splatting very quickly and combines meshing with Gaussian Splatting for scene manipulation and realistic rendering.

  • SuGaR provides a very fast algorithm for obtaining an accurate 3D triangle mesh from a scene represented with Gaussian Splatting.
  • Combining meshing with Gaussian Splatting enables intuitive manipulation of captured scenes while maintaining realistic rendering.

Supplementary Material

The supplementary material adds parameterization and implementation details, quantitative rendering and mesh-rendering ablations, and a video overview with additional qualitative demonstrations.

  • The supplement details the parameterization of bound Gaussians optimized during joint refinement.
  • It provides additional implementation details and quantitative results for real-time rendering of real scenes and mesh-rendering ablations.
  • A supplementary video overviews the approach and showcases additional qualitative results, including SuGaR-mesh animation demonstrations.

7. Parameterisation of Gaussians bound to the surface

The refinement strategy binds Gaussians to extracted mesh triangles, jointly optimizes the mesh and Gaussians, and parameterizes their orientations within triangle-aligned coordinate frames. The representation supports mesh editing and animation, although scaling or deformation requires corresponding scale adjustments.

  • After mesh extraction, new Gaussians are bound to mesh triangles and the Gaussians and mesh are jointly optimized with the Gaussian Splatting rasterizer.
  • Gaussian means are computed from mesh vertices using predefined barycentric coordinates, keeping the Gaussians flat and aligned with their corresponding triangles.
  • Each Gaussian learns a complex number x + iy that encodes its 2D rotation within the corresponding triangle’s plane instead of directly optimizing a quaternion.
  • The triangle’s 3D rotation matrix uses the triangle normal, a fixed edge, and their cross-product to define its three columns before applying the learned in-plane rotation.
  • Mesh rotation or movement during editing does not require updating learned 2D rotations because they remain unchanged when triangles rotate or move.
  • Scaling or deforming the mesh requires adjusting bound-Gaussian scaling factors in proportion to the changed triangle size.For example, doubling the mesh size requires multiplying all Gaussian scaling factors by 2.

8. Additional implementation details

The implementation uses standard PyTorch-based Gaussian Splatting components and separates foreground and background mesh extraction with Poisson reconstruction. Additional results compare rendering quality across datasets and show that normal consistency marginally improves visual smoothness without changing standard metrics.

  • Implementation: SuGaR uses PyTorch, PyTorch3D, and the differentiable Gaussian Splatting rasterizer from the original implementation.These tools support model implementation, 3D data processing, and rendering.
  • Mesh extraction: Foreground and background points are reconstructed separately with two Poisson reconstructions.Foreground points lie inside the bounding box of training camera poses; background points lie outside.
  • Mesh extraction: Custom foreground bounding boxes could improve extracted-mesh quality and precision for particular scenes and objects.The default distinction is chosen for generality.
  • Joint refinement: Normal consistency during joint refinement marginally enhances visual quality by promoting smoother mesh surfaces.It does not affect PSNR, SSIM, or LPIPS performance.
  • Evaluation: On Tanks&Temples and DeepBlending, SuGaR renders less well than vanilla 3D Gaussian Splatting but better than methods that do not recover a mesh.The comparison is reported in the quantitative evaluations for both datasets.

9. Additional Results for Real-Time Rendering of Real Scenes

Additional real-scene evaluation measures SuGaR rendering with PSNR, SSIM, and LPIPS across multiple datasets. The supplementary tables report dataset-level and per-scene results.

  • Evaluation: SuGaR rendering quality is evaluated with PSNR, SSIM, and LPIPS.These are computed using extracted meshes and their bound surface Gaussians.
  • Datasets: The supplementary evaluation covers Mip-NeRF360, Tanks&Temples, and DeepBlending.Mip-NeRF360 results appear in Table 1, while Tanks&Temples and DeepBlending results appear in Tables 4 and 5.
  • Results: Tables 6, 7, and 8 provide detailed results for all scenes in the evaluated datasets.These tables organize the supplementary per-scene metrics.

10. Additional Results for Mesh Renderig Ablation

The mesh-rendering ablation examines refinement, mesh resolution, and surface-aligned Gaussian texturing. Surface Gaussians preserve high rendering quality with relatively low-resolution meshes, while refinement recovers finer details and UV textures remain viable but weaker.

  • Refinement: Surface-constrained Gaussians substantially improve rendering quality by acting as an efficient texturing tool and recovering fine details missing from the mesh.The refinement examples show this effect directly.
  • Refinement: 2,000 refinement iterations are usually sufficient for high-quality rendering because the extracted mesh with surface Gaussians provides an excellent initialization.Further iterations capture texturing details and extremely thin geometry, including bicycle spokes.
  • Mesh resolution: Increasing mesh vertices improves rendering quality with surface Gaussians, but lower-resolution meshes already reach state-of-the-art results.The comparison includes meshes with 200,000 and 1,000,000 vertices.
  • Mesh resolution: Meshes with fewer triangles receive more surface-aligned Gaussians per triangle to keep the total Gaussian count comparable.This controls the rendering-resolution comparison across mesh sizes.
  • Texturing: Surface-aligned Gaussians render better than traditional UV textures, although UV-textured meshes still produce satisfying results.The qualitative comparison uses meshes with 200,000 vertices.
  • Metric ablation: Lower vertex counts improve surface regularity but increase LPIPS, while higher vertex counts improve PSNR and SSIM rendering quality.The metric-specific trends are reported in the supplementary tables.
Loading 2311.12775v3…