Source-linked AI summary
MeshSplatting: Differentiable Rendering with Opaque Meshes
Jan Held, Sanghyun Son, Renaud Vandeghen, Daniel Rebain, Matheus Gadelha, Yi Zhou, Anthony Cioppa, Ming C. Lin, Marc Van Droogenbroeck, Andrea Tagliasacchi
TL;DR
MeshSplatting addresses the incompatibility between high-fidelity point-based novel-view representations and mesh-based graphics pipelines by jointly optimizing connected, opaque meshes through differentiable rendering. It produces accurate, visually high-quality meshes that train 2× faster than current state-of-the-art methods and support real-time rendering in standard engines.
Problem
Point-based splatting achieves high-fidelity real-time rendering but remains incompatible with polygonal mesh pipelines used in games, simulators, and AR/VR applications.
Method
MeshSplatting jointly optimizes geometry and appearance using shared-vertex triangle parameterization, restricted Delaunay connectivity, and opacity-aware differentiable rendering.
Results
MeshSplatting produces accurate surface meshes, achieving the best Chamfer distance on 5 of 15 scenes and outperforming Triangle Splatting and 3DGS in mean score.
Takeaways & Limitations
The method provides a compact, real-time-renderable mesh representation that combines high-quality appearance with accurate geometry for traditional graphics pipelines.
Takeaways & Limitations
Performance degrades in sparsely covered background regions and outside the training-view orbit, while transparent objects remain difficult to represent.
Abstract
from arXiv · showhide
Primitive-based splatting methods like 3D Gaussian Splatting have revolutionized novel view synthesis with real-time rendering. However, their point-based representations remain incompatible with mesh-based pipelines that power AR/VR and game engines. We present MeshSplatting, a mesh-based reconstruction approach that jointly optimizes geometry and appearance through differentiable rendering. By enforcing connectivity via restricted Delaunay triangulation and refining surface consistency, MeshSplatting creates end-to-end smooth, visually high-quality meshes that render efficiently in real-time 3D engines. On Mip-NeRF360, it boosts PSNR by +0.69 dB over the current state-of-the-art MiLo for mesh-based novel view synthesis, while training 2x faster and using 2x less memory, bridging neural rendering and interactive 3D graphics for seamless real-time scene interaction. The project page is available at https://meshsplatting.github.io/.
1. Introduction
MeshSplatting addresses the incompatibility between Gaussian-based novel-view synthesis and classical graphics pipelines by directly optimizing connected meshes with differentiable rasterization. It aims to retain visual quality while enabling end-to-end mesh optimization that trains 2× faster than current state-of-the-art methods.
- Motivation: 3D Gaussian Splatting achieves photorealistic reconstruction and real-time rendering with millions of 3D Gaussian primitives, but its primitives are not immediately compatible with classical graphics pipelines.The affected pipelines include simulators, games, and AR/VR applications.
- Motivation: Supporting 3DGS in classical pipelines is non-trivial because its sorting and alpha blending prevent standard techniques such as depth buffers and occlusion culling.One alternative is engineering rendering engines and simulators to support Gaussian splatting directly.
- Key Contributions: MeshSplatting directly optimizes mesh-based scene representations through differentiable rasterization instead of relying on non-differentiable Gaussian-to-mesh conversion.Prior conversion approaches require complex post-processing and typically lose visual quality because conversion is non-differentiable.
- Key Contributions: 2× faster training while retaining visual quality is a stated advantage over current state-of-the-art mesh-based methods.
- Key Contributions: MeshSplatting generates a connected mesh by refining vertex locations in a restricted Delaunay triangulation rather than producing a polygon soup.The triangulation provides connectivity between triangles as part of the representation.
2. Related work
Differentiable rendering supports end-to-end learning of explicit scene representations, while recent work has advanced Gaussian primitives and mesh reconstruction from images. Existing methods often separate mesh extraction or color learning from optimization, whereas MeshSplatting jointly optimizes opaque triangles and vertex colors for direct game-engine compatibility.
- Differentiable rendering and Gaussian primitives: Differentiable rendering propagates image-based losses to scene parameters, enabling explicit point-cloud [25], voxel-grid, polygonal-mesh [25] [32], and Gaussian-primitive [26] representations.3D Gaussian Splatting [26] demonstrated fitting millions of anisotropic Gaussians in minutes for high-fidelity real-time rendering.
- Differentiable rendering and Gaussian primitives: Subsequent Gaussian research explored 2D Gaussians [20], generalized Gaussians [15], alternative kernels [21], learnable basis functions [6], and non-Gaussian primitives including smooth 3D convexes [17] and linear primitives.
- Mesh reconstruction from images: Mesh reconstruction methods convert implicit radiance fields or optimized Gaussian scenes into meshes, using approaches such as baking textured triangle meshes, TSDF extraction, Poisson reconstruction, or Marching Tetrahedra.These methods generally treat mesh extraction as a separate post-processing step, limiting compatibility with traditional mesh-based game engines.
- Mesh reconstruction from images: MeshSplatting directly optimizes opaque triangles with vertex colors, unlike MiLo, which still requires separate color learning, producing results immediately compatible with game engines without post-processing.MiLo integrates surface mesh extraction into optimization and jointly refines mesh geometry and Gaussian representation, but does not jointly optimize color.
3. Methodology
MeshSplatting uses differentiable Triangle Splatting primitives and transforms an independently optimized triangle soup into a connected, fully opaque mesh. Its pipeline combines shared-vertex parameterization, restricted Delaunay triangulation, refinement, and scheduled opacity and window parameters for stable optimization.
- Triangle Splatting: Each triangle uses differentiable screen-space signed-distance splatting, with color, smoothness, and opacity controlling its rendered influence.The signed distance is positive outside, negative inside, and zero on the boundary; the smoothness parameter controls the transition toward a solid triangle.
- Mesh representation: MeshSplatting parameterizes triangles through shared vertices storing 3D position, color, and opacity, with barycentric color interpolation and minimum incident-vertex opacity.Shared vertices accumulate gradients from neighboring faces, ensuring consistent updates across incident triangles.
- Optimization stages: The two-stage pipeline first optimizes an unconstrained semi-transparent triangle soup, then applies restricted Delaunay triangulation and refinement to produce a connected mesh.The unconstrained stage enables rapid optimization, while the second stage restores global connectivity and refines geometric and appearance inconsistencies.
- Mesh creation: Restricted Delaunay triangulation approximates the triangle-soup surface while preserving Delaunay properties and introducing connectivity through shared adjacent vertices.The operation constructs a Delaunay tetrahedralization and selects faces whose dual Voronoi edges intersect the input surface.
- Parameter scheduling: Opacity is increased smoothly from semi-transparent to fully opaque, while σ decreases from 1.0 to sharpen triangles and maintain gradient flow during training.After training, opacity is discarded and all triangles are treated as fully opaque for compatibility with standard game engines.
4. Experiments
MeshSplatting is evaluated for mesh-based novel view synthesis on Mip-NeRF360 and Tanks and Temples using visual-quality, compactness, speed, and memory measures. It outperforms concurrent methods while producing directly usable colored opaque meshes, and additional experiments examine self-supervised reconstruction, connectivity, and design choices.
- Quantitative results: 4–10 dB higher PSNR and significantly lower LPIPS than 2DGS and Triangle Splatting are achieved with similar vertex counts, while using 2–10× fewer vertices than GOF, RaDe-GS, and MiLo with higher SSIM and lower LPIPS.These comparisons are reported for Mip-NeRF360 and Tanks and Temples mesh-based novel view synthesis.
- Qualitative and perceptual results: On T&T, GOF and MiLo achieve higher PSNR but lower SSIM and higher LPIPS, indicating more rendering artifacts despite detailed meshes.On Mip-NeRF360, MeshSplatting achieves substantially higher PSNR than GOF, RaDe-GS, and MiLo.
- Practicality: MeshSplatting directly produces colored opaque meshes, whereas competing methods require mesh extraction and/or texture-coloring post-processing before use in game engines.2DGS, GOF, and RaDe-GS require both extraction and coloring; MiLo directly outputs a mesh but still requires texturing.
- Efficiency: 48 minutes of training versus 106 minutes for MiLo yields a 35–55% speedup, because MeshSplatting runs restricted Delaunay triangulation once rather than at every iteration.The optimized triangulation takes under two minutes and contributes negligibly to total training time.
- DTU evaluation: With self-supervised regularization on DTU, MeshSplatting attains the lowest Chamfer distance in 5 of 15 scenes and reaches mesh quality comparable to concurrent methods.The evaluation sets βd to zero to ensure a fair self-supervised comparison, although the method is designed primarily for large, complex real scenes.
- Connectivity and ablations: About 92% of triangles have at least three neighbors after restricted Delaunay triangulation, while delaying pruning until after training preserves connectivity by removing non-contributing tiny triangles later.The ablations show that Ld, Lz, and Ln slightly reduce visual quality but improve geometric accuracy and surface smoothness.
5. Conclusions
The method jointly optimizes mesh-based scene representations into connected, compact meshes that preserve visual fidelity, accurate geometry, and real-time renderability.
- Contributions: The framework enables end-to-end optimization of mesh-based scene representations.It reformulates triangle parameterization to enable vertex sharing.
- Mesh connectivity: Vertex sharing produces connected meshes while maintaining high visual fidelity.
- Representation: Training toward opaque triangles and connectivity unifies high-quality appearance and accurate geometry in a compact, real-time-renderable mesh.
Supplementary Material
The supplementary material reports detailed rendering, surface, scaling, ablation, topology, and object-extraction results for MeshSplatting. It also identifies limitations in sparse background regions and views outside the training orbit.
- Detailed results: MeshSplatting produces less noisy renderings than MiLo, yielding higher SSIM and lower LPIPS despite MiLo’s strong PSNR on T&T.Per-scene results are provided for T&T and indoor and outdoor MipNeRF360 scenes.
- Surface reconstruction: MeshSplatting attains the best Chamfer distance on 5 of 15 scenes and a better mean score than Triangle Splatting and 3DGS, while remaining competitive with 2DGS.These results show that the method also produces accurate surface meshes, although its primary focus is mesh-based novel view synthesis.
- Impact of triangle count on visual quality: Increasing triangle count consistently improves visual quality, with gains in both PSNR and especially LPIPS on outdoor MipNeRF360 scenes.The evaluation shows that MeshSplatting scales effectively as more triangles are used.
- Ablations: Hard pruning, maximum-blending-weight pruning, supersampling, spherical harmonics, and vertex regularization each materially support MeshSplatting’s visual quality and surface smoothness.Removing hard pruning leaves unnecessary triangles; disabling supersampling increases aliasing; RGB-only appearance misses fine textures; and vertex regularization reduces artifacts and improves normal consistency.
- Mesh creation & refinement: Restricted Delaunay triangulation alone leaves uncovered and geometrically inconsistent regions, whereas refinement improves visual fidelity and geometry; linear scheduling also preserves gradients better than cosine scheduling.Supersampling renders at s× target resolution and downsamples with area interpolation to implement box anti-aliasing, following Chen et al. [7].
- Topology and limitations: MeshSplatting’s global vertex–face ratio is 0.48, near the 0.5 closed-manifold expectation, but sparse backgrounds remain incomplete and quality degrades outside the training-view orbit.The reconstruction is globally compact and near-manifold despite remaining local nonmanifold regions.