Source-linked AI summary
BakedSDF: Meshing Neural SDFs for Real-Time View Synthesis
Lior Yariv, Peter Hedman, Christian Reiser, Dor Verbin, Pratul P. Srinivasan, Richard Szeliski, Jonathan T. Barron, Ben Mildenhall
TL;DR
Real-time novel view synthesis for large, unbounded real-world scenes requires both high-quality reconstruction and efficient rendering, while existing volumetric approaches can be costly or produce less useful meshes. BakedSDF optimizes a contracted-space hybrid neural volume-surface representation, bakes it into a triangle mesh, and adds spherical-Gaussian appearance that is optimized against captured views. The resulting system delivers state-of-the-art real-time rendering in speed and accuracy, with lower power consumption, while producing meshes suitable for appearance editing and physics simulation.
Problem
Existing NeRF-like methods struggle to provide high-quality, well-behaved meshes for unbounded real-world scenes while supporting real-time rendering on commodity hardware.
Method
BakedSDF optimizes a contracted-space hybrid neural volume-surface representation, extracts its zero level set as a triangle mesh, and equips the mesh with optimized spherical-Gaussian appearance.
Results
The system achieves state-of-the-art real-time view synthesis in speed and accuracy, with lower power consumption, and produces accurate, detailed meshes for downstream graphics applications.
Takeaways & Limitations
BakedSDF combines polygon-rasterization-friendly real-time rendering with mesh quality sufficient for appearance editing and physics simulation.
Takeaways & Limitations
The fully opaque mesh may struggle with semi-transparent content, and mesh reconstruction can fail in areas with small or detailed geometry.
Abstract
from arXiv · showhide
We present a method for reconstructing high-quality meshes of large unbounded real-world scenes suitable for photorealistic novel view synthesis. We first optimize a hybrid neural volume-surface scene representation designed to have well-behaved level sets that correspond to surfaces in the scene. We then bake this representation into a high-quality triangle mesh, which we equip with a simple and fast view-dependent appearance model based on spherical Gaussians. Finally, we optimize this baked representation to best reproduce the captured viewpoints, resulting in a model that can leverage accelerated polygon rasterization pipelines for real-time view synthesis on commodity hardware. Our approach outperforms previous scene representations for real-time rendering in terms of accuracy, speed, and power consumption, and produces high quality meshes that enable applications such as appearance editing and physical simulation.
1 INTRODUCTION
BakedSDF targets high-quality mesh reconstruction and real-time novel view synthesis for large, unbounded real-world scenes. It combines a contracted-space neural SDF, mesh baking, and spherical-Gaussian appearance modeling to support fast rendering and downstream graphics applications.
- NeRF-like methods produce photorealistic novel views but are slow because their MLPs require hundreds of queries per pixel.
- Voxel-based accelerations reduce computation but require substantial GPU memory and custom raymarching, limiting commodity-hardware real-time rendering.
- BakedSDF extends a hybrid volume-surface representation to unbounded scenes, designing a well-behaved zero level set for high-resolution marching-cubes mesh extraction.
- Defining the SDF in contracted coordinates regularizes distant content and distributes the triangle budget toward the scene center.
- The method equips the mesh with spherical-Gaussian view-dependent appearance and fine-tunes it to reproduce captured input images.
- The resulting system renders at real-time frame rates on commodity devices and supports appearance editing and physics simulation with accurate, detailed meshes.
2 RELATED WORK
Prior view-synthesis methods use meshes or volumetric representations, while NeRF-based systems improve image quality but remain costly to render. BakedSDF addresses real-time rendering and mesh quality for unbounded scenes through a surface-oriented representation and baked appearance model.
- View synthesis renders novel scene views from captured images; sparse viewpoints make reconstructing a 3D representation important for convincing results.
- Classical pipelines commonly reconstruct triangle meshes and render novel views by reprojecting and blending observed images.
- Recent methods often use voxel grids or multiplane images because volumetric representations are well-suited to optimizing rendering losses.
- NeRF extensions improve appearance editing or decompose view-dependent appearance into material and lighting properties.
- Hybrid volume-surface methods augment NeRF representations but, according to the passage, target objects and bounded scenes rather than real-time unbounded-scene rendering.
- NeRF acceleration replaces expensive MLPs with grids, smaller networks, low-rank or sparse representations, and hash encodings, often trading computation for storage.
- Baking NeRFs into sparse volumetric structures enables real-time rendering on high-end hardware, but volumetric raymarching prevents real-time performance on commodity hardware.
3 PRELIMINARIES
NeRF renders novel views by sampling learned density and color along camera rays and compositing them into pixels, while mip-NeRF 360 adapts this process to unbounded scenes through contraction and proposal-based sampling.
- NeRF maps 3D positions and outgoing ray directions to volumetric density and color for view synthesis.
- Pixel colors are produced by evaluating NeRF at sampled points along each ray and compositing the resulting densities and colors.The compositing formula is a quadrature-based approximation of the volume rendering equation.
- NeRF learns scene geometry and appearance by optimizing an MLP so rendered training rays match their corresponding input colors.
- Sufficient multiview coverage allows this optimization to recover volumetric density and appearance that accurately describe the scene.
- Mip-NeRF 360 handles unbounded scenes with a contraction function and a proposal MLP for efficient hierarchical ray sampling.Contraction maps unbounded points to a bounded domain, while the proposal MLP bounds density and focuses samples for the NeRF MLP.
4 METHOD
BakedSDF combines a surface-based neural representation with mesh extraction and a spherical-Gaussian appearance model for efficient real-time rendering. Its pipeline is designed for unbounded scenes, well-behaved geometry, view-dependent effects, and practical mesh applications.
- Pipeline: The method first optimizes a surface-based representation using NeRF-like volume rendering, then bakes the geometry into a triangle mesh and trains a spherical-Gaussian appearance model.The final representation replaces the expensive first-stage appearance model with spherical Gaussians embedded at mesh vertices.
- Modeling density with an SDF: The scene representation combines mip-NeRF 360’s unbounded-scene modeling with VolSDF’s well-behaved surface properties by parameterizing density through an MLP-based signed distance function.The SDF is defined in contracted space, and the scene surface is its zero-level set.
- Modeling density with an SDF: The representation regularizes the SDF with an Eikonal constraint, while its contracted-space formulation also makes that constraint operate in contracted space.The regularization encourages the SDF to approximate signed Euclidean distance to its level set.
- Baking a high-resolution mesh: Mesh extraction queries the learned SDF on a regular contracted-space grid and applies Marching Cubes, using an iso-value of 0.001 to compensate for VolSDF’s density fall-off.Visibility and free-space culling remove unsupported crossings, and region growing fills holes missed during initial extraction.
- Baking a high-resolution mesh: Contracted-space sampling creates smaller world-space triangles near the origin and larger distant triangles, effectively using contraction as a level-of-detail strategy.The strategy matches the fact that desired rendered views are close to the scene origin.
- Modeling view-dependent appearance: Each mesh vertex stores diffuse color and spherical-Gaussian lobes for view-dependent appearance, with three lobes centrally and one in the periphery.The lobes use view direction to model effects such as specularities while keeping rasterization and appearance queries efficient.
5 EXPERIMENTS
Experiments evaluate BakedSDF on rendering accuracy, speed, energy, storage, mesh quality, and appearance modeling. The method performs strongly against real-time baselines while retaining detailed meshes, with limitations for transparency and fine structures.
- The evaluation measures rendering accuracy alongside speed, energy, memory, and storage requirements against offline and real-time baselines.
- Across all three benchmark error metrics, the real-time model outperforms the two evaluated state-of-the-art real-time baselines on mip-NeRF 360 scenes.Qualitative renderings also show more detail and fewer artifacts than prior real-time methods.
- BakedSDF uses 1.27× the storage of MobileNeRF and 4.07× that of Instant-NGP, while performance is measured at 1920 × 1080 on different hardware setups.
- Its performance advantage over MobileNeRF is likely associated with avoiding alpha-masking overdraw and replacing MobileNeRF’s radiance MLP with spherical Gaussians.
- Compared with Deep Blending, BakedSDF achieves higher quality while using a simpler browser-renderable representation; Deep Blending’s outdoor storage is 2.66× higher, averaging 1154.78 MB.
- Mesh extraction: Qualitative mesh comparisons find BakedSDF smoother and higher fidelity than COLMAP, MobileNeRF, and Mip-NeRF 360-derived meshes, whose geometry contains noise or inaccuracies.
- Appearance model ablation: Reducing spherical Gaussians from the proposed configuration monotonically degrades accuracy, while three peripheral Gaussians overfit and a single peripheral Gaussian reduces vertex size by 1.52×.
- Limitations: The fully opaque mesh representation may struggle with semi-transparent content, and mesh-based reconstruction can fail on small or detailed geometry such as foliage and thin structures.
6 CONCLUSION
BakedSDF produces high-quality meshes for real-time rendering of large unbounded real-world scenes. Its meshed view-dependent representation achieves state-of-the-art speed and accuracy while supporting downstream applications.
- BakedSDF optimizes a hybrid neural volume-surface representation, extracts a triangle mesh with view-dependent vertex attributes, and optimizes the meshed representation against captured images.
- The resulting mesh achieves state-of-the-art real-time view synthesis in speed and accuracy and is sufficiently high-quality for downstream applications.
A TRAINING AND OPTIMIZATION DETAILS
Training uses mip-NeRF 360-inspired optimization settings and hierarchical sampling, while vertex attributes are optimized through a compressed hash-grid representation.
- The SDF model is trained with 250k Adam iterations, batch size 214, and a learning rate interpolated from 2 · 10^-3 to 2 · 10^-5.
- The model uses two proposal resampling stages with 64 samples and one NeRF evaluation stage with 32 samples, optimizing Lprop + 0.1LSDF.
- During optimization, vertex attributes use Instant NGP with L = 18, T = 221, and Nmax = 8192, trained with Adam for 150k iterations.
B TWEAKS FOR A COMPELLING VIEWER
The viewer pipeline optimizes background appearance and adds conservative bounding geometry to reduce visible transitions between reconstructed content and the background.
- A global clear color is optimized as an appearance parameter for pixels in the training data without valid scene content.
- Before mesh extraction, the SDF is enclosed by a conservative convex hull formed from 32 randomly oriented planes and inflated by ×1.025.
C BASELINES DETAILS
The baselines are configured and evaluated to align rendering performance with test-set image-quality measurements, including standardized resolution choices and disabled dynamic upscaling.
- The MobileNeRF viewer is evaluated at test-set resolution for image quality and 1920 × 1080 for runtime performance.These choices separate quality evaluation from a display-representative performance measurement.
- Instant NGP is adapted for unbounded large scenes using the official big.json configuration, a 4× larger batch size, and scene scale increased from 16 to 32.The cited changes are reported not to significantly affect Instant NGP render time.
- Dynamic upscaling is disabled for Instant NGP performance comparisons so runtime figures correspond to test-set quality metrics computed without upscaling.