Source-linked AI summary
Neural Geometric Level of Detail: Real-time Rendering with Implicit 3D Shapes
Towaki Takikawa, Joey Litalien, Kangxue Yin, Karsten Kreis, Charles Loop, Derek Nowrouzezahrai, Alec Jacobson, Morgan McGuire, Sanja Fidler
TL;DR
Large fixed neural networks make neural SDF rendering too expensive for real-time graphics. The paper introduces a sparse-octree feature representation with adaptive and continuous LODs, decoded by small MLPs and rendered with tailored traversal. It reports state-of-the-art reconstruction quality alongside real-time rendering and substantial efficiency gains.
Problem
Large fixed-size neural SDF networks require many expensive evaluations per pixel, limiting their use in real-time graphics while smaller networks can reduce generality or reconstruction quality.
Method
The method stores learned feature vectors in a sparse voxel octree whose levels provide LODs, decodes them with small MLPs, and uses tailored sphere-tracing traversal.
Results
The method achieves state-of-the-art reconstruction results across datasets and metrics, with 4 737 inference floats and 99% fewer inference parameters than FFN [47].
Takeaways & Limitations
The representation combines high-fidelity, expressive neural implicit geometry with real-time rendering and a reasonable memory footprint.
Takeaways & Limitations
The approach depends heavily on training point samples, making extremely large scenes and very thin, volume-less geometry difficult to represent; traditional animation and deformation are also difficult.
Abstract
from arXiv · showhide
Neural signed distance functions (SDFs) are emerging as an effective representation for 3D shapes. State-of-the-art methods typically encode the SDF with a large, fixed-size neural network to approximate complex shapes with implicit surfaces. Rendering with these large networks is, however, computationally expensive since it requires many forward passes through the network for every pixel, making these representations impractical for real-time graphics. We introduce an efficient neural representation that, for the first time, enables real-time rendering of high-fidelity neural SDFs, while achieving state-of-the-art geometry reconstruction quality. We represent implicit surfaces using an octree-based feature volume which adaptively fits shapes with multiple discrete levels of detail (LODs), and enables continuous LOD with SDF interpolation. We further develop an efficient algorithm to directly render our novel neural SDF representation in real-time by querying only the necessary LODs with sparse octree traversal. We show that our representation is 2-3 orders of magnitude more efficient in terms of rendering speed compared to previous works. Furthermore, it produces state-of-the-art reconstruction quality for complex shapes under both 3D geometric and 2D image-space metrics.
1. Introduction
Neural SDFs represent complex, topology-unrestricted geometry but are difficult to render in real time because large fixed networks require many expensive evaluations. Neural Geometric LOD addresses this with an adaptive sparse-octree representation, small decoders, and tailored traversal for high-quality real-time rendering.
- Motivation: Large fixed MLP-based neural SDFs can require hundreds of distance evaluations per pixel and hundreds of millions of operations per pixel.Small networks improve speed but can sacrifice generality and reconstruction quality, while fixed-size networks cannot express geometry beyond their capacity.
- Approach: The method adaptively scales neural SDFs across multiple levels of detail and smoothly interpolates between geometric scales.It uses a sparse voxel octree with learned features and a shallow decoder, enabling detailed reconstruction with a reasonable memory footprint.
- Efficiency: The tailored sparse-octree traversal renders geometry close to 100× faster than DeepSDF [39].The paper also reports frametimes over 500× faster than NeRF [34] and 50× faster than NSVF [26] in similar experimental settings.
- Quality and scope: The representation captures multiple LODs while reconstructing 3D geometry with state-of-the-art quality.It is also described as compressed, with higher visual fidelity than traditional methods, and as generalizing across different geometries from a single learned example.
- Applications: Real-time operation positions the method as a modular building block for scene reconstruction from images, robotics navigation, and shape analysis.These are envisioned downstream applications of the representation's real-time nature.
2. Related Work
Related work spans mesh-based LOD, neural implicit surfaces, and differentiable rendering. The paper positions neural SDFs as a representation that can preserve smooth surfaces and blend LODs while using learned features and specialized rendering.
- Level of Detail: Mesh decimation reduces geometry by removing mesh elements, but meshes can suffer discretization errors under low memory and blend poorly between LODs.SDFs instead represent smooth surfaces with less memory and can smoothly blend between LODs.
- Neural Implicit Surfaces: Implicit surface methods encode geometry in latent vectors or network weights that parameterize surfaces through level-sets.Neural SDFs are one such implicit representation.
- Neural Geometric LOD: The proposed architecture stores learned feature vectors in a sparse voxel octree and decodes them into signed distances with a shallow MLP.Its octree levels define LODs, while trilinear feature interpolation supports local surface representation.
- Neural Implicit Surfaces: Prior neural shape methods include large auto-decoder MLPs, periodic-function networks, compact single-shape overfitting, and progressively growing networks.These approaches differ in representation size, reconstruction quality, and whether intermediate representations are retained.
- Neural Rendering: Differentiable renderers for implicit surfaces use ray marching or sphere tracing and require differentiability with respect to ray-surface intersections.The paper's representation can leverage these rendering techniques while adding a traversal algorithm tailored to its architecture.
3. Method
The method represents neural SDFs with a sparse octree feature volume whose levels provide adaptive, continuously interpolated detail, then renders it using sparse ray traversal and adaptive sphere tracing. Small LOD-specific MLPs decode interpolated features while avoiding queries in empty space.
- Neural Geometric Levels of Detail: An SDF is represented over a sparse voxel octree, with learnable corner features and increasingly fine geometric detail at deeper levels.Voxels are allocated only where they contain surface geometry, while each octree level defines a distinct LOD.
- Neural Geometric Levels of Detail: At a requested LOD, the method traverses containing voxels, trilinearly interpolates features across levels, sums them, and decodes the result with an LOD-specific MLP.The query combines the point x with the accumulated feature vector before producing a signed distance.
- Neural Geometric Levels of Detail: Continuous LOD is obtained by linearly interpolating predicted signed distances from neighboring discrete octree levels.The fractional part of the desired LOD determines the interpolation weight, enabling smooth transitions between levels.
- Training: Jointly training every discrete LOD with summed per-level losses ensures that each level represents valid geometry.Optimization updates both decoder parameters and octree features using importance-sampled points from the bounding volume and surfaces.
- Interactive Rendering: Rendering uses precomputed ray-SVO intersections, ray-AABB skipping through empty space, and sphere-tracing steps computed from interpolated features.For points inside a voxel, parent features are recursively gathered and decoded; outside points advance to the next intersected voxel instead of querying empty space.
4. Experiments
Experiments evaluate reconstruction quality, convergence, generalization, rendering speed, and geometry simplification across mesh and analytic SDF datasets. The representation achieves strong reconstruction and rendering results while supporting multiple LODs and reduced inference or storage costs.
- Experimental setup: Experiments cover mesh reconstruction, analytic SDFs, convergence, real-time rendering, generalization, and geometry simplification across several datasets.Evaluations use ShapeNet, Thingi10K, TurboSquid, Thingi32, and difficult Shadertoy examples.
- Mesh reconstruction: Across all datasets and metrics, the architecture achieves state-of-the-art results, with better results starting at LOD 3 and 99% fewer inference parameters than FFN.It also uses 37% fewer inference parameters than Neural Implicits while showing better reconstruction quality.
- Analytic SDFs: The architecture captures high-frequency details in difficult Oldcar and Mandelbulb analytic SDFs, whereas FFN and SIREN appear to fail entirely.These cases include discontinuities, non-metric distance fields, and recursive fractal structure.
- Convergence: The model converges to better reconstruction with roughly 45% of DeepSDF and FFN training time on Thingi32.Finetuning from pretrained weights helps lower LODs, but the difference is small.
- Rendering performance: The sparse-optimized renderer is more than 100× faster than DeepSDF while achieving better visual quality with fewer parameters.Lower LODs reduce memory consumption and storage, although optimized rendering benefits less from reduced LOD because MLP inference and memory I/O dominate.
- Geometry simplification: Against mesh decimation, the method’s perceptual-quality advantage increases as memory decreases because SDFs preserve smooth features and blend between LODs.The comparison uses image-based normal error on the Lion statue from Thingi32.
5. Limitations and Future Work
Neural Geometric LOD combines a sparse-octree geometry encoding with a small neural network to provide real-time, expressive implicit-shape rendering. The approach remains constrained by training samples, extremely large or thin geometry, and conventional animation or deformation.
- The method achieves state-of-the-art geometry reconstruction quality while supporting real-time rendering with an acceptable memory footprint.
- A small surface-extraction network, sparse-octree encoding, LOD support, and tailored sphere tracing jointly provide computational performance and expressiveness.
- Training depends heavily on point samples, making extremely large scenes or very thin, volume-less geometry difficult to represent.
- The representation does not easily support animation or deformation using traditional methods.
- The authors identify these representation challenges as promising directions for future research.
A.1. Architecture
The architecture uses shallow single-hidden-layer MLPs to decode SVO feature vectors into distances, while the SVO organizes geometry across resolution levels and allocates nodes sparsely near surfaces.
- Single-hidden-layer MLPs use hidden dimension h = 128, ReLU intermediate activation, and no output activation to support arbitrary distances.
- The SVO stores feature vectors with dimension m = 32 and initializes voxel features with a Gaussian prior of σ = 0.01.
- SVO level resolution follows rL = r0 · 2L with r0 = 4 and is capped at Lmax ∈{5, 6} according to geometry complexity.
- Each level can contain up to rL3 voxels, but the implementation allocates far fewer nodes because surfaces are sparse in R3.
A.2. Sampling
The point-cloud datasets use uniform, surface, and near-surface sampling schemes to provide training points across the bounding volume and around geometry.
- The dataset pipeline implements multiple sampling schemes for generating point clouds.
- Uniform: Uniform sampling draws random positions from the bounding volume B = [−1, 1]3 using three uniformly distributed random numbers.
- Surface: Surface sampling uses separate mesh and SDF procedures to select points on or associated with the surface.
- Near: Near-surface mesh points are generated by perturbing surface samples with Gaussian noise having σ = 0.01.
A.3. Training
Training uses resampled mixtures of surface, near, and uniform points, with L2 losses summed across LODs; CUDA-based sparse rendering implements efficient ray processing and sphere tracing.
- Training setup: Training uses 500 000 resampled points per epoch, batch size 512, Adam learning rate 0.001, and a 2:2:1 surface-near-uniform split.
- Training setup: The representation minimizes summed per-LOD L2 distance losses, training all LODs jointly for ShapeNet150 and Thingi32.
- Training setup: TurboSquid16 uses progressive training, adding levels from Lmax −1 downward every 100 epochs after first training the highest LOD.
- Rendering implementation: The sparse renderer uses CUDA, cub, and libtorch with kernel fusion, while ray-AABB intersection supports the sparse octree algorithm.
- Rendering implementation: Sphere tracing stops when the iteration limit or bd < δ criterion is met, checks oscillation, clips at depth 5, and caps iterations at k = 200.
- Rendering implementation: Reported frametimes include only primary ray tracing and normal computation, excluding secondary effects such as shadows.
B. Experiment Details
The experiments implement four neural SDF baselines with controlled architectural and training choices, including a shared sampling scheme and loss setup.
- All baselines omit a final-layer activation function to avoid restricting the range of predicted distances.
- DeepSDF is overfit to each shape using spatial coordinates without latent vectors or weight normalization.
- FFN uses σ = 8 as a trade-off between high-frequency noise and detail, although reconstruction quality is sensitive to this hyperparameter.
- SIREN omits the Eikonal regularizer and uses a simple L2 loss to fit non-metric SDFs.
- Neural Implicits is implemented without paper-specific changes, except for a shared sampling scheme controlling training variability across baselines.
B.2. Reconstruction Metrics
Reconstruction is evaluated with geometry and image-space metrics, using ray-based surface sampling and parallel sparse ray-octree traversal procedures.
- Geometry Metrics: Chamfer-L1 distance uses surface samples obtained by sphere-tracing rays rather than Marching Cubes meshes to avoid additional extraction error.
- Image Metrics: Normal-L2 is measured from 32 fixed spherical-Fibonacci camera positions at 512×512 resolution on the shared predicted and ground-truth masks.
- Sparse Ray-Octree Intersection: DECIDE marks each ray-voxel pair as intersecting or non-intersecting and records occupied-child counts unless the final LOD is reached.
- Sparse Ray-Octree Intersection: EXCLUSIVESUM converts the decision list into conflict-free output indices for parallel list construction.
- Sparse Ray-Octree Intersection: COMPACTIFY retains hit ray-voxel pairs at the final step, while SUBDIVIDE expands nonterminal hits into ordered child voxels for the next level.
- Sparse Ray-Octree Intersection: SUBDIVIDE orders child voxels front-to-back relative to each ray using a precomputed 8 × 8 table.
D. Additional Results
Additional examples extend the reported results across the evaluated datasets, while revealing artifacts and thin-feature weaknesses in particular cases.
- Additional result examples are provided for each dataset, with a supplementary video demonstrating real-time rendering.
- FFN exhibits white patch artifacts on City and Cabin because sphere tracing can miss surfaces when learning a conservative metric SDF is difficult.
- Additional Thingi32 results are presented as zoomed-in examples.
- Our method struggles with thin, flat, low-volume features such as Jetfighter wings and the back of the Chair.