Source-linked AI summary
DONeRF: Towards Real-Time Rendering of Compact Neural Radiance Fields using Depth Oracle Networks
Thomas Neff, Pascal Stadlbauer, Mathias Parger, Andreas Kurz, Joerg H. Mueller, Chakravarty R. Alla Chaitanya, Anton Kaplanyan, Markus Steinberger
TL;DR
NeRF-like rendering is limited by the cost of evaluating many samples along each ray, hindering real-time use of compact neural scene representations. DONeRF addresses this with a classified depth-oracle network that guides local sampling in a shading network, achieving large cost reductions while retaining compact MLPs. Its scope is static synthetic content, and standard depth maps remain problematic for partially transparent or mirror-like surfaces.
Problem
Excessive per-ray network evaluations make compact NeRF-like representations too expensive for real-time rendering.
Method
DONeRF uses a compact dual-network design: a classification-based depth oracle predicts important ray samples, and a locally sampled shading network accumulates radiance.
Results
24–98× lower computational cost is reported at NeRF’s memory budget, with equal quality for small scenes and significant improvements for large scenes.
Takeaways & Limitations
DONeRF enables high-quality real-time rendering of large synthetic scenes from compact neural representations without additional data structures or increased storage.
Takeaways & Limitations
The evaluation focuses on static content, while standard depth maps can provide insufficient sample locations for partially transparent and mirror-like surfaces.
Abstract
from arXiv · showhide
The recent research explosion around implicit neural representations, such as NeRF, shows that there is immense potential for implicitly storing high-quality scene and lighting information in compact neural networks. However, one major limitation preventing the use of NeRF in real-time rendering applications is the prohibitive computational cost of excessive network evaluations along each view ray, requiring dozens of petaFLOPS. In this work, we bring compact neural representations closer to practical rendering of synthetic content in real-time applications, such as games and virtual reality. We show that the number of samples required for each view ray can be significantly reduced when samples are placed around surfaces in the scene without compromising image quality. To this end, we propose a depth oracle network that predicts ray sample locations for each view ray with a single network evaluation. We show that using a classification network around logarithmically discretized and spherically warped depth values is essential to encode surface locations rather than directly estimating depth. The combination of these techniques leads to DONeRF, our compact dual network design with a depth oracle network as its first step and a locally sampled shading network for ray accumulation. With DONeRF, we reduce the inference costs by up to 48x compared to NeRF when conditioning on available ground truth depth information. Compared to concurrent acceleration methods for raymarching-based neural representations, DONeRF does not require additional memory for explicit caching or acceleration structures, and can render interactively (20 frames per second) on a single GPU.
1. Introduction
DONeRF targets the prohibitive cost of NeRF rendering by predicting important surface-near samples with a compact dual-network design. It preserves compactness while enabling real-time rendering of large synthetic scenes.
- Real-time photorealistic rendering remains demanding despite increasingly widespread, efficient machine-learning accelerators.
- NeRF uses 256 samples per view ray, with each sample requiring a full network evaluation.
- 37 petaFLOPS would be required to render NeRF at 1440×1600 pixels per eye and 90 Hz, while larger scenes require multiple NeRFs.
- DONeRF proposes a compact dual-network design in which an oracle predicts ray-sample locations and a shading network uses a few guided samples for final color.
- 24–98× lower computational cost is reported at NeRF’s memory budget, with equal quality for small scenes and improvements for large scenes.
- DONeRF targets real-time rendering of large-scale computer-graphics scenes from compact neural representations and is evaluated on static synthetic content.
2. Related work
Prior approaches accelerate neural rendering through explicit representations, optimized raymarching, or baking, but often trade away viewing range, compactness, or scalability. DONeRF instead retains compact MLPs and uses a local-sampling pipeline.
- Multi-plane and spherical image representations can render efficiently but require densely sampled inputs, increased memory, or limited viewing regions.
- Implicit neural scene representations avoid fixed internal resolution by mapping continuous inputs such as ray origins and directions to outputs.
- NeRF introduced fixed differentiable raymarching, transforming ray samples into Fourier features before MLP evaluation and accumulation.
- NeRF variants reduce runtime through techniques such as empty-space skipping with sparse voxel structures, but these approaches add representation structure.
- Baking NeRF components can improve performance but may require hundreds of MBs to GBs instead of the 4 MB compact representation.
- DONeRF combines a five-stage pipeline with compact MLPs, avoiding additional data structures or increased storage while targeting large-scale scenes.
3. Efficient Neural Rendering using Depth Oracle Networks
DONeRF replaces exhaustive NeRF raymarching with local sampling guided by a depth-oracle network. Its unified ray representation and view-cell formulation support compact rendering across scene views.
- DONeRF replaces NeRF’s MLP-based raymarching with local sampling focused on important samples around surfaces.
- The method uses a sampling oracle to predict view-ray sample locations and a shading network to accumulate NeRF-like samples into RGBA output.
- Rays are transformed into a unified space to remove input ambiguity, while depth is handled in a non-linear representation.
- View cells are bounding boxes with a primary orientation and maximum viewing angle that capture rays originating within the box.
- Smaller view cells reduce visible scene content and may improve quality, whereas larger view cells can be more memory efficient depending on network capacity.
4. Efficient Neural Sampling
Efficient neural sampling reduces raymarching cost by concentrating samples near surfaces and transforming depth space, while learned local sampling can preserve quality with very few samples. The section evaluates these strategies and motivates an oracle that predicts sampling locations without storing explicit surfaces.
- Non-linear and warped sampling: Logarithmic sampling focuses samples nearer the camera, while log+warp additionally pulls 3D space toward the view cell center for positional encoding.The warping uses a radial distortion based on an inverse square root transform.
- Local sampling: Given ground truth surface depth, local sampling places a fraction of NeRF’s samples around surfaces while retaining mostly equal quality.This establishes the best-case quality-speed tradeoff targeted by the learned oracle.
- Ablation results: With log+warp, non-local sampling can halve the sample count while achieving equal quality to uniform sampling.The result comes from the ablation over sample counts and four diverse scenes.
- Ablation results: With local sampling, reducing the ray to two samples causes nearly no quality decrease, while local log+warp adds about 1.3 dB over uniform sampling.Local NDC sampling remains less competitive until more than 64 samples.
- Sampling oracle: DONeRF replaces explicit ground-truth-depth inference with an MLP sampling oracle that predicts ideal locations for a locally sampled shading network.Ground truth depth is used during training because explicit surface representations are costly for compact neural rendering.
5. Sampling Oracle Network
The sampling oracle predicts likely surface regions along each ray so a shading network can concentrate samples without relying on explicit surface representations. DONeRF improves oracle consistency through classification, filtering, ray unification, and logarithmic depth processing.
- Sampling Oracle Network: The oracle network predicts sample locations along each ray, replacing explicit surface representations while guiding a raymarched shading network.It is designed to reduce the number of samples needed for neural rendering.
- 5.1. Classified Depth: Classification assigns each discretized ray segment a likelihood indicating whether it should receive samples or be skipped.High classification values identify segments for one or more samples; low values identify segments that can be skipped.
- 5.1. Classified Depth: The classification target uses the first surface depth and discretization boundaries to form a one-hot target trained with binary cross-entropy.C denotes classification value, ds the first surface depth, and dz and dz+1 the boundaries of segment z.
- 5.1. Classified Depth: Filtering the target across image dimensions and depth smooths labels, reducing false negatives while accepting additional samples in free space.This trades more false positives for a lower chance of missing surfaces, which would reduce image quality.
- Sampling Oracle Network: The oracle output is treated as a piecewise-constant PDF and sampled through the inverse CDF, concentrating shading samples where classification values are high.This converts the single oracle evaluation into local sample locations along the ray.
- Sampling Oracle Network: Ray unification maps ray origins onto a sphere around the view cell, while logarithmic ray inputs and 3D positions support consistent oracle predictions.The mapping makes identical rays share identical depth values and supports arbitrary views outside the view cell.
- 5.3. Optimal Filter Sizes and Depth Oracle Inputs: 3.3 dB–4.7 dB: the combined additions improve PSNR, with neighborhood filtering contributing 1.3 dB–1.8 dB and depth smoothing 0.1 dB.Ray unification adds about 0.6 dB, classification adds 0.6 dB–1.5 dB, and multiple ray samples as input add 1 dB.
6. Evaluation
DONeRF is evaluated against neural, explicit-structure, and image-based scene representations across image quality, rendering efficiency, and compactness. Across these comparisons, it achieves strong quality with few samples, real-time performance, and favorable memory requirements, while remaining sensitive to challenging view-dependent phenomena.
- Evaluation setup: DONeRF is evaluated on image quality, generation efficiency, and representation compactness against NeRF, NSVF, LLFF, and NeX variants.The study uses PSNR and FLIP for quality, FLOP per pixel and compactness-related measures for efficiency and storage, with reference depth available during training and estimated depth as a proof of concept.
- Efficiency: 20 frames per second are achieved for an 800×800 image with 2 samples per ray on an NVIDIA RTX 3090.The five-stage pipeline takes 51.3 ms total; shading remains the dominant cost, and runtime increases nearly linearly from 34.9 ms at 2 samples to 270.4 ms at 16.
- Comparison with NeRF: 15–78× faster evaluation and up to 7 dB higher PSNR are achieved than NeRF, while equal or better quality requires only 2−4 samples instead of 256.With the improved log+warp sampling strategy applied to NeRF, DONeRF still reaches equal quality with 8 samples; the comparison is especially favorable for large open scenes.
- Comparison with NSVF: NSVF requires more than 2× DONeRF’s memory for competitive quality and takes about 6× longer to train.NSVF variants with similar quality take 43× longer to evaluate, while NSVF-small is 2.5 dB lower in quality and 27× slower than DONeRF-2.
- Comparison with LLFF: LLFF has the lowest quality among tested methods and requires more than 1000× DONeRF’s memory, but it is the fastest for novel-view generation.For Forest and San Miguel, LLFF’s PSNR is close to NeRF but remains 1 dB–7 dB below DONeRF-2; its low-power advantage applies when memory is not a concern.
- Comparison with NeX: 25× more compactness and a 10× speedup are reported for DONeRF over NeX-MLP while using better quality with 4−8 samples for most scenes.NeX is faster to evaluate and competitive on San Miguel, but its MPI representation uses 25× more memory than DONeRF and suffers when views differ substantially from the reference.
- Without ground-truth depth: About 1 dB average loss is observed without ground-truth depth, yet DONeRF-X-noGT still achieves better quality than NeRF at much lower sample counts.Filtering depth targets and sampling around foliage support high-quality reconstruction, while transparent surfaces and fully reflective mirrors remain difficult for the low-sample, single-depth-estimate setup.
7. Conclusion, Limitations and Future Work
DONeRF achieves comparable or better quality than NeRF with far fewer samples and compact memory, but its evaluation remains bounded by limitations involving transparency, mirrors, depth supervision, and dynamic content.
- Conclusion: DONeRF achieves equal or better quality than NeRF with 256 samples across most scenes, using as few as 2 samples per ray and only two MLPs.The approach combines local sampling with a classified depth oracle to guide a raymarched shading network.
- Conclusion: DONeRF compares favorably in storage requirements and quality against other scene representations and light-field methods, while only image-based methods render faster.
- Limitations and Future Work: Standard depth maps can misplace samples for partially transparent and mirror-like objects because the first surface depth is insufficient.The authors identify multiple surface depths and removing the full-NeRF round trip without ground-truth depth as next steps.
- Limitations and Future Work: The evaluation focuses only on static content, although the sampling strategy could be adapted to dynamic-content consistency and combined with dynamic caching.The proposed combination is described as a possible route to further improve rendering efficiency while preserving compactness for streaming.
- Conclusion: DONeRF is presented as the first reliable raymarched neural scene representation to reach interactive frame rates without exhaustive caching.The authors describe this as opening the door to compact, high-quality dynamic rendering in real time.
Appendix A: Additional Results: Efficient Neural Sampling
The appendix reports per-scene results across different numbers of samples per ray using the sampling methods introduced in Section 4.
- Additional Results: Efficient Neural Sampling: Tables 7–10 show per-scene results for varying numbers of samples per ray and the sampling methods described in Section 4.
Appendix B: Additional Results: Sampling Oracle Network
The appendix reports per-scene results for different sample counts and sampling-oracle configurations, with further configuration details in Section 5.
- Additional Results: Sampling Oracle Network: Tables 3–6 present per-scene results for varying numbers of samples per ray and multiple sampling-oracle configurations.Section 5 details the depth-oracle configurations used in these results.
Appendix C: Additional Evaluation Setup: Datasets
The evaluation uses diverse synthetic scenes spanning fine geometric detail, foliage, indoor structure, and detailed textures, while comparing NeRF and DONeRF with matched MLP architectures.
- Datasets: The Bulldozer scene contains fine, high-frequency building-brick details and mostly diffuse shading, and was re-rendered to fit the view-cell methodology.
- Datasets: The Forest scene contains high-frequency cel-shaded foliage and trees, with a foreground person requiring foreground and background representation.
- Evaluation Setup: NeRF and DONeRF use the same MLP architecture: 8 layers with 256 hidden units each and one skip connection, while the depth oracle has no skip connection.
- Datasets: The Classroom scene tests thin geometric details such as chair legs and detailed texture work in an indoor setting.
Appendix D: Additional Evaluation Setup: DONeRF
DONeRF and NeRF use related MLP architectures, but DONeRF separates depth prediction from shading and simplifies the oracle network.
- DONeRF and NeRF use an adapted NeRF architecture with 8 layers and 256 hidden units.Both use a single skip connection for encoded directions.
- DONeRF’s depth oracle omits skip connections and positional or directional encoding.It still uses ray unification and non-linear sampling.
- DONeRF uses two networks: an oracle for ray sampling and a shading network for rendering.The architecture is designed around distinct depth and appearance functions.
Appendix E: Additional Evaluation Setup: Baselines
The evaluation standardizes storage and computational accounting across neural-rendering baselines while documenting their implementation-specific training and evaluation procedures. Additional tables report sampling and depth-oracle ablations across four scenes.
- Evaluation protocol: Baseline storage includes all network weights, checkpoints, and images required to render novel views.FLOP per pixel are measured by NVIDIA Nsight Compute or directly from network evaluation.
- Baseline configurations: NeRF uses 64 coarse and 128 fine samples, totaling 256 network evaluations, with 300000 training iterations.The evaluation uses the same network architecture as DONeRF and samples from two images per iteration.
- Baseline configurations: NSVF is evaluated through a grid search over initial voxel size, selecting variants for quality, quality-performance tradeoff, and other constraints.The supplied passage specifically describes the quality and quality-performance selection scenarios.
- Baseline configurations: LLFF quality metrics are computed after removing a 10 % border from every image side, following the authors’ recommendation.
- Baseline configurations: NeX uses parameters fitting within 11 GB GPU RAM and is rendered with the authors’ real-time web viewer.Training uses 300000 iterations with 4096 samples per iteration.
- Ablations: Depth-oracle configurations are ablated on Bulldozer, Forest, Classroom, and San Miguel scenes.
- Ablations: Sampling methods are ablated separately on Bulldozer, Forest, Classroom, and San Miguel scenes.