Source-linked AI summary
Real-time Neural Radiance Caching for Path Tracing
Thomas Müller, Fabrice Rousselle, Jan Novák, Alexander Keller
TL;DR
Real-time path-traced global illumination must handle dynamic scenes without relying on scene-specific pretraining or complex cache management. The paper trains a neural radiance cache online during rendering, using self-training and fused implementation to provide multi-bounce illumination efficiently. It reports substantial noise reduction with little bias, real-time performance, and a 13.6× average speedup at equal MRSE, while retaining limitations in sharp-detail capture and frequency-encoding artifacts.
Problem
Path-traced global illumination is difficult for real-time rendering, while pretrained networks have not demonstrated generalization to novel dynamic configurations and unseen content.
Method
The method trains a neural radiance cache online during rendering, using self-training for inexpensive multi-bounce targets and a streamlined fully fused network implementation.
Results
13.6× average speedup at equal MRSE across test scenes, while ReSTIR plus NRC reduces MRSE by 1–2 orders of magnitude and exceeds 60 frames per second in all scenes.
Takeaways & Limitations
The cache provides real-time global illumination with low noise and little bias while handling dynamic content with predictable performance and resource consumption.
Takeaways & Limitations
The cache does not capture sharp detail absent from network inputs and exhibits subtle axis-aligned stripes from frequency encoding.
Abstract
from arXiv · showhide
We present a real-time neural radiance caching method for path-traced global illumination. Our system is designed to handle fully dynamic scenes, and makes no assumptions about the lighting, geometry, and materials. The data-driven nature of our approach sidesteps many difficulties of caching algorithms, such as locating, interpolating, and updating cache points. Since pretraining neural networks to handle novel, dynamic scenes is a formidable generalization challenge, we do away with pretraining and instead achieve generalization via adaptation, i.e. we opt for training the radiance cache while rendering. We employ self-training to provide low-noise training targets and simulate infinite-bounce transport by merely iterating few-bounce training updates. The updates and cache queries incur a mild overhead -- about 2.6ms on full HD resolution -- thanks to a streaming implementation of the neural network that fully exploits modern hardware. We demonstrate significant noise reduction at the cost of little induced bias, and report state-of-the-art, real-time performance on a number of challenging scenarios.
1 INTRODUCTION
The paper targets real-time path-traced global illumination in fully dynamic, arbitrary scenes, avoiding pretraining through online neural-cache adaptation. Self-training, a streamlined architecture, and fused GPU implementation enable efficient multi-bounce illumination and real-time rendering.
- Motivation: Path-traced global illumination remains difficult for real-time rendering, particularly with high-order indirect illumination.Spatial, directional, and temporal correlations motivate radiance reuse.
- Motivation: Radiance caching can accelerate rendering but often requires human intervention or complex heuristics to reduce artifacts.The proposed neural cache aims to replace these difficult heuristics.
- Design goals: Fully dynamic, robust caching requires arbitrary camera, lighting, geometry, and material changes without precomputation.The cache is intended to remain agnostic of materials and scene geometry.
- Challenge: Pretrained networks face an un demonstrated generalization challenge on novel dynamic configurations and previously unseen content.The paper states that prior approaches had not demonstrated this form of generalization for radiative quantities.
- Approach: Online optimization during rendering sidesteps the generalization challenge through fast adaptation, but fitting training and inference into real-time loops is non-trivial.The method relies solely on optimizing the model online.
- Contributions: Self-training uses later path vertices to produce inexpensive radiance estimates, providing multi-bounce illumination while tracing single rays or very short paths.A streamlined architecture and fully fused GPU implementation improve convergence, fidelity, and hardware efficiency.
- Results: The system achieves real-time framerates on current hardware across diverse material and lighting configurations and adapts temporally to dynamic geometry and lighting.An off-the-shelf denoiser also shows improved temporal coherence when using the cache.
2 RELATED WORK
Related work spans precomputation-based, fully dynamic, and neural approaches for accelerating global illumination. These methods reuse or approximate radiance, visibility, and shading information under differing assumptions and representations.
- Scope: Radiance-caching research includes precomputation-based techniques, fully dynamic algorithms, and artificial-neural-network approaches.The paper points to a broader survey for an extensive treatment.
- Radiance caching: Traditional radiance caching assumes irradiance varies smoothly across scenes, recovering texture detail through albedo modulation.Irradiance probe volumes extended this family toward reusable spatial representations.
- Precomputation-based techniques: Precomputation methods commonly assume fixed scene lighting and geometry before storing irradiance in texture space or world space.Lightmaps and light probes are representative storage mechanisms.
- Fully dynamic techniques: Fully dynamic global-illumination methods reuse shading and visibility computations across pixels and gain efficiency through approximations.Examples include photon mapping, many-light rendering, radiosity maps, and hierarchical point-cloud representations.
- Path guiding: Path-guiding methods learn incident-radiance approximations suitable for importance sampling using mixtures, probability trees, or neural normalizing flows.This makes path guiding closely related to radiance caching.
- Neural techniques: Neural approaches approximate visual phenomena in screen or world space and may be pretrained across scenes or fitted to a single scene.Single-scene fitting is identified as most closely related to this work.
3 NEURAL RADIANCE CACHING
The method trains a single neural radiance cache online while rendering, using short rendering paths, sparse training suffixes, and self-training to approximate multi-bounce illumination. EMA weight averaging and specialized input encodings improve temporal stability and adaptation, while path termination limits computation.
- Cache representation: The cache maps spatio-directional and material-related inputs to scattered radiance, targeting the most computationally expensive part of the rendering equation.Inputs include position, direction, normal, roughness, diffuse reflectance, and specular reflectance; the network approximates scattered radiance by cached radiance.
- Rendering: Rendering traces one short path per pixel and queries the cache only after path spread is large enough to blur small cache inaccuracies.Intermediate vertices use next-event estimation, while termination follows a footprint-based heuristic.
- Online training: Training extends typically under 3% of rendering paths by a few vertices and reuses their radiance estimates as cache-training references.The initial rendering segment is shared, and most training suffixes contain one additional vertex.
- Online training: Self-training evaluates the cache at terminal training vertices and propagates those values backward, exchanging much of path-tracing noise for potential bias while iterated updates increase simulated light bounces.A small unbiased subset, u = 1/16, injects source values, but the method may miss transport or query insufficiently trained locations.
- Temporal stability: Aggressive online optimization can cause temporal flickering, so an EMA of network weights is used for rendering without feeding back into training.The chosen setting is α = 0.99, balancing fast adaptation and temporal stability.
4 FULLY FUSED NEURAL NETWORKS
The fully fused implementation accelerates the neural network by minimizing global-memory traffic and exploiting on-chip GPU memory. It improves throughput substantially, while training remains slower than inference because backpropagation requires additional memory traffic and matrix multiplications.
- Memory-efficient execution: Narrow networks are bottlenecked by linear memory traffic because modern GPUs provide more computational throughput than memory bandwidth.The design therefore prioritizes reducing global-memory traffic and using caches, shared memory, and registers.
- Memory-efficient execution: A single GPU kernel implements the entire neural network, limiting slow global-memory accesses to reading inputs and writing outputs.The from-scratch implementation is tailored to the network architecture and target GPU.
- GPU mapping: Thread blocks independently evaluate network layers while keeping intermediate activations in fast memory when they fit.The batch is partitioned into block-column segments, with each segment processed by one thread block.
- GPU mapping: 64-neuron-wide networks use register-resident weight blocks that are reused across computations, reducing repeated memory loads.Each warp loads its weight-matrix block from global memory once.
- GPU mapping: On an NVIDIA RTX 3090, the implementation reaches a shared-memory sweet spot when each thread block processes 128 batch elements.Each block multiplies a 64 × 64 weight matrix by a 64 × 128 data chunk.
- Training cost: Training is roughly 4×–5× slower than inference because backpropagation adds global-memory traffic and weight-gradient matrix multiplications.The additional matrix multiplications use CUTLASS GEMM routines.
5 PRACTICAL CONSIDERATIONS
The paper combines architectural simplifications, reflectance factorization, inexpensive encoding approximations, and relative-loss training to improve practical rendering efficiency and robustness. It also compares optimizers and presents complementary direct- and indirect-illumination techniques for real-time rendering.
- Architecture: The network has seven fully connected layers, with five 64-neuron ReLU hidden layers and a three-value RGB output layer.Removing bias vectors simplifies the fused implementation without measurable quality benefit.
- Reflectance factorization: Reflectance factorization multiplies network output by diffuse albedo plus specular reflectance, improving textured-color reproduction and focusing prediction on complementary detail.For Lambertian materials, this corresponds to irradiance factorization.
- Encoding: Replacing Gaussian kernels with quartic kernels and sine functions with triangle waves reduces cost per frame by 0.25ms with no visible quality loss.These approximations target the expensive primitives used by one-blob and frequency encodings.
- Training: The relative L2 loss provides unbiased gradient estimates when reflected-radiance training targets are noisy.The loss is normalized by the neural prediction, with ε = 0.01 and stop-gradient applied to the normalization term.
- Optimization: Adam converges in the fewest iterations among the tested first-order optimizers with practically zero overhead, so it is used for all results.Shampoo converges slightly faster but adds 0.3 milliseconds per frame.
- Rendering pipeline: At 1 spp, ReSTIR supplies low-variance direct illumination while the neural radiance cache supplies low-variance indirect illumination.The two techniques are presented as complementary components alongside unbiased path tracing.
6 RESULTS AND DISCUSSION
The neural radiance cache reduces indirect-lighting noise while maintaining real-time performance, and remains effective across varied materials, geometry, and lighting. Its trade-offs include cache-query and training overhead, alongside residual detail and encoding artifacts.
- Real-time rendering: 1–2 orders of magnitude: ReSTIR combined with NRC reduces MRSE compared with path tracing while exceeding 60 frames per second in all scenes.The reported speedup is achieved with comparatively small performance impact.
- Real-time rendering: 13.6×: the average speedup across test scenes when the path-tracing baseline is rendered to equal MRSE.The comparison treats one rendered frame of the method as the target quality for baseline convergence.
- Self-training: Self-training captures multi-bounce light transport using inexpensive training paths, with roughly 1% overall overhead from an additional cache query per path.This avoids tracing longer paths to learn global illumination.
- Quality of the cache: The cache handles complex glossy transport, distant shadow detail, thin geometry without light leakage, and overall scene color across the evaluated scenes.These results are attributed to the cache’s agnosticism toward underlying materials and geometry.
- Limitations: The cache does not capture sharp detail absent from network inputs and exhibits subtle axis-aligned stripes caused by frequency encoding.The encoding remains necessary for representing spatial detail at scale, including far-away shadows.
- Comparison with DDGI: NRC trades slightly reduced performance and sometimes more noise for minimized bias, while recovering ambient occlusion without assuming a material model.DDGI is more performant and less noisy but can expose bias and lacks ambient occlusion; NRC’s termination criterion helps avoid remaining inaccuracies.
- Performance breakdown: 1.59 ms: a full-frame neural radiance cache query, compared with 0.58 ms for DDGI; both remain within reasonable real-time cost.The neural query is slower, but the difference is smaller than might be expected given the cost of neural networks.
- Performance breakdown: 1.11 ms vs. 1.37 ms: NRC training is cheaper than DDGI volume training on average.NRC uses 65536 training records per frame, while the stated DDGI configuration traces 1048576 update rays.
7 DISCUSSION AND FUTURE WORK
The discussion identifies scope boundaries and future directions for neural radiance caching, including limited benefits for difficult transport and opportunities for broader rendering applications.
- Precomputation: Precomputation is optional because the cache rapidly learns the current situation, with 8 frames sufficient in one experiment.Static network weights and their domain of validity remain open questions.
- Cache artifacts: Residual low-frequency scintillation and axis-aligned oscillations remain, limiting desirable use of the cache at primary path vertices.An exponential moving average suppresses high-frequency temporal flickering, but further stabilization is needed.
- Additional network inputs: Shadows and caustics correlate poorly with available network inputs, so they are learned slowly or not at all when the network or targets are insufficient.These high-frequency features are not adequately represented by input encodings alone.
- Offline rendering: Neural radiance caching could benefit offline rendering by capturing high-order indirect illumination without tracing and shading long paths.This may help address path-length limitations in batch rendering.
- Volumes: The cache parameterization also applies to volumetric rendering, although the straightforward implementation requires in-depth investigation.A single neural radiance cache is used for the volume queries.
- Path guiding: Indirect cache use remains the main noise source because hemispherical sampling increases variance despite a noise-free cache approximation.Real-time neural importance sampling is suggested as a possible complementary direction.
- Improved path termination: Neural radiance caching provides little benefit for transport dominated by long, branching specular chains, such as scenes containing substantial glass.Improved termination heuristics and sharper cache representations are proposed for this case.
- Denoising: The cache can complement screen-space denoising, with a preliminary off-the-shelf experiment suggesting potential for tighter coupling.The denoiser was trained on data not representative of this method.
8 CONCLUSION
The paper concludes that neural radiance caching enables real-time global illumination for dynamic content through online adaptation and fused neural networks. Its compute-oriented design trades substantial computation for robustness, while path guiding offers a potential improvement.
- Conclusion: The technique handles dynamic content with predictable performance and resource consumption through fully fused neural networks and online adaptation.The authors describe robustness as a collateral benefit of the engineering required for performance.
- Conclusion: The neural cache trades compute for memory efficiency, evaluating neurons broadly instead of storing sophisticated data structures whose memory is only partly accessed.This contrasts compute-wasteful neural processing with memory-wasteful competing techniques.
- Conclusion: Even with denoising enabled, the method stays well above 30 frames per second in the Living Room scene.The off-the-shelf denoiser achieves the lowest relative squared bias in that experiment.
- Conclusion: The neural approach is only twice as expensive as irradiance probes in the timings of Table 3.The comparison concerns reasonably well-optimized implementations.