Source-linked AI summary
Tri-MipRF: Tri-Mip Representation for Efficient Anti-Aliasing Neural Radiance Fields
Wenbo Hu, Yuling Wang, Lin Ma, Bangbang Yang, Lin Gao, Xiao Liu, Yuewen Ma
TL;DR
Neural radiance fields face a trade-off between high-quality anti-aliased rendering and efficient reconstruction because common point-based sampling ignores pixel area. Tri-MipRF factorizes pre-filtered 3D features into three orthogonal 2D mipmaps and uses cone casting with spheres for area-aware sampling. It achieves state-of-the-art rendering quality and reconstruction speed, with reconstruction within five minutes and a model size 25% smaller than Instant-ngp.
Problem
Point-based neural radiance-field rendering can blur close-up views and alias distant views, while super-sampling and direct 3D pre-filtering increase computational or memory costs.
Method
Tri-MipRF factorizes pre-filtered 3D feature space into three orthogonal 2D mipmaps and uses cone casting with inscribed spheres for efficient area-aware sampling.
Results
Tri-MipRF achieves state-of-the-art rendering quality and reconstruction speed, reconstructing within five minutes on one GPU with a model size 25% smaller than Instant-ngp.
Takeaways & Limitations
The representation supports fine-grained close-up details, alias-free distant views, compact storage, and real-time rendering above 60 FPS on consumer-level devices.
Abstract
from arXiv · showhide
Despite the tremendous progress in neural radiance fields (NeRF), we still face a dilemma of the trade-off between quality and efficiency, e.g., MipNeRF presents fine-detailed and anti-aliased renderings but takes days for training, while Instant-ngp can accomplish the reconstruction in a few minutes but suffers from blurring or aliasing when rendering at various distances or resolutions due to ignoring the sampling area. To this end, we propose a novel Tri-Mip encoding that enables both instant reconstruction and anti-aliased high-fidelity rendering for neural radiance fields. The key is to factorize the pre-filtered 3D feature spaces in three orthogonal mipmaps. In this way, we can efficiently perform 3D area sampling by taking advantage of 2D pre-filtered feature maps, which significantly elevates the rendering quality without sacrificing efficiency. To cope with the novel Tri-Mip representation, we propose a cone-casting rendering technique to efficiently sample anti-aliased 3D features with the Tri-Mip encoding considering both pixel imaging and observing distance. Extensive experiments on both synthetic and real-world datasets demonstrate our method achieves state-of-the-art rendering quality and reconstruction speed while maintaining a compact representation that reduces 25% model size compared against Instant-ngp.
1. Introduction
Tri-MipRF addresses the quality–efficiency dilemma in neural radiance fields with a compact representation and area-aware rendering. It combines high-fidelity, anti-aliased results with reconstruction within five minutes on a single GPU.
- Motivation: MipNeRF delivers high-quality anti-aliased renderings but requires more than three days for reconstruction and minutes to render a frame.Hash encoding reduces reconstruction from days to minutes but can produce blurred close-up and aliased distant renderings when point-based sampling ignores pixel area.
- Results: Tri-MipRF achieves high-fidelity close-up details, alias-free distant views, and reconstruction within five minutes on a single GPU.The paper reports extensive quantitative and qualitative evaluation on public benchmarks and images captured in the wild.
- Method: Tri-MipRF models pre-filtered 3D feature space using three orthogonal 2D mipmaps and a shallow MLP.The three planes are planeXY, planeXZ, and planeYZ; only their base mipmap levels are stored, supporting a compact hybrid representation.
- Method: Cone casting represents each pixel with a cone and samples it using inscribed spheres whose occupied areas determine Tri-Mip features.The resulting features are mapped to density and color before volume rendering integrates the samples into the pixel color.
- Results: 25% smaller model size than Instant-ngp accompanies Tri-MipRF’s state-of-the-art rendering quality and reconstruction speed.The method also achieves real-time rendering on consumer-level devices through a hybrid volume-surface rendering strategy.
2. Related Work
Neural rendering work has pursued anti-aliasing and acceleration separately, but area-ignorant point sampling remains problematic across viewing distances. Tri-MipRF addresses this with pre-filtered 3D features and cone-based area sampling.
- Anti-aliasing: Super-sampling reduces aliasing by increasing sampling rate but significantly raises computation as the number of samples per pixel grows.Pre-filtering computes filtered content before rendering and is therefore more suitable for real-time rendering.
- NeRF acceleration: NeRF acceleration methods reduce inference or reconstruction cost through spatial cells, fewer samples, cached values, or compact scene representations.These approaches target faster rendering or reconstruction but do not by themselves resolve area-ignorant sampling.
- Remaining gap: Treating each pixel as a single point causes excessive close-up blur and distant aliasing because sampling ignores the pixel’s area.Super-sampling can relieve the issue, but casting multiple rays per pixel increases reconstruction and rendering cost.
- Tri-MipRF: Tri-MipRF models pre-filtered 3D feature space with three 2D mipmaps and samples pixels through cones and spheres.The representation is designed to combine efficient area sampling with anti-aliased high-fidelity rendering.
3. Method
Tri-MipRF replaces point-based ray sampling with cone casting and inscribed spheres, then pre-filters sphere features through three orthogonal mipmaps for efficient anti-aliased rendering.
- Cone Casting: Cone casting models each pixel as a disc and samples the resulting cone with spheres inscribed along its central axis.Sphere centers lie on the cone axis, and their radii are chosen to fit the cone geometry.
- Volume Rendering: The tiny MLP maps sphere features and view direction to density and color, whose numerical quadrature produces each pixel’s rendered color.Photometric loss jointly optimizes the MLP weights and Tri-Mip parameters during training.
- Tri-Mip Encoding: Tri-Mip encoding decomposes 3D space into XY, XZ, and YZ planes, representing each plane with a mipmap to model pre-filtered features.The encoding uses three trainable mipmaps and queries features for sampled spheres through their orthogonal projections.
- Tri-Mip Encoding: Mipmaps derive lower levels by 2× downscaling, maintaining consistency across scales and helping reconstructed objects remain coherent at different distances.The base level is trainable, while subsequent levels are generated from the preceding level.
4. Experimental Evaluation
Tri-MipRF is evaluated on synthetic multi-scale and single-scale Blender data, plus in-the-wild captures, using rendering quality, reconstruction speed, storage, and qualitative detail as criteria.
- Multi-scale Blender Dataset: The multi-scale Blender dataset combines downscaled images at factors of 2, 4, and 8 to test reconstruction accuracy and anti-aliasing across resolutions.
- Quantitative Results: Tri-MipRF reconstructs in five minutes on one GPU, whereas MipNeRF takes around three days and Instant-ngp ↑5× takes around ten minutes.
- Qualitative Results: In close-up views, Tri-MipRF preserves fine-grained details, while Instant-ngp is blurry and Instant-ngp ↑5× improves quality at higher reconstruction cost.
- Qualitative Results: In distant views, Instant-ngp shows severe aliasing and jaggies, Instant-ngp ↑5× slightly reduces them, and Tri-MipRF produces anti-aliased renderings.
- Single-scale Blender Dataset: On the single-scale Blender dataset, Tri-MipRF outperforms the compared methods on PSNR, SSIM, and LPIPS while matching Instant-ngp for fastest reconstruction.
- Single-scale Blender Dataset: Tri-MipRF has a 48.2 MB model size, the smallest among hybrid methods, reducing storage consumption by 25% versus Instant-ngp.
- In-the-wild Captures: In-the-wild novel views reproduce detailed structures and appearances, with PSNR/SSIM values providing evidence of applicability.
5. Conclusion
Tri-MipRF combines Tri-Mip encoding, cone casting, and hybrid volume-surface rendering to target detailed close-up views, anti-aliased distant views, efficient reconstruction, and compact deployment.
- Tri-MipRF uses three mipmaps to model pre-filtered 3D features so cone-casting spheres can be encoded through area sampling.
- The method reconstructs within five minutes, uses a model 25% smaller than Instant-ngp, and enables real-time rendering above 60 FPS on consumer devices.
A. Characteristics Matrix
The characteristics matrix compares NeRF methods across quality and efficiency properties, identifying Tri-MipRF as the only method supporting all listed properties simultaneously.
- The matrix covers anti-aliasing, fast reconstruction, real-time rendering, and compact models as key quality and efficiency characteristics.
- Tri-MipRF is the only compared method marked as supporting anti-aliasing, fast reconstruction, real-time rendering, and a compact model together.
B.1. Tiny MLP
A tiny MLP maps Tri-Mip features and viewing direction to density and color for each sampled sphere.
- The tiny MLP takes the Tri-Mip feature vector f and view direction d as inputs and outputs density τ and color c for the sampled sphere S.
- The first two MLP layers estimate density and a 15-dimensional geometric feature, while the final layers combine that feature with spherical-harmonics direction encoding for view-dependent color.
B.2. Optimization
Tri-MipRF optimizes both its tiny MLP weights and Tri-Mip encoding mipmaps with AdamW, using distinct initialization and learning-rate settings for each parameter group.
- B.2. Optimization: Tri-MipRF optimizes the tiny MLP weights Θ and Tri-Mip encoding mipmaps M.Both parameter sets are trained jointly with AdamW.
- B.2. Optimization: Θ is initialized using the method from, while M is initialized uniformly in [−0.01, 0.01] to encourage sparsity.
- B.2. Optimization: M uses a learning rate scaled 10× above the base rate used for Θ.The larger rate reflects that M is a direct representation.
C. Detailed Results
Detailed Blender evaluations show Tri-MipRF consistently improves rendering quality across multi-scale and single-scale settings, with qualitative comparisons covering full and 1/8 resolution outputs.
- Multi-scale Blender Dataset: Tri-MipRF outperforms other methods on almost all multi-scale Blender scenes across the reported quantitative metrics.Table 4 reports arithmetic means averaged over the dataset’s four scales.
- Multi-scale Blender Dataset: Tri-MipRF consistently renders finer-grained and more anti-aliased images than Instant-ngp and Instant-ngp ↑5×.The comparisons include full-resolution and 1/8-resolution renderings, with depth maps shown for the full-resolution results.
- Single-scale Blender Dataset: Tri-MipRF outperforms point-sampling-based methods on all three metrics even on the single-scale Blender dataset.This setting observes scenes at roughly constant distance, reducing the scale issue for those methods.
- Single-scale Blender Dataset: The single-scale results demonstrate applicability to reconstructing objects at constant or varying observing distances.