Source-linked AI summary
MobileNeRF: Exploiting the Polygon Rasterization Pipeline for Efficient Neural Field Rendering on Mobile Architectures
Zhiqin Chen, Thomas Funkhouser, Peter Hedman, Andrea Tagliasacchi
TL;DR
NeRF rendering is difficult to deploy interactively because ray marching and repeated neural evaluations are poorly matched to common hardware. MobileNeRF represents the field with textured polygons and uses z-buffered rasterization plus a lightweight fragment-shader MLP, achieving order-of-magnitude speedups over SNeRG at equivalent image quality. Its scope is bounded by binary opacity, fixed mesh and texture resolutions, and weaker handling of challenging surfaces.
Problem
Traditional NeRF rendering evaluates a large MLP at hundreds of samples along each ray, making interactive visualization too slow for commonly available hardware.
Method
MobileNeRF represents scenes with textured polygons storing binary opacities and features, then uses z-buffer rasterization and a lightweight fragment-shader MLP to produce colors.
Results
An order of magnitude faster than SNeRG while producing images of equivalent quality.
Takeaways & Limitations
The rasterization-based representation runs on a wide range of platforms, including mobile phones, using standard polygon rendering hardware.
Takeaways & Limitations
MobileNeRF uses binary opacities, fixed mesh and texture resolutions, and does not handle glossy surfaces as well as recent methods.
Abstract
from arXiv · showhide
Neural Radiance Fields (NeRFs) have demonstrated amazing ability to synthesize images of 3D scenes from novel views. However, they rely upon specialized volumetric rendering algorithms based on ray marching that are mismatched to the capabilities of widely deployed graphics hardware. This paper introduces a new NeRF representation based on textured polygons that can synthesize novel images efficiently with standard rendering pipelines. The NeRF is represented as a set of polygons with textures representing binary opacities and feature vectors. Traditional rendering of the polygons with a z-buffer yields an image with features at every pixel, which are interpreted by a small, view-dependent MLP running in a fragment shader to produce a final pixel color. This approach enables NeRFs to be rendered with the traditional polygon rasterization pipeline, which provides massive pixel-level parallelism, achieving interactive frame rates on a wide range of compute platforms, including mobile phones.
1. Introduction
NeRF produces high-quality novel views but its ray-marched volumetric rendering is poorly matched to common hardware and too slow for interactive visualization. MobileNeRF replaces this with textured polygons and standard rasterization, targeting interactive rendering across common devices.
- NeRF represents scenes with an MLP that estimates density and view-dependent radiance for volumetric novel-view rendering.
- Hundreds of MLP evaluations along each pixel ray make traditional NeRF rendering too slow for interactive visualization.
- MobileNeRF represents scenes as textured polygons whose atlases store binary opacity and feature vectors.
- Z-buffered rasterization produces per-pixel features, which a lightweight view-dependent MLP in a fragment shader converts into output colors.
- 10× faster than SNeRG with the same output quality, while using less memory and supporting browsers, mobile devices, and real-time scene manipulation.
2. Related work
Related work spans image-based, geometry-based, neural, voxel, and mesh-rendering approaches to real-time view synthesis. MobileNeRF differs by combining textured triangle meshes with deferred neural shading in a classical rasterization pipeline.
- Light fields, Lumigraphs, and related image-based representations support real-time rendering but limit camera freedom or incur substantial storage overhead.
- Explicit geometry methods provide more camera freedom, but convolutional post-processing can reduce 3D consistency and typically requires powerful desktop GPUs.
- NeRF provides a 3D-consistent continuous field with high-quality results, but multiple neural evaluations per pixel prevent real-time rendering.
- Acceleration strategies include ray-segment modeling, spatial subdivision, learned sampling, and feature decoding from dense or sparse voxel embeddings.
- MobileNeRF uses a triangle mesh textured with deep features, rasterizes it into a deferred buffer, and applies a neural shader to visible fragments.
3. Method
MobileNeRF optimizes a textured polygonal mesh and renders it through a two-stage deferred rasterization pipeline. Training progressively learns continuous features and opacity, accelerates ray intersections, binarizes opacity, and applies feature supersampling for efficient antialiased rendering.
- 3. Method: The representation uses a polygonal mesh whose texture maps store features and opacity, followed by a neural deferred renderer that converts rasterized features into colors.Rendering is split into mesh rasterization and fragment-shader color synthesis.
- 3.1. Continuous training: The mesh topology is fixed while vertex locations and three MLPs are optimized by minimizing mean squared error between predicted and ground-truth training colors.The MLPs represent opacity, features, and view-dependent color.
- 3.1. Continuous training: Ray quadrature is accelerated by selecting intersected voxels, pruning voxels unlikely to contain geometry, and explicitly intersecting remaining mesh faces.The regular-grid mesh computes ray intersections in O(P) complexity for a P × P × P grid.
- 3.1. Continuous training: The acceleration grid is supervised to upper-bound visibility and regularized for pointwise sparsity and spatial smoothness.A stop-gradient prevents this grid from negatively affecting image reconstruction quality.
- 3.2. Binarized training: Smooth opacity is converted to binary opacity because standard rasterization does not natively support semitransparent meshes without per-frame depth sorting.A straight-through estimator and co-training stabilize optimization of the discrete representation.
- 3.4. Anti-aliasing: Anti-aliasing averages supersampled features before the deferred shader, reducing shader evaluations to one query per output pixel.The implementation averages four subpixel features and directions, then applies the same strategy to discrete occupancies.
4. Experiments
MobileNeRF is evaluated across diverse scenes and devices against SNeRG, with experiments covering rendering speed, quality, resources, geometry, and ablations. The method achieves similar image quality to SNeRG, while its design choices affect quality, speed, memory, and surface fidelity.
- Comparisons: The evaluation covers 8 synthetic 360° scenes, 8 forward-facing scenes, and 5 unbounded outdoor scenes, comparing MobileNeRF with SNeRG across devices.Experiments also include extensive ablations of the method’s design choices.
- Comparisons: SNeRG cannot represent unbounded 360° scenes and does not run on phones or tablets because of representation and compatibility limitations.Rendering tests use browser-based camera motion through 360 frames at dataset-specific resolutions.
- Rendering quality: MobileNeRF has roughly the same image quality as SNeRG and better quality than NeRF under PSNR, SSIM, and LPIPS comparisons.When zoomed in, SNeRG tends to produce over-smoothed images, whereas MobileNeRF retains similar quality at appropriate camera distances.
- Polygon count: MobileNeRF retains only visible triangles, removing most vertices and triangles from the initial mesh.The polygon-count analysis reports average retained vertices and triangles together with their percentages of the initial mesh.
- Shading mesh: The extracted shading mesh often does not align with the actual object surface, despite achieving good rendering quality.The authors identify better regularization losses or training objectives as necessary for improved surface quality, while vertex optimization improves rendering quality.
- Ablation studies: Ablations show that removing super-sampling causes a significant performance drop, while larger texture size improves quality at the cost of quadrupled texture size.The small MLP and super-sampling step have the most significant impact among the evaluated speed and space-cost variants.
5. Conclusions
MobileNeRF uses classical rasterization for efficient surface-based neural-field rendering, achieving roughly an order-of-magnitude speedup over SNeRG at equivalent image quality. Its explicit mesh representation also provides direct editing control, though the paper only superficially explores these possibilities.
- MobileNeRF uses z-buffers and fragment shaders to render surface-based neural fields efficiently across diverse compute platforms.The architecture exploits the classical rasterization pipeline rather than specialized volumetric rendering.
- It achieves frame rates an order of magnitude faster than SNeRG while producing images of equivalent quality.
- MobileNeRF’s explicit mesh representation provides direct editing control without requiring complex architectural changes.
- The paper only superficially investigates the editing possibilities enabled by the explicit mesh representation.
A. More results
The method produces nearly identical rendered images in Python and web browsers. The few differing pixels occur at part boundaries.
- Nearly identical images are produced when the method is rendered in Python or web browsers.The Python renders are used for computing quantitative metrics.
- The project-page demo models are the same models used in the paper.
- The few substantially different pixels appear on the boundaries of scene parts.
B. [Post-submission] Shader code optimization
Shader-level optimizations substantially improve MobileNeRF’s mobile rendering speed, and optimized forward rendering can outperform deferred rendering on some devices. Forward rendering remains device-dependent because excessive overdraw can make it slower.
- Optimized shader execution reaches 35 FPS on five real unbounded scenes using a Samsung Galaxy S22 Ultra, versus 26 FPS originally.This is a 35% speedup from injecting weights into shader source and using mat4 and vec3 operations.
- The default deferred renderer rasterizes features first and runs the MLP once per output pixel during final color rendering.
- 84 FPS is achieved with optimized forward rendering on five real unbounded scenes using a Samsung Galaxy S22 Ultra.This is 223% faster than the original deferred implementation at 26 FPS and 140% faster than optimized deferred rendering at 35 FPS.
- Forward rendering remains slower than deferred rendering on some devices because of excessive overdraw.A depth pre-pass may help, but may be unavailable on certain devices.
- Browser demos may differ slightly because the models were trained with deferred-rendering supersampling, whereas forward rendering supersamples pixel colors.
C. Scene editing
MobileNeRF’s textured-mesh representation supports compositing, manipulation, and real-time interaction with reconstructed objects. These operations leverage standard mesh tools and rendering techniques, though scene illumination is not fully accounted for.
- A textured mesh with baked lighting can be combined, rendered, or manipulated using mesh-based applications.
- Compositing meshes learned from four photo sets produces a 1920×1080 scene at 150 FPS while consuming 1.5 GB of GPU memory.The scene is rendered without supersampling on a gaming laptop.
- Triangle-mesh editing enables parts or objects to be edited or removed in 3D modeling software.
- Rendered edited scenes do not account for illumination differences between captured photos or indirect illumination between meshes.
- Learned NeRF objects can be manipulated interactively in real time, including moving objects, deforming chairs, and repelling leaves with a ball.The operations are implemented through mesh controls and vertex-shader changes.
- Figure 10 compares Python and browser images using pixel values in the 0-255 range.
D. Training
Training is organized into staged optimization problems that progressively fit the grid, polygon attributes, and shading networks. The stages use color, distortion, visibility, and grid regularization losses with scene-dependent weighting.
- The first stage optimizes the representation using color, distortion, visibility, and grid losses.The grid loss includes boundary, sparsity, and smoothness terms.
- The second stage optimizes the opacity, feature, and shader MLPs with stage-two color, distortion, visibility, and grid losses.Equation 19 remains part of the second-stage objective.
- The distortion-loss weight is 0.0 for synthetic 360° scenes, 0.01 for forward-facing scenes, and 0.001 for unbounded 360° scenes.
E. Network architectures
The architecture combines grid-based ray–mesh intersection with textured polygon storage and scene-specific mesh transformations. Regular-grid acceleration reduces intersection work, while mobile texture constraints shape feature and opacity packing.
- Two four-channel PNG images store 8-dimensional features, with binary opacity packed into the first feature channel.Large textures are split because phones require power-of-two dimensions no larger than 4096 × 4096.
- The regular-grid mesh computes ray intersections in O(P) complexity and uses an acceleration grid to prune unlikely geometry-containing voxels.The pruning threshold is τG = 0.1.
- Training progressively reduces retained intersections from 3P to 3P/2 and then 3P/4 while doubling batch size after each reduction.During the first quarter, all 3P intersected voxels are retained because the acceleration grid may be inaccurate.
- Normalized coordinates lie in [-0.5, 0.5], and scene-specific transformations place regular grids around objects or concentrate voxels near cameras.Synthetic 360° scenes use scaling, while forward-facing scenes use a camera-focused transformation.
- Unbounded 360° scenes add L + 1 fixed concentric boxes around the regular grid to model surrounding environments.The implementation uses P = 128, L = 64, and box distances spanning 0.5 to 8.
I. Per-Scene metrics
Per-scene evaluation reports image-quality metrics alongside rendering speed, GPU memory, and disk storage across synthetic 360°, forward-facing, and unbounded 360° scenes. Phone and Chromebook FPS measurements are capped at 60 by vertical synchronization.
- Per-scene breakdowns mark OOM when a device cannot run a scene because of GPU memory and ICP when compatibility issues prevent execution.GPU memory and disk storage were tested on the Desktop.
- Phone and Chromebook FPS is capped at 60 because vertical synchronization could not be easily disabled.
- Rendering-speed tables report frames per second, GPU memory, and disk storage for all three scene categories.