Source-linked AI summary
MERF: Memory-Efficient Radiance Fields for Real-time View Synthesis in Unbounded Scenes
Christian Reiser, Richard Szeliski, Dor Verbin, Pratul P. Srinivasan, Ben Mildenhall, Andreas Geiger, Jonathan T. Barron, Peter Hedman
TL;DR
Large-scale unbounded view synthesis needs representations that avoid both expensive per-sample computation and excessive memory use. MERF combines sparse volumetric and high-resolution planar features, uses a piecewise-projective contraction and lossless baking, and achieves browser-based real-time rendering while preserving radiance-field quality. Its main scope limits are semi-transparent view-dependent appearance, complex reflections, and devices with insufficient GPU power.
Problem
Existing radiance-field representations are either too compute-intensive for real-time rendering or too memory-intensive to scale to large scenes.
Method
MERF combines a sparse feature grid with high-resolution 2D feature planes, contracts unbounded coordinates with piecewise-projective transformations, and losslessly bakes the optimized representation for rendering.
Results
31.6% (MSE) higher quality in outdoor scenes than MobileNeRF, while achieving faster and more accurate real-time rendering of large, complicated scenes than prior real-time NeRF-like models.
Takeaways & Limitations
MERF enables high-quality real-time rendering of large-scale scenes in a browser and preserves the optimized radiance field through baking.
Takeaways & Limitations
MERF cannot faithfully model view-dependent appearance for rays intersecting semi-transparent objects, and its volume rendering requires sufficiently powerful GPUs.
Abstract
from arXiv · showhide
Neural radiance fields enable state-of-the-art photorealistic view synthesis. However, existing radiance field representations are either too compute-intensive for real-time rendering or require too much memory to scale to large scenes. We present a Memory-Efficient Radiance Field (MERF) representation that achieves real-time rendering of large-scale scenes in a browser. MERF reduces the memory consumption of prior sparse volumetric radiance fields using a combination of a sparse feature grid and high-resolution 2D feature planes. To support large-scale unbounded scenes, we introduce a novel contraction function that maps scene coordinates into a bounded volume while still allowing for efficient ray-box intersection. We design a lossless procedure for baking the parameterization used during training into a model that achieves real-time rendering while still preserving the photorealistic view synthesis quality of a volumetric radiance field.
1 INTRODUCTION
MERF addresses the difficulty of rendering large, unbounded scenes in real time by combining a memory-efficient representation with separate optimization and rendering parameterizations. Its baking procedure preserves the optimized radiance field while enabling browser-based rendering.
- Motivation: Large-scale real-time rendering remains difficult because neural queries are compute-intensive, while volumetric representations consume substantial memory and require many ray samples.These constraints prevent methods designed for limited-extent scenes from scaling effectively to larger unbounded scenes.
- Motivation: Radiance-field methods trade image quality and optimization suitability against compactness and rendering cost when moving from volumetric toward surface-like representations.The paper identifies both volume-versus-surface and memory-bound-versus-compute-bound trade-offs.
- MERF: MERF combines a voxel grid and triplane structure, using an NGP hash grid during optimization and explicitly baking it into a binary-occupancy-grid representation afterward.The two parameterizations represent the same underlying radiance-field function.
- MERF: The shared radiance-field function lets the quality obtained during optimization carry over to real-time browser rendering.The representation is designed for both differentiable optimization and fast rendering.
2 RELATED WORK
Prior work accelerates view synthesis through image interpolation, geometry proxies, grids, baking, ray-segment processing, sampling networks, image upsampling, and constrained representations. These approaches often remain limited by memory, high-resolution speed, input requirements, or reduced quality in large unbounded scenes.
- Scope: MERF targets real-time view synthesis in large unbounded scenes, complementing broader surveys by focusing on rendering acceleration and large-space reconstruction.Its use of triplanes builds on prior plane-based scene representations while retaining a volumetric formulation.
- Large-scale view synthesis: Early systems used many captured images or geometry proxies, while NeRF improved reconstruction quality through a volumetric MLP suited to thin structures and semi-transparent objects.Later approaches extended NeRF to large scenes using crowdsourced data, tiled networks, or warped domains.
- Accelerated representations: Grid-based radiance fields improve reconstruction speed, but baked volumetric methods and voxelized MLP systems can consume substantial graphics memory and remain limited in scene scale.Some prior compressed methods still evaluate an MLP for every volume sample, preventing real-time rendering of large scenes.
- Alternative acceleration strategies: Ray-segment methods, sampling networks, and image-upsampling pipelines accelerate rendering but have not consistently achieved high-resolution real-time rates or may require additional scene inputs.The cited upsampling approach required a 3D scan, whereas MERF uses posed images and runs in a browser on commodity hardware.
- Alternative acceleration strategies: Plane- or polygon-constrained representations render extremely quickly for objects or limited camera motion but introduce quality loss for large unbounded scenes.This motivates a representation designed specifically for large-scale unbounded view synthesis.
3 PRELIMINARIES
Radiance fields map 3D positions and viewing directions to density and color, with ray colors computed by sampling and compositing along rays. MERF uses deferred shading and a hybrid grid representation to reduce per-sample neural computation while retaining view-dependent effects.
- Radiance fields: A radiance field maps each 3D position and viewing direction to volumetric density and emitted RGB color.Ray color is computed by sampling points along the ray and compositing their densities and colors using numerical quadrature.
- Deferred shading: NeRF parameterizes this mapping with an MLP, whereas SNeRG decomposes it into density, diffuse color, and feature fields to reduce MLP evaluations to one per ray.The deferred-shading decomposition retains view-dependent information in feature vectors.
- MERF representation: MERF represents scenes with volumetric densities, diffuse RGB colors, and feature vectors rendered using SNeRG’s deferred-shading model.Figure 2 shows the corresponding hybrid representation and feature processing.
- MERF representation: In the MERF representation, each ray location queries a low-resolution 3D grid and three high-resolution axis-aligned 2D planes, whose interpolated features are combined and mapped to density, diffuse color, and view-dependent features.The 3D grid uses trilinear interpolation, while the 2D planes use bilinear interpolation.
- Prior techniques: SNeRG bakes a trained large MLP into a block-sparse grid for real-time rendering, while mip-NeRF 360 contracts unbounded scenes into a finite sphere.These approaches provide context for MERF’s deferred rendering and bounded-scene handling.
4 SCENE REPRESENTATION
MERF uses a unified radiance-field representation combining a low-resolution voxel grid with high-resolution 2D planes, and introduces a piecewise-projective contraction for efficient rendering of unbounded scenes.
- 4 SCENE REPRESENTATION: MERF represents unbounded scenes with volume densities, diffuse RGB colors, and feature vectors rendered using deferred shading.The representation is designed for real-time volumetric rendering while maintaining a low memory footprint.
- 4 SCENE REPRESENTATION: The field is parameterized by a low-resolution 3D voxel grid and three high-resolution 2D grids aligned with the cardinal planes.Each grid stores feature vectors, and the continuous field sums trilinear and bilinear interpolations.
- 4 SCENE REPRESENTATION: The continuous feature vector is split into density, diffuse color, and view-dependence components before nonlinearities produce the rendered quantities.The sigmoid constrains colors and features to (0, 1), and nonlinearities are applied after interpolation and summation.
- 4.2 Piecewise-projective Contraction: The spherical contraction maps straight rays to curves, making empty-space skipping and ray-box intersections more difficult for discretized rendering.It maps far-away space into a bounded region while allocating capacity toward high-resolution content near the camera.
- 4.2 Piecewise-projective Contraction: MERF’s piecewise-projective contraction preserves straight lines within seven regions, making contracted rays piecewise-linear and standard ray-AABB tests applicable.The unit cube is preserved, while the other six regions use projective transformations; the contraction performs on par with the original spherical contraction in Table 2(c).
5 TRAINING AND BAKING
MERF trains and bakes a unified radiance field while explicitly accounting for discretization, quantization, and occupied-space selection so the rendered representation matches the optimized field.
- To avoid a training-rendering mismatch, MERF simulates finite grid resolution during training with virtual grid corners and bilinear or trilinear interpolation.
- MERF restricts baking to regions considered occupied by the proposal MLP, using empty-space skipping to concentrate representation queries around surfaces.
- The method quantizes grid values during optimization by applying a sigmoid, byte quantization, and affine remapping to [-m,m], rather than quantizing only afterward.It uses m=14 for densities and m=7 for diffuse colors and features; a stop-gradient enables straight-through optimization of rounding.
- Baking evaluates MLP outputs on discrete grids and constructs a binary occupancy mask from training-ray samples whose weight and opacity exceed 0.005.The mask excludes occluded, unsampled, or low-opacity content; occupied voxels are marked around qualifying samples.
6 REAL-TIME RENDERING
MERF’s browser renderer combines occupancy hierarchies, selective memory access, and optimized shader execution to accelerate volume rendering while reducing runtime memory use.
- The real-time viewer is a JavaScript three.js web application whose rendering is orchestrated by a single GLSL fragment shader.
- Multi-resolution occupancy grids are formed by max-pooling the full-resolution mask A with filter sizes 16, 32, and 128.For a base resolution of 4096, the resulting grids have sizes 256, 128, and 32 and occupy 18 MB of video memory.
- Table 1 reports rendering quality on all Mip-NeRF 360 scenes and highlights models that render in real time; Mobile-NeRF lacks indoor metrics.
- The renderer queries occupancy grids coarse-to-fine and skips empty space using efficient ray-AABB intersections, accessing MERF data only for potentially occupied samples.
- MERF separates density from appearance data and reads appearance only when the returned density implies nonzero opacity.An optimized deferred-rendering MLP provides an additional 4× speed-up.
7 EXPERIMENTS
MERF is evaluated on unbounded indoor and outdoor scenes for quality, memory, and rendering speed, showing strong quality-memory-speed trade-offs alongside limitations from its shallow decoder and volume-rendering requirements.
- Evaluation setup: MERF is evaluated against offline and real-time view-synthesis methods on rendering quality, video memory consumption, and real-time rendering performance.
- Evaluation setup: The mip-NeRF 360 benchmark contains five outdoor and four indoor unbounded scenes requiring a high-resolution 2048^3 representation for faithful reproduction.
- Quality comparison: MERF outperforms DeepBlending and Mobile-NeRF on all three quality metrics and exceeds several offline methods on outdoor scenes, but scores slightly lower than them indoors.The paper attributes the indoor gap partly to stronger view-dependent effects that its shallow decoder MLP cannot model well.
- MERF vs. SNeRG++: At R=2048 and L=512, MERF achieves similar quality to SNeRG++ while requiring a fraction of the memory; removing the 3D grid significantly reduces quality.
- Real-time rendering: MERF renders faster than SNeRG++ while consuming only one fifth of its memory, whereas MobileNeRF is faster but uses twice the memory and loses 1.24 dB PSNR.These browser benchmarks use an M1 MacBook Pro at 1280 × 720; the Instant-NGP comparison uses an RTX 3090 at 1920 × 1080.
- Limitations: MERF inherits a limitation in which view-dependent color is evaluated once per ray, preventing faithful modeling for rays intersecting semi-transparent objects.
- Limitations: Because it still performs volume rendering, MERF is limited to devices with sufficiently powerful GPUs, while phones and headsets require further reductions in memory and runtime.
8 CONCLUSION
MERF combines hybrid volumetric parameterization, a line-preserving contraction function, and lossless baking to enable real-time browser rendering of large-scale scenes. It reports higher quality than prior real-time models while using less memory.
- MERF enables real-time rendering of large-scale scenes in a browser using a compressed volume representation.
- 31.6% (MSE) higher quality than MobileNeRF is achieved on outdoor scenes while requiring less than half the GPU memory.
- Table 2 evaluates baking fidelity, quantization-aware training, and the proposed contraction function on outdoor scenes.
- Table 3 reports performance comparisons on the outdoor scenes.
gardenvase
The visual comparison contrasts MERF with SNeRG++ models at multiple resolutions and shows the associated rendering memory usage. Only the 512-resolution SNeRG++ model has comparable size to MERF.
- Only SNeRG++ (512) has comparable size to MERF, while SNeRG++ (1024) and SNeRG++ (2048) are significantly larger.Total VRAM usage is listed beneath each method name.
- The comparison includes MERF and SNeRG++ models at varying resolutions, with total rendering VRAM reported for each method.
A TRAINING DETAILS
The training setup uses a fixed 25,000-iteration optimization with pixel batches sampled across training images and Adam optimization with a scheduled learning rate.
- Training runs for 25000 iterations with a batch size of 216 pixels sampled from all training images.
- Adam optimization uses learning-rate warmup from 1e−4 to 1e−2 during the first 100 iterations, followed by decay to 1e−3.
B ARCHITECTURE
The appendix describes architecture and benchmarking details for MERF and related view-synthesis methods, including browser rendering comparisons and a 3D-grid ablation.
- Architecture: The architecture parameterizes 3D voxel grids and 2D planes with an MLP using a multi-resolution hash encoding.
- Architecture: The hash-encoded MLP outputs an 8-dimensional vector representing density, diffuse RGB, and view-dependency features.
- Rendering comparisons: Mobile-NeRF is the only other method identified as real-time at 30fps or better; Instant NGP runs around 5fps and mip-NeRF 360 takes around 30 seconds per frame.
- Ablation: Removing the low-resolution 3D voxel grid often causes parts of the scene to be poorly reconstructed, including missing background regions.
- Architecture: Proposal-MLPs use two layers with 64 hidden units and a 10-level hash encoding with maximum grid resolution 512 and hash table size 216.
- Benchmarking: Benchmarking fixes camera pose, rendering resolution, and camera intrinsics across viewers, then averages frame rate over 150 frames.