Source-linked AI summary
Efficient Neural Radiance Fields for Interactive Free-viewpoint Video
Haotong Lin, Sida Peng, Zhen Xu, Yunzhi Yan, Qing Shuai, Hujun Bao, Xiaowei Zhou
TL;DR
Interactive free-viewpoint video needs fast generation and rendering, while generalizable radiance-field methods remain slow because they sample many points in empty space. ENeRF predicts coarse geometry with a cascade cost volume, uses it for depth-guided surface sampling, and jointly learns the depth and radiance-field networks from RGB images. Across multiple benchmarks, it achieves competitive performance and runs at least 60 times faster than previous generalizable radiance-field methods.
Problem
Generalizable radiance-field methods can process dynamic videos frame by frame, but their rendering is too slow for interactive free-viewpoint applications.
Method
ENeRF predicts coarse scene geometry with a cascade cost volume and uses learned depth-guided sampling to render with few points near the scene surface.
Results
At least 60 times faster: ENeRF achieves this speedup over previous generalizable radiance-field methods while maintaining competitive results across multiple benchmarks.
Takeaways & Limitations
ENeRF supports interactive free-viewpoint videos and can synthesize novel views of human performers in real time.
Takeaways & Limitations
ENeRF focuses on solid surfaces and may degrade when target regions are invisible in the input views.
Abstract
from arXiv · showhide
This paper aims to tackle the challenge of efficiently producing interactive free-viewpoint videos. Some recent works equip neural radiance fields with image encoders, enabling them to generalize across scenes. When processing dynamic scenes, they can simply treat each video frame as an individual scene and perform novel view synthesis to generate free-viewpoint videos. However, their rendering process is slow and cannot support interactive applications. A major factor is that they sample lots of points in empty space when inferring radiance fields. We propose a novel scene representation, called ENeRF, for the fast creation of interactive free-viewpoint videos. Specifically, given multi-view images at one frame, we first build the cascade cost volume to predict the coarse geometry of the scene. The coarse geometry allows us to sample few points near the scene surface, thereby significantly improving the rendering speed. This process is fully differentiable, enabling us to jointly learn the depth prediction and radiance field networks from RGB images. Experiments on multiple benchmarks show that our approach exhibits competitive performance while being at least 60 times faster than previous generalizable radiance field methods.
1 INTRODUCTION
Interactive free-viewpoint video requires both rapid video generation and fast novel-view rendering, but existing approaches face lengthy optimization or slow rendering. ENeRF addresses these constraints with learned depth-guided sampling and achieves competitive quality with substantially higher speed.
- Free-viewpoint video applications require quickly generating videos from multi-view inputs and rendering novel views at interactive rates.These requirements support immersive telepresence and user experience.
- Cached radiance-field methods can render in real time but may struggle to recover complex scene motions and still require lengthy optimization on new data.
- Generalizable radiance-field methods avoid explicit temporal-motion recovery and adapt quickly to new scenes, but rendering remains slow because each pixel requires many neural-network forward passes.
- ENeRF constructs a cascade cost volume to predict a depth interval, samples only a few 3D points near the likely surface, and uses cost-volume features for radiance-field prediction.
- The depth-guided sampling strategy is jointly learned with the radiance field in a fully differentiable pipeline using only RGB images.
- 60 times faster: ENeRF runs at least this much faster than previous generalizable radiance-field methods while achieving competitive results across multiple benchmark datasets.
2 RELATED WORK
Related work spans classical and neural novel-view synthesis, dynamic-scene radiance fields, generalizable radiance fields, and multi-view stereo cost volumes. ENeRF builds on these directions by using cost-volume geometry for efficient rendering.
- Novel view synthesis: Light-field interpolation and image-based rendering provide established approaches for free-viewpoint and novel-view synthesis.
- Novel view synthesis: Neural representations, including NeRF, optimize continuous color and density fields from input images to produce photorealistic novel views.
- View synthesis of dynamic scenes: Dynamic-scene methods model time explicitly, aggregate temporal information, or use deformation fields to establish correspondences across motion.
- Proposed approach: ENeRF uses a cascade cost volume to produce coarse geometry and a 3D feature volume, then samples around surfaces to accelerate volume rendering while supporting generalizable radiance fields.
- View synthesis of dynamic scenes: Generalizable radiance fields can process each dynamic-scene frame independently, avoiding temporal-motion recovery, but their rendering speed is slow.
- Multi-view stereo methods: Multi-view stereo methods use cost volumes built from image features and 3D CNN regularization for depth estimation, though memory consumption can be large.
3 METHOD
ENeRF estimates coarse scene geometry from multi-view images with a cascade cost volume, then uses that geometry to guide neural radiance-field construction and sample points near surfaces for efficient rendering. The model is trained end-to-end from RGB images using view-synthesis losses.
- 3.1 Overview: ENeRF extracts multi-scale image features, using low- and mid-resolution features for cost volumes and full-resolution features for radiance-field reconstruction.A 2D UNet produces three feature stages; F_i,1 and F_i,2 construct cost volumes, while F_i,3 reconstructs neural radiance fields.
- 3.2 Coarse-to-fine depth prediction: The cascade cost volume first estimates low-resolution depth, then uses that estimate to build a fine-level high-resolution cost volume and 3D feature volume.The coarse-to-fine process predicts a high-resolution depth map for the target view and produces geometry-aware features.
- 3.2 Coarse-to-fine depth prediction: The cost volume warps input-view features into target-view depth planes using camera calibration and aggregates multi-view features by voxel-wise variance.The homography uses input and target camera intrinsics, rotations, and translations before feature aggregation.
- 3.3 Neural radiance fields construction: ENeRF combines pixel-aligned image features with voxel-aligned 3D features, then uses MLPs to predict point features, density, and view-dependent color.Color is estimated by predicting blending weights from point features, image features, and the difference between source and target ray directions.
- 3.3 Neural radiance fields construction: Depth-guided sampling uniformly samples N_k points within each predicted surface-depth interval before volume-rendering their predicted densities and colors.Restricting samples to the predicted interval avoids sampling broadly through empty space and accelerates rendering.
- 3.4 Training: Gradients pass through sampled 3D points into the depth probability distribution, allowing joint learning of depth prediction and neural radiance fields from RGB images.Training uses rendered-versus-ground-truth color error, with an additional perceptual loss enabled by low sampling and memory usage.
4 EXPERIMENTS
Experiments evaluate ENeRF on static and dynamic benchmarks against generalizable and per-scene baselines, emphasizing rendering speed, image quality, depth prediction, and component ablations.
- Experiments setup: ENeRF is evaluated on DTU, Real Forward-facing, NeRF Synthetic, DynamicCap, and ZJU-MoCap using established novel-view-synthesis benchmarks.Static datasets assess generalization and per-scene optimization; dynamic datasets use synchronized, calibrated multi-view videos.
- Performance on image synthesis: ENeRF achieves state-of-the-art performance with real-time rendering speed on dynamic scenes.Dynamic-scene sequences generally contain 600–1000 frames, and DNeRF requires separate models for divided subsequences to handle complex motion.
- Performance on image synthesis: ENeRF delivers competitive static-scene image quality while significantly improving rendering speed over baselines.Reported metrics include PSNR, SSIM, and LPIPS; PixelNeRF renders almost 10x more slowly because it uses a much wider MLP.
- Quality of reconstructed depth: ENeRF’s reconstructed depth outperforms baseline methods, while its cost-volume depth also produces reasonable results under RGB supervision.The paper distinguishes density-derived “Ours-NeRF” depth from fine-level cost-volume “Ours-MVS” depth and compares both with generalizable methods and MVSNet.
- Ablation studies and analysis: Depth-guided sampling preserves rendering quality when samples per ray decrease from 128 to 2, while cascade cost volumes improve speed without reducing performance.Adding ground-truth depth supervision does not improve rendering performance.
- Ablation studies and analysis: With 3 input views and 2 samples per ray, ENeRF renders 512×512 images at 25.21 FPS on an RTX 3090.Feature extraction, cost-volume processing, and radiance-field inference take 4.3 ms, 16.1 ms, and 19.2 ms, respectively.
5 CONCLUSION AND DISCUSSION
ENeRF uses explicit depth maps as coarse scene geometry to guide implicit radiance-field rendering. It supports interactive free-viewpoint videos with competitive quality, while remaining limited to solid surfaces and nearby-view coverage.
- ENeRF guides implicit radiance-field rendering with explicit depth maps as coarse scene geometry.
- The method achieves competitive rendering performance while being significantly faster than previous generalizable radiance-field methods.
- ENeRF focuses on solid surfaces and cannot handle scenes where multiple surfaces contribute to appearance, such as transparent scenes.
- Rendering quality may degrade when target regions under a novel view are invisible in the input views.
1 METHOD DETAILS
The method aggregates source-view features, predicts radiance-field quantities with MLPs, and addresses depth discontinuities that can cause artifacts at foreground-background boundaries.
- A weighted pooling operator aggregates source-view features so the model can accept any number of input views.
- The density MLP infers density from image and voxel features, while a second MLP predicts blending weights for source-view colors.
- Depth estimators tend to smooth predictions across foreground-background discontinuities, producing inaccurate edge depths and artifacts.
2 DETAILS OF THE EXPERIMENTAL SETUP
The experiments evaluate static and dynamic view synthesis under specified masking, sampling, resolution, and rendering-speed protocols, with additional implementation and compositing details.
- Dynamic-scene evaluation uses synchronized calibrated videos, simple backgrounds, high-quality masks, and half the views as inputs.
- Static-scene evaluation follows MVSNeRF settings, including foreground metrics on DTU and black-background handling for SSIM and LPIPS.
- Rendering FPS is measured as network-forwarding time on an RTX 3090, with bound-mask rendering increasing dynamic-scene speed from 30.57 to 40.21 FPS.
- DNeRF required multiple separately trained models because directly applying it to benchmark sequences with complex motions failed to converge.
- The implementation includes weighted feature pooling, MLP-based density and color prediction, and compositional foreground-background radiance fields.
- Occlusions can cause artifacts in rendered regions such as chair legs.
3 MORE EXPERIMENTAL RESULTS
Additional analyses examine occlusion artifacts, depth quality, sampling efficiency, rendering-head choices, input settings, and comparisons with other methods across static and dynamic benchmarks.
- ENeRF may produce artifacts in occluded regions under the generalization setting.
- Fine-tuning on the target sequence improves the method’s visual rendering performance on ZJU-MoCap and DynamicCap.
- The cost volume produces reasonable depth from RGB-only supervision, enabling few samples near surfaces for photorealistic synthesis.
- Reducing samples per ray from 128 to 2 nearly preserves rendering quality, while the cascade cost volume accelerates cost-volume construction without quality loss.
- On DTU, 3 input views and 2 samples per ray provide high-quality synthesis at interactive frame rates.
- The IBRNet-style rendering head achieves 27.88/0.96 PSNR/SSIM, compared with 26.77/0.95 for the alternative rendering head.
- Setting the depth-range hyperparameter λ to 1 outperforms progressively reducing λ from 3 to 1, with PSNR/SSIM of 27.61/0.96 versus 25.25/0.89.
- On Real Forward-facing, ENeRF reaches 25.71/8.42 PSNR/FPS@756x1008 versus NeuralRays at 25.35/0.03.
4 DISCUSSIONS
The discussions examine depth prediction, baseline behavior, cross-dataset generalization, and failure cases tied to multi-view depth estimation. They also include quantitative comparisons on DTU.
- Learning the depth prediction with the rendering loss: Depth prediction can produce good results even when volume rendering samples only one point.The authors suggest softargmax depth estimation and gradients to depth planes may explain this behavior.
- Visual ablation results: The visual ablation compares a version without depth guidance to MVSNeRF on DTU.The cited figure identifies “w/o depth-gui.” as similar to MVSNeRF.
- Baseline behavior: The baseline does not converge well and obtains bad rendering results when estimating the distance to the surface.This contrasts with the proposed model’s surface-distance estimation.
- Performance of PixelNeRF: PixelNeRF performs poorly on NeRF Synthetic and Real Forward-facing because its absolute XYZ coordinates do not generalize across differing coordinate systems.MVSNeRF is also reported as discussing this PixelNeRF behavior.
- Limitations: ENeRF may fail to render high-quality results in multi-surface, fuzzy, specular-highlight, and semitransparent regions because MVS generally estimates their depth incorrectly.A layered representation is suggested as future work.
5 PER-SCENE BREAKDOWN
Per-scene comparisons on NeRF Synthetic and Real Forward-facing are presented alongside the reported averaged results. These results show performance comparable to the baselines.
- Cross-scene comparison: The per-scene results are consistent with the paper’s averaged results and show comparable performance to baselines.The passage summarizes the per-scene comparisons across the referenced tables.
- NeRF Synthetic: Per-scene comparisons are reported for the NeRF Synthetic dataset.Table 8 presents these quantitative comparisons.
- Real Forward-facing: Per-scene comparisons are reported for the Real Forward-facing dataset.Table 9 presents these quantitative comparisons.