Source-linked AI summary
Streaming Radiance Fields for 3D Video Synthesis
Lingzhi Li, Zhen Shen, Zhongshu Wang, Li Shen, Ping Tan
TL;DR
Dynamic-scene radiance-field methods need faster, online-capable reconstruction than implicit NeRF formulations provide. StreamRF uses incremental explicit-grid modeling with narrow-band tuning, model-difference compression, and pilot-model guidance. It reports 15-second-per-frame training with competitive rendering quality, while reducing storage and retaining visual quality.
Problem
Implicit dynamic radiance fields are computationally expensive and generally require offline access to many frames, limiting efficient online video modeling.
Method
StreamRF incrementally adapts an explicit voxel-grid base model by training and storing per-frame differences, using narrow bands, compression, and pilot-model guidance.
Results
15 seconds per-frame training with competitive rendering quality is reported for dynamic scenes.
Takeaways & Limitations
StreamRF provides an on-the-fly framework for dynamic-scene reconstruction while reducing the storage burden of explicit grids.
Takeaways & Limitations
Rendering can lose high-frequency details and fail to reconstruct transparent or translucent objects, limitations associated with explicit representations.
Abstract
from arXiv · showhide
We present an explicit-grid based method for efficiently reconstructing streaming radiance fields for novel view synthesis of real world dynamic scenes. Instead of training a single model that combines all the frames, we formulate the dynamic modeling problem with an incremental learning paradigm in which per-frame model difference is trained to complement the adaption of a base model on the current frame. By exploiting the simple yet effective tuning strategy with narrow bands, the proposed method realizes a feasible framework for handling video sequences on-the-fly with high training efficiency. The storage overhead induced by using explicit grid representations can be significantly reduced through the use of model difference based compression. We also introduce an efficient strategy to further accelerate model optimization for each frame. Experiments on challenging video sequences demonstrate that our approach is capable of achieving a training speed of 15 seconds per-frame with competitive rendering quality, which attains $1000 \times$ speedup over the state-of-the-art implicit methods. Code is available at https://github.com/AlgoHunt/StreamRF.
1 Introduction
3D video synthesis seeks free-viewpoint photorealistic rendering of dynamic scenes, but complex real-world geometry and appearance challenge general high-performance methods. StreamRF addresses this with incremental explicit-grid modeling for on-the-fly video processing.
- 3D video synthesis targets free-viewpoint photorealistic rendering of dynamic scenes captured from multiple posed cameras.
- Complex geometry and appearance in real-world scenes make it difficult to develop a general high-performance modeling methodology.
- NeRF methods provide high-fidelity static-scene rendering but incur costly training and inference from extensive neural-network computation.
- StreamRF renders novel views at approximately 10 FPS on the Meet Room and N3DV datasets.
- StreamRF reconstructs streaming radiance fields for novel-view synthesis of 3D videos using an incremental learning framework.It trains a base voxel-grid model on the first frame and stores per-frame model differences for subsequent frames.
2 Problem Formulation
NeRF represents scenes implicitly and renders colors through volumetric integration, but its repeated neural-network evaluations make dynamic-video training slow and primarily offline. Explicit voxel grids reduce this computational burden by storing opacity and color information directly.
- NeRF maps 3D position and viewing direction to radiance color and opacity, then renders pixel colors through volumetric integration along camera rays.
- Repeated forward and backward MLP evaluations for hundreds of samples per ray cause slow training and rendering; JaxNeRF takes over 20 seconds per 800 × 800 image.
- Dynamic NeRF training can exceed 600 GPU-days, while N3DV still requires about 56 GPU-days for reliable rendering quality and implicit methods are poorly suited to online training.
- Explicit representations discretize scene space into sparse voxel grids, storing opacity and spherical-harmonic coefficients instead of relying on large MLPs.
- Voxel opacity and color are trilinearly interpolated from neighboring voxels, and zero-opacity voxels are pruned using an occupancy mask.
3 Streaming Radiance Fields for Dynamic Scenes
StreamRF models dynamic scenes incrementally with explicit sparse grids, adapting a base model through per-frame differences and narrow-band tuning. Difference-based compression substantially reduces storage while preserving rendering quality and enabling fast streaming reconstruction.
- Streaming Radiance Fields for Dynamic Scenes: StreamRF incrementally adapts a first-frame base grid by learning per-frame model differences, enabling online video modeling.The current grid is formed by adding each stored difference to the previous frame's grid.
- Narrow Band Finetune: Narrow-band tuning addresses motion-induced changes by restoring empty voxels near the active region and optimizing only voxels within that band.The band is extracted from the previous occupancy mask using dilation and erosion, then empty voxels inside it are restored for tuning.
- Narrow Band Finetune: ∼15s per-frame training delivers competitive quality and is about three orders of magnitude faster than an implicit method requiring 4.5 GPU-hours per frame.The method also achieves over 500× rendering speedup, with more than 60 s versus 121 ms reported for the comparison.
- Difference-based Compression: 178 times less storage reduces each frame from 1015 MB to 5.7 MB on average through difference-based compression.The method stores masks and changed representations rather than complete grid models.
4 Efficient Training with Pilot Model Guidance
Pilot model guidance accelerates full-scale grid optimization by first identifying likely changes on a downsampled model. The resulting guidance mask focuses updates on relevant voxels while preserving high-frequency detail.
- Pilot Model Guidance: Pilot model guidance downsamples the previous grid, trains a smaller model, and upsamples its merged change masks to guide full-scale tuning.At S = 1/2, the pilot model has roughly 1/8 of the full grid's capacity.
- Pilot Model Guidance: The guidance mask freezes voxels outside predicted changes while filling added and erased voxels back into the full-scale grid.This restricts optimization to regions indicated by the pilot model's induced difference.
- Pilot Model Guidance: Pilot-guided training facilitates a higher-fidelity model than direct full-scale training under the same training time.The strategy is intended to reduce unnecessary full-scale modifications and avoid losing high-frequency details during downsampling and upsampling.
5 Experiments
Experiments evaluate StreamRF on the Meet Room and N3DV datasets against representative methods, measuring rendering quality, efficiency, storage, and targeted ablations. The method achieves substantially faster per-frame optimization while preserving competitive quality and storage, with narrow-band tuning, difference compression, and pilot guidance supporting these results.
- Experimental Comparison: StreamRF achieves three orders of magnitude faster training than N3DV, whose per-frame training requires over 4 GPU hours, while also improving inference efficiency and reducing storage needs relative to explicit baselines.The comparison reports mean per-frame PSNR, training time, inference time, and storage cost.
- Experimental Comparison: StreamRF reaches about 15 seconds and 10 seconds per-frame optimization on the evaluated datasets while maintaining competitive rendering quality, inference time, and storage.It substantially outperforms the baseline methods in training efficiency.
- Narrow Band Finetune: Narrow-band tuning addresses motion by restricting optimization to relevant regions, whereas sparse-grid tuning misses moved voxels and dense-grid tuning struggles with convergence under limited iterations.The ablation compares sparse and dense alternatives with narrow-band finetuning on the Meet Room dataset.
- Diff Based Compression: Difference-based compression reduces storage from 1015 MB to an average of 5.7 MB per frame with negligible rendering-quality loss and stable incremental storage over time.The reported preservation of performance does not degenerate across time steps.
- Pilot Model Guidance: Pilot model guidance reduces artifacts in static backgrounds and dynamic foregrounds compared with training without guidance under matched per-frame time.The reported artifacts include vanishing hands, blurry elbows, sparkle noise, and fuzzy regions between people.
- Limitation and Discussion: Explicit-grid representations still produce artifacts in high-frequency details and transparent or translucent objects, motivating possible explicit-implicit hybrid extensions.The limitation is attributed as potentially inherited from explicit representation.
6 Related Work
Related work spans classical and neural approaches to static and dynamic novel view synthesis, including geometry-based representations, radiance fields, and object-specific priors. Existing dynamic radiance-field methods can provide strong visual results but may remain costly or specialized, motivating a general efficient alternative.
- Static Novel View Synthesis: Static novel view synthesis has progressed from light-field and geometry-based interpolation toward neural radiance fields and explicit representations designed to improve rendering efficiency.Representative directions include Lumigraph, light fields, multi-plane images, NeRF, and voxel-based methods.
- Dynamic Novel View Synthesis: Dynamic novel view synthesis includes explicit depth-based systems, streamable free-viewpoint video, camera-array displays, and neural models for monocular or multi-camera videos.Neural approaches commonly model time directly or separate canonical templates from deformation fields.
- Dynamic Novel View Synthesis: N3DV extends NeRF with a timecode and sampling and training accelerations, producing strong visual results but requiring over 5 GPU hours per frame.Its computational cost is described as impractical for real-world applications.
- Dynamic Novel View Synthesis: Some neural-rendering methods target specific objects such as human bodies using skeleton or SMPL priors, whereas the presented method targets general scenes without object priors.The comparison emphasizes support for scenes with complex deformation.
- Dynamic Novel View Synthesis: Explicit representations can leave visual artifacts in high-frequency details and transparent or translucent objects.These failure cases are shown as a limitation of the explicit representation used in the work.
7 Conclusion
The paper proposes a streaming radiance-field method that incrementally reconstructs and renders dynamic scenes using explicit grids. Narrow-band tuning supports on-the-fly video processing with fast training convergence.
- The method formalizes dynamic-scene modeling as incremental learning for reconstructing and rendering streaming video with explicit grid representations.
- Narrow-band tuning enables fast training convergence while processing video sequences on the fly without recording scenes in advance.
A Meet Room dataset
The Meet Room dataset uses a 13-camera Azure Kinect DK capture system, with one central test view and twelve training views. Table 2 presents model-size reduction across compression steps in megabytes.
- A Meet Room dataset: The Meet Room dataset uses 13 Azure Kinect DK cameras, assigning the center view for testing and the other 12 views for training.The cameras use identical exposure times and shutter synchronization through a daisy-chain audio-cable topology.
- A Meet Room dataset: Table 2 reports model-size reduction at each diff-based compression step, with all sizes measured in megabytes.
B Model Size Reduction
The comparison evaluates performance and per-frame training time against deformation-based methods, while the compression procedure stores masks and opacity or spherical-harmonic features for changed and remaining voxels.
- B Model Size Reduction: Table 3 compares rendering performance and per-frame training time with deformation-based methods.
- B Model Size Reduction: Diff-based compression represents erased, added, and remaining voxels using occupancy masks plus opacity and spherical-harmonic features.Occupancy masks between adjacent frames allow the implementation to store a single mask instead of separate erase and add masks.
C More Comparison
The method is compared with monocular-video techniques extended to multi-camera settings on N3DV sequences. It achieves better rendering quality and training efficiency, while the supplied figures show the capture system, training views, test view, and scene examples.
- C More Comparison: The method achieves better rendering quality and training efficiency than extended multi-camera versions of methods originally designed for monocular video.The comparison uses PSNR and per-frame GPU hours on N3DV video sequences.
- C More Comparison: Figure 6 depicts the multi-camera capture system used for the experiments.
- C More Comparison: Figures 7 and 8 provide example training and test views, respectively.
- C More Comparison: Figure 9 shows examples of different scenes captured by the system.