Source-linked AI summary
DeVRF: Fast Deformable Voxel Radiance Fields for Dynamic Scenes
Jia-Wei Liu, Yan-Pei Cao, Weijia Mao, Wenqiao Zhang, David Junhao Zhang, Jussi Keppo, Ying Shan, Xiaohu Qie, Mike Zheng Shou
TL;DR
Dynamic radiance-field methods achieve high-fidelity novel views but often require days to train, motivating faster learning for non-rigid scenes. DeVRF uses voxel-based canonical and deformation fields with a static-to-dynamic paradigm, achieving 100× faster training with comparable high-fidelity results. Its scope remains constrained by large model size and limited handling of drastic deformations.
Problem
NeRF-based dynamic radiance-field methods can require days to train because deformation and time-varying field computations are expensive.
Method
DeVRF models 3D canonical space and 4D deformation with voxel representations, then transfers a static volumetric prior to dynamic learning from few-view sequences.
Results
100× faster training with comparable high-fidelity novel view synthesis quality is reported across synthetic and real-world dynamic scenes.
Takeaways & Limitations
Static-to-dynamic learning enables efficient deformable radiance-field reconstruction using a low-cost capture setup and few fixed cameras.
Takeaways & Limitations
DeVRF has a large model size and may not model drastic deformations because the canonical-space prior is not synchronously optimized in the second stage.
Abstract
from arXiv · showhide
Modeling dynamic scenes is important for many applications such as virtual reality and telepresence. Despite achieving unprecedented fidelity for novel view synthesis in dynamic scenes, existing methods based on Neural Radiance Fields (NeRF) suffer from slow convergence (i.e., model training time measured in days). In this paper, we present DeVRF, a novel representation to accelerate learning dynamic radiance fields. The core of DeVRF is to model both the 3D canonical space and 4D deformation field of a dynamic, non-rigid scene with explicit and discrete voxel-based representations. However, it is quite challenging to train such a representation which has a large number of model parameters, often resulting in overfitting issues. To overcome this challenge, we devise a novel static-to-dynamic learning paradigm together with a new data capture setup that is convenient to deploy in practice. This paradigm unlocks efficient learning of deformable radiance fields via utilizing the 3D volumetric canonical space learnt from multi-view static images to ease the learning of 4D voxel deformation field with only few-view dynamic sequences. To further improve the efficiency of our DeVRF and its synthesized novel view's quality, we conduct thorough explorations and identify a set of strategies. We evaluate DeVRF on both synthetic and real-world dynamic scenes with different types of deformation. Experiments demonstrate that DeVRF achieves two orders of magnitude speedup (100x faster) with on-par high-fidelity results compared to the previous state-of-the-art approaches. The code and dataset will be released in https://github.com/showlab/DeVRF.
1 Introduction
DeVRF accelerates dynamic radiance-field learning by combining voxel-based canonical and deformation representations with static-to-dynamic training. It achieves comparable high-fidelity synthesis while reducing training time by 100×.
- 1 Introduction: NeRF-based dynamic-scene methods require days to train because deformation networks or time-varying texture and density queries add computation.
- 1 Introduction: DeVRF explicitly represents both the 3D canonical space and 4D deformation field with discrete voxel-based representations.The representation is designed for neural modeling of non-rigid dynamic scenes.
- 1 Introduction: The large parameter count makes direct dynamic training prone to local optima and overfitting, motivating the staged learning design.
- 1 Introduction: The static-to-dynamic paradigm transfers a volumetric canonical space learned from multi-view static images to deformation-field learning from few-view dynamic sequences.The resulting capture setup uses a few fixed cameras, which the authors describe as more convenient than moving-camera capture in practice.
- 1 Introduction: 100× faster training in about 10 minutes on an RTX3090 matches SOTA approaches in novel view synthesis quality across synthetic and real-world scenes.The evaluation covers five inward-facing synthetic scenes, two forward-facing real-world scenes, and one inward-facing real-world scene.
- 1 Introduction: DeVRF is presented as a first dynamic radiance-field approach incorporating a 4D voxel deformation field.
2 Related Work
Related work spans explicit and neural representations for static view synthesis, dynamic radiance fields, and NeRF acceleration. Existing dynamic-scene systems face capture or training-cost constraints that motivate more efficient approaches.
- 2 Related Work: Static-scene view synthesis progressed from explicit 3D reconstructions and multi-plane images to NeRF’s high-fidelity neural radiance fields.
- 2 Related Work: Dynamic-scene methods include voxel-plus-warp representations and NeRF extensions that map dynamic fields to canonical fields.
- 2 Related Work: Dynamic-scene capture is difficult because multi-view sequences may require dozens of cameras, while moving monocular cameras encounter deformation and occlusion challenges.
- 2 Related Work: Explicit volumetric representations such as DVGO and Plenoxels reduce static NeRF training to minutes using discretized or sparse voxel-based scene models.
3 Method
DeVRF combines a voxel-based 3D canonical space with a 4D deformation field, learned through a static-to-dynamic pipeline and optimized with several regularization strategies.
- 3.1 Capture Setup: DeVRF first learns a 3D volumetric canonical prior from multi-view static images, then optimizes a 4D deformation field using few-view dynamic sequences.The capture setup separates static-state acquisition with a moving monocular camera from motion capture with a few fixed cameras.
- 3.2 Deformable Voxel Radiance Fields: The 3D canonical space stores density and color in voxel grids, queried at arbitrary points through trilinear interpolation.Density uses softplus and post-activation, while a shallow MLP models view-dependent color effects.
- 3.2 Deformable Voxel Radiance Fields: The 4D voxel deformation field maps sampled points from a deformed frame to canonical space through interpolation across neighboring voxels and time steps.The resulting canonical points provide density and color for volume rendering and novel-view synthesis.
- 3.3 Optimization: Coarse-to-fine optimization progressively increases deformation-field resolution from 10 × 10 × 10 to 160 × 160 × 160 to reduce overfitting and suboptimal solutions.The strategy lets the field learn a rough motion before representing finer motion details.
- 3.3 Optimization: Training combines photometric rendering loss with deformation cycle consistency, optical-flow supervision, and total-variation regularization.These objectives constrain rendered colors, forward-backward motion agreement, image-plane motion, and smoothness between neighboring motion voxels.
4 Experiments
DeVRF is evaluated across synthetic and real-world deformable scenes against state-of-the-art baselines and ablations. It delivers high-quality results with substantially faster training, while static data and the proposed optimization components materially support performance.
- Comparisons with SOTA Approaches: DeVRF achieves the best PSNR and LPIPS, second- or third-best SSIM, and comparable high-fidelity views across synthetic and real-world scenes.The comparison covers averaged inward-facing results and forward-facing real-world scenes.
- Comparisons with SOTA Approaches: Less than 10 minutes of optimization on one RTX3090 GPU is about two orders of magnitude faster than other approaches, using 4.6GB to 6.5GB GPU memory.The reported timing applies to per-scene optimization.
- Comparisons with SOTA Approaches: Baselines trained with dynamic data alone perform significantly worse, while static multi-view data supplements few-view dynamic sequences that otherwise lack complete scene information.Neural Volumes also performs poorly with the few-view dynamic sequences because it requires dozens of dynamic sequences as input.
- Comparisons with SOTA Approaches: DeVRF uses four cameras for dynamic capture, with three views for training and one for testing in the forward-facing real-world scenes.This setup is used for the two collected forward-facing deformable scenes.
- Ablation Studies: Removing optical flow, cycle consistency, total variation, or coarse-to-fine training progressively lowers performance, with coarse-to-fine removal causing the largest drop.The ablation results are reported on both synthetic and real-world scenes.
5 Conclusion
The conclusion presents DeVRF as a voxel-based approach that learns a 3D canonical space and 4D deformation field through static-to-dynamic training and additional optimization strategies. It reports 100× faster training with comparable high-fidelity results, while identifying model size and limited handling of drastic deformations as limitations.
- 5 Conclusion: DeVRF models dynamic scenes with voxel-based 3D canonical and 4D deformation representations, optimized through static-to-dynamic learning and several regularization strategies.The strategies include coarse-to-fine learning, deformation cycle consistency, optical-flow supervision, and total-variation priors.
- 5 Conclusion: 100× faster training achieves comparable high-fidelity results relative to state-of-the-art approaches.This is the conclusion's reported overall outcome for DeVRF.
- 5 Conclusion: The model is large, does not synchronously optimize the 3D canonical prior in its second stage, and may therefore be unable to model drastic deformations.The paper identifies these limitations as future work directions.
Appendix
The appendix supplies implementation details, additional per-scene evaluations, further ablations, and supplementary visual material. It also points readers to a video comparing DeVRF with other state-of-the-art methods.
- Appendix: Section A provides additional implementation details for DeVRF.
- Appendix: Section B reports additional per-scene evaluation results, while Section C presents further ablations of DeVRF.
- Appendix: A supplementary video visualizes and compares DeVRF with state-of-the-art methods on synthetic and real-world deformable scenes.
A Implementation Details
The implementation uses a single RTX3090 GPU and explicit voxel resolutions for the canonical space and deformation field. The deformation field is optimized progressively from coarse to fine spatial resolutions.
- A Implementation Details: Experiments use PyTorch on a single NVIDIA GeForce RTX3090 GPU.
- A Implementation Details: The 3D canonical voxel grids use 160 × 160 × 160 resolution for inward-facing scenes and 256 × 256 × 128 for forward-facing scenes.The canonical representation includes density and color grids with shallow MLPs.
- A Implementation Details: The 4D deformation field is progressively upscaled from 10 × 10 × 10 to 160 × 160 × 160 spatial resolution during coarse-to-fine optimization.The full field is modeled at 50 × 3 × 160 × 160 × 160 resolution.
B.1 Per-scene Evaluation on Inward-facing Synthetic Deformable Scenes.
DeVRF is quantitatively evaluated against baselines and ablations on five inward-facing synthetic dynamic scenes using PSNR, SSIM, and LPIPS.
- PSNR, SSIM, and LPIPS measure reconstruction quality across five inward-facing synthetic dynamic scenes, including Lego, Floating robot, and Daisy.LPIPS with VGG is included as a perceptual metric, while PSNR and SSIM provide additional image-quality measures.
B.2 Per-scene Video Comparisons on Synthetic and Real-world Deformable Scenes.
DeVRF achieves on-par, high-fidelity novel-view synthesis across synthetic and real-world deformable scenes, with clean depth maps and smooth motion at unseen time steps. Its 4D voxel deformation interpolation supports novel-time rendering that some baselines cannot provide.
- Video comparisons: DeVRF achieves on-par high-fidelity results across all five synthetic and three real-world deformable scenes, with the cleanest depth maps among compared approaches.The supplementary video provides the qualitative comparison across these scenes.
- Quantitative comparisons: Tables 4 and 5 provide per-scene quantitative evaluations on inward-facing synthetic scenes against baselines and DeVRF ablations.The supplied table captions identify the evaluated scenes and comparison scope but do not provide cell values.
- Video comparisons: Quadruple interpolation of DeVRF’s 4D voxel deformation field synthesizes smooth motions at novel time steps, unlike intermittent motions observed for D-NeRF and NSFF in plant and rabbit.The clearest differences are reported in the plant and rabbit scenes, alongside improvements in floating robot, kuka, and flower-360°.
- Video comparisons: DeVRF can synthesize novel views at unseen time steps, whereas Nerfies and HyperNeRF are evaluated only at training and testing time steps because they cannot render unseen times.This difference determines how the compared video examples are generated.
C Additional Ablations
The additional ablations examine how dynamic-view count affects DeVRF and show that few dynamic views can be effective when combined with static multi-view data. The capture setup uses a few deployable cameras and reaches near-saturation with six dynamic views.
- Capture setup: The static-to-dynamic setup combines complete 3D geometry and appearance from multi-view static images with few-view dynamic sequences showing deformation over time.The capture process requires only a few cameras that are convenient to deploy in practice.
- Dynamic-view ablation: The dynamic-view ablation evaluates PSNR and LPIPS as the number of dynamic training views changes.Figure 4 reports these two quality metrics for the ablation.
- Dynamic-view ablation: Four dynamic training views yield results comparable to six views, while performance almost saturates at six dynamic views in the ablation.The paper therefore uses four dynamic views while retaining comparable performance to the six-view setting.