Source-linked AI summary
Mixed Neural Voxels for Fast Multi-view Video Synthesis
Feng Wang, Sinan Tan, Xinghang Li, Zeyue Tian, Yafei Song, Huaping Liu
TL;DR
Real-world multi-view video synthesis remains difficult because complex environments and highly dynamic motion make existing methods slow and computationally demanding. MixVoxels uses separate static and dynamic voxel processing with variation-based separation and efficient inner-product time queries. It achieves competitive rendering quality after 15 minutes of training and reports a 5000× speedup over implicit dynamic-scene representations, though complex lighting can produce inconsistent predictions at static–dynamic boundaries.
Problem
Real-world multi-view dynamic-scene reconstruction remains difficult because existing methods require substantial computation and highly dynamic motions are hard to track.
Method
MixVoxels separates static and dynamic voxels with a variation field, processes them through different branches, and queries multiple time steps using inner products.
Results
5000× training speedup is reported with competitive or better rendering quality, while competitive results are achieved with only 15 minutes of training.
Takeaways & Limitations
MixVoxels makes training and rendering of complex dynamic scenes more practical by reducing computation and focusing the dynamic branch on dynamic regions.
Takeaways & Limitations
Complex lighting conditions can cause inconsistent property predictions at boundaries between dynamic and static voxels.
Abstract
from arXiv · showhide
Synthesizing high-fidelity videos from real-world multi-view input is challenging because of the complexities of real-world environments and highly dynamic motions. Previous works based on neural radiance fields have demonstrated high-quality reconstructions of dynamic scenes. However, training such models on real-world scenes is time-consuming, usually taking days or weeks. In this paper, we present a novel method named MixVoxels to better represent the dynamic scenes with fast training speed and competitive rendering qualities. The proposed MixVoxels represents the 4D dynamic scenes as a mixture of static and dynamic voxels and processes them with different networks. In this way, the computation of the required modalities for static voxels can be processed by a lightweight model, which essentially reduces the amount of computation, especially for many daily dynamic scenes dominated by the static background. To separate the two kinds of voxels, we propose a novel variation field to estimate the temporal variance of each voxel. For the dynamic voxels, we design an inner-product time query method to efficiently query multiple time steps, which is essential to recover the high-dynamic motions. As a result, with 15 minutes of training for dynamic scenes with inputs of 300-frame videos, MixVoxels achieves better PSNR than previous methods. Codes and trained models are available at https://github.com/fengres/mixvoxels
1. Introduction
MixVoxels targets slow, difficult multi-view dynamic-scene reconstruction with a mixed static–dynamic voxel representation and efficient time querying. It separates voxel types and processes them differently to reduce computation while preserving rendering quality.
- Multi-view dynamic-scene reconstruction supports applications including free-viewpoint control, cinematic effects, sports replays, and VR/AR.
- Existing real-world methods still require substantial time and computation, while highly dynamic scenes with complex motions remain difficult to track.
- MixVoxels represents 4D scenes with separate static and dynamic voxels processed by different models to reduce redundant computation.
- The method uses an inner-product time query to query multiple time steps efficiently and recover sharp details for highly dynamic objects.
- 5000× training speedup is reported for MixVoxels while maintaining competitive or better rendering quality than implicit dynamic-scene representations.
2. Related Works
Related work spans surface, volume, and implicit representations for static and dynamic novel-view synthesis, with growing emphasis on accelerating neural rendering. Multi-view input offers denser supervision for complex real-world dynamic scenes than monocular video alone.
- Static-scene novel-view synthesis uses mesh-based, voxel-grid, multiplane-image, and neural-radiance-field representations.
- Dynamic-scene methods extend NeRFs with implicit or explicit deformation fields, but monocular videos remain limiting for complex general scenes.
- Synchronized multi-view videos provide dense supervision across viewpoints and time for reconstructing real-world dynamic scenes.
- Dynamic-scene acceleration methods model frame differences, decompose static and changing components, or design efficient rendering strategies.
- NeRF acceleration methods reduce computation through sparse voxel representations, pretabulated radiance, or smaller neural networks.
3. Method
MixVoxels represents dynamic scenes with separate static and dynamic voxel branches, using a variation field for voxel classification and an inner-product time query for efficient dynamic modeling.
- Mixed voxel representation: MixVoxels extends explicit voxel grids to dynamic scenes by representing 4D content as mixtures of static and dynamic voxels.Static and dynamic components are processed with separate branches to avoid applying a heavy time-varying model everywhere.
- Static voxel representation: The static branch stores voxel densities and color features, interpolates continuous positions, and uses a small MLP to map features and view direction to RGB values.Explicit voxel storage avoids the millions of neural-network queries required by conventional neural radiance fields.
- Dynamic voxel representation: The dynamic branch uses compact spatial voxel features, shared decompression MLPs, and learned time queries to produce time-dependent density and color.The temporally implicit representation reduces memory compared with directly storing high-dimensional features for every time step.
- Dynamic voxel representation: Inner-product time querying computes outputs by combining learned time latents with decompressed voxel features, enabling simultaneous queries over many time steps.Its cost is FLOPmlp + T·FLO Pinn, compared with more than T·FLOPmlp for concatenation queries, with FLOPmlp >> FLO Pinn.
- Variation field: Voxel-level separation reduces redundant dynamic processing because only a small fraction of voxels around object surfaces are dynamic, while the learned field tolerates some occlusions.The method uses the proposed variation field rather than transmittance weighting because the latter caused a large efficiency drop with similar performance; the estimated field typically trains in under 30 seconds.
- Variation field: The variation field estimates voxel temporal variance from binarized pixel-level temporal variance, using ray constraints to separate static and dynamic space.Static rays encourage all traversed voxels to be static, while dynamic rays require at least one dynamic voxel; multiple rays resolve the allocation implicitly.
4. Experiments
Experiments evaluate MixVoxels on challenging multi-view dynamic scenes using quantitative metrics, visual comparisons, training schedules, and ablations. The results show fast training and rendering with competitive quality, while longer training and mixed voxel processing improve motion and detail reconstruction.
- Experiment setting: The evaluation uses six Plenoptic Video scenes and two additional complex scenes, including rapid Rubik’s-cube rotations and vehicles crossing the frame.Experiments use 18 views for training and one for evaluation, measuring PSNR, DSSIM, and LPIPS on novel views.
- Quantitative results: MixVoxels reduces DyNeRF training from 1.3K GPU hours to 15 minutes while achieving 37.7 fps at 1K resolution.The comparison covers 300-frame scenes and reports training time, rendering speed, and model size alongside quality metrics.
- Quantitative results: With 15 minutes of training, MixVoxels reaches 31.03 PSNR, comparable to methods trained for hours; longer training further improves all metrics.Increasing training time also sharpens boundaries and fine details.
- Qualitative results: Visual comparisons show near photo-realistic novel-view rendering and improved reconstruction of moving objects and textual details.Representative patches inspect details and motion across state-of-the-art comparisons and multiple views.
- Ablation studies: Full-dynamic voxel processing is more time-consuming and recovers blurrier motions than mixed voxels under the same training iterations.The ablation attributes the blurred motion to static regions affecting dynamic-information capture.
- Ablation studies: Querying more time steps per iteration clarifies moving boundaries by providing denser supervision and broader temporal information.The time-query ablation examines concatenation, Fourier-head, and inner-product alternatives, while larger query counts improve moving-hand and flowing-coffee details.
- Limitations: Complex lighting can produce inconsistent density or color predictions at boundaries between dynamic and static voxels.The authors suspect under-sampling of dynamic regions and identify this as a limitation for future work.
5. Conclusion
MixVoxels reconstructs 4D dynamic scenes by separating static and dynamic components and processing them through different branches. Its variation-field separation and inner-product time query enable practical novel-view video synthesis with only 15 minutes of training.
- MixVoxels splits 3D space into static and dynamic components to reconstruct 4D scenes and synthesize novel-view videos.
- The variation field separates the components, while different branches process them for more efficient reconstruction.
- An inner-product time query supports efficient dynamic voxel-grid representation and simultaneous querying of multiple time steps.
- 15 minutes of training makes training and rendering complex dynamic scenes more practical.