Source-linked AI summary
3DGStream: On-the-Fly Training of 3D Gaussians for Efficient Streaming of Photo-Realistic Free-Viewpoint Videos
Jiakai Sun, Han Jiao, Guangyuan Li, Zhanjie Zhang, Lei Zhao, Wei Xing
TL;DR
Dynamic-scene FVV methods commonly require complete-video offline training and lack real-time rendering, limiting streaming applications. 3DGStream uses 3D Gaussians with a Neural Transformation Cache and adaptive additions for online reconstruction. It reports roughly 10-second per-frame training and roughly 200 FPS rendering at megapixel resolution, with moderate storage and photo-realistic quality.
Problem
Existing neural FVV methods generally require complete video sequences for time-consuming offline training and do not achieve real-time rendering.
Method
3DGStream uses 3D Gaussians in a two-stage per-frame pipeline combining a Neural Transformation Cache for transformations with adaptive 3DG additions for emerging objects.
Results
∼10s per-frame training and ∼200FPS rendering are achieved at megapixel resolution with moderate requisite storage and photo-realistic image quality.
Takeaways & Limitations
3DGStream enables online reconstruction and real-time rendering of photo-realistic dynamic-scene FVVs from video streams.
Takeaways & Limitations
The method depends on the initial 3DG-S point cloud and limited iterations restrict modeling of drastic motion and complex emerging objects.
Abstract
from arXiv · showhide
Constructing photo-realistic Free-Viewpoint Videos (FVVs) of dynamic scenes from multi-view videos remains a challenging endeavor. Despite the remarkable advancements achieved by current neural rendering techniques, these methods generally require complete video sequences for offline training and are not capable of real-time rendering. To address these constraints, we introduce 3DGStream, a method designed for efficient FVV streaming of real-world dynamic scenes. Our method achieves fast on-the-fly per-frame reconstruction within 12 seconds and real-time rendering at 200 FPS. Specifically, we utilize 3D Gaussians (3DGs) to represent the scene. Instead of the naïve approach of directly optimizing 3DGs per-frame, we employ a compact Neural Transformation Cache (NTC) to model the translations and rotations of 3DGs, markedly reducing the training time and storage required for each FVV frame. Furthermore, we propose an adaptive 3DG addition strategy to handle emerging objects in dynamic scenes. Experiments demonstrate that 3DGStream achieves competitive performance in terms of rendering speed, image quality, training time, and model storage when compared with state-of-the-art methods.
1. Introduction
FVV construction for dynamic scenes remains difficult because existing neural methods generally require offline training on complete videos and lack real-time rendering. 3DGStream addresses these constraints with online 3DG-based reconstruction, achieving fast training and real-time rendering while preserving competitive quality and storage.
- Motivation: Existing NeRF-like FVV methods typically require complete video sequences for time-consuming offline training and generally fail to achieve real-time rendering.These limitations prevent streaming dynamic scenes and hinder practical applications.
- Approach: 3DGStream uses 3D Gaussians to construct dynamic-scene FVVs through a per-frame online pipeline initialized from the previous timestep.The method combines Neural Transformation Cache modeling with adaptive 3DG addition for emerging objects.
- Results: 3DGStream achieves competitive training time, image quality, and model storage, while outperforming StreamRF in the relevant aspects and providing real-time rendering.The comparison covers static per-frame methods, offline dynamic methods, and StreamRF.
- Approach: The method targets on-the-fly construction of photo-realistic, real-time-renderable FVVs from video streams, eliminating lengthy offline training on entire sequences.This directly addresses the streaming constraint identified for prior neural rendering methods.
2. Related Work
Prior dynamic FVV methods use specialized priors, motion correspondences, deformations, spatio-temporal representations, or per-frame training, each with important scope or efficiency limitations. 3DGStream extends the streamable per-frame paradigm with 3DGs while targeting efficient online reconstruction.
- Dynamic FVV methods: Dynamic FVV research includes prior-driven, flow-based, warp-based, spatio-temporal-input, and per-frame-training approaches.These categories reflect different strategies for modeling scene dynamics and novel views.
- Dynamic FVV methods: Prior-driven methods improve reconstruction for specific objects such as humans but are limited in application and generalizability to broader scenes.They use parametric models or additional priors such as skeletons.
- Dynamic FVV methods: Flow-based methods can produce impressive results from monocular videos but often require depth, optical flow, and motion-segmentation priors for intricate dynamic scenes.Their challenge stems from the ill-posedness of monocular reconstruction.
- Dynamic FVV methods: Warp-based methods assume dynamics arise from deformation of static structures, which can prevent them from handling topological variations.They map each frame’s radiance field onto canonical frames.
- Dynamic FVV methods: Spatio-temporal methods synthesize dynamic-scene views effectively, but entangled scene parameters can constrain downstream adaptability.These methods query radiance fields using spatio-temporal coordinates.
- Streamable methods: Among streamable methods, ReRF and Dynamic3DG require minute-level per-frame training, while StreamRF requires only a few seconds per frame.NeRFPlayer streams only pre-trained models.
- 3DG-based methods: 3DGStream differs from concurrent dynamic-3DG extensions by targeting efficient FVV streaming with online training rather than offline reconstruction.Deformable3DG, hexplane-based methods, and 4DG primitives are described as lacking streamable capabilities.
3. Background: 3D Gaussian Splatting
3D Gaussian Splatting represents scenes with anisotropic Gaussian primitives and differentiable point-based rendering, enabling fast high-fidelity novel-view synthesis. Its pipeline projects 3DGs into 2D splats and alpha-blends their view-dependent colors.
- 3D Gaussian representation: 3DG-S represents scenes with anisotropic 3D Gaussians paired with a fast differentiable rasterizer.This explicit representation supports real-time novel-view synthesis after only minutes of training.
- 3D Gaussian representation: A 3DG is defined by a covariance matrix centered at a mean, with its Gaussian value determined by the position relative to that distribution.The representation also includes view-dependent spherical-harmonics coefficients and opacity.
- 3D Gaussian representation: The covariance matrix is decomposed into rotation and scaling factors to preserve positive semi-definiteness during optimization.Rotation uses a unit quaternion, while scaling uses a 3D vector.
- Differentiable rasterization: For rendering, 3DG-S projects 3DGs into 2D Gaussian splats using camera and projection transformations.The projected covariance is reduced to a 2×2 image-space covariance, and the mean is projected into image space.
- Differentiable rasterization: Pixel colors are computed by alpha-blending ordered overlapping splats using their view-dependent colors and effective opacities.Each effective opacity combines a 3DG’s opacity with the corresponding 2DG evaluation.
- Performance: For megapixel-scale real-world scenes, a few minutes of optimization produce photo-realistic quality and rendering speeds exceeding 100 FPS.The speed comes from the optimized rasterization pipeline and custom CUDA kernels.
4. Method
3DGStream reconstructs dynamic-scene FVV streams on-the-fly by transforming prior-frame 3DGs and adding frame-specific Gaussians for emerging objects. Its compact NTC models Gaussian motion efficiently, while adaptive spawning addresses scene content absent from the initial frame.
- Pipeline: The method initializes 3DGs at timestep 0 and processes later frames through NTC-based transformation followed by adaptive 3DG addition.Previous-frame 3DGs provide initialization for each subsequent timestep.
- Neural Transformation Cache: The NTC uses multi-resolution hash encoding and a shallow fully-fused MLP to model compact, adaptive Gaussian transformations.The design supports local smoothness, combines global and local information, and focuses optimization on dynamic regions.
- Neural Transformation Cache: 3DG means and rotations are updated using predicted translation dµ and quaternion rotation dq, with spherical-harmonic coefficients adjusted after rotation.The transformed mean is µ′ = µ + dµ, while rotation is q′ = norm(q) × norm(dq).
- Training: The rendering loss combines L_1 and D-SSIM, while previous-frame 3DGs remain frozen during training.The loss uses λ = 0.2, and freezing keeps the NTC input consistent.
- Adaptive 3DG Addition: Adaptive addition targets emerging objects by spawning limited, frame-specific 3DGs near regions with large view-space positional gradients.The method selects Gaussians whose average gradient magnitude exceeds τgrad = 0.00015, avoiding persistent accumulation of added Gaussians.
- Evaluation: Across N3DV and Meet Room, 3DGStream provides fast online training and real-time rendering while retaining competitive storage and image quality.The method requires the minimal model storage among methods capable of streaming FVVs and captures dynamic and intricate objects qualitatively.
- Ablations: NTC achieves comparable results to direct optimization with substantially lower storage overhead, whereas removing hash encoding or warm-up causes slower convergence and accumulated errors.Direct optimization produces the best outcomes in the reported sequence evaluation but requires inflated storage.
- Ablations: The full adaptive-spawning model reconstructs emerging objects with minimal added 3DGs, unlike random spawning or the model without Stage 2.Examples include coffee in a pot, a dog’s tongue, and flames.
6. Discussion
3DGStream depends on the initial 3DG-S reconstruction and limits training iterations to enable on-the-fly processing, constraining performance in difficult scenes.
- Initial-frame quality is crucial because 3DGStream inherits 3DG-S dependence on the initial point cloud.Artifacts beyond windows arise from COLMAP’s inability to reconstruct distant landscapes.
- Limited training iterations restrict modeling of drastic motion in Stage 1 and complex emerging objects in Stage 2.
7. Conclusion
3DGStream enables online reconstruction of dynamic scenes in video streams using an NTC and adaptive 3DG addition. It delivers photo-realistic megapixel FVV rendering with approximately 10 seconds of per-frame training and 200 FPS rendering.
- 3DGStream combines an NTC for object-motion capture with adaptive 3DG addition for emerging objects.
- ~10s per-frame training and ~200FPS rendering enable online FVV reconstruction at megapixel resolution.The method maintains photo-realistic image quality with moderate storage requirements.
9. Implementation Details
The implementation uses 3DG-S for initial reconstruction, a hash-based NTC with a shallow MLP, and specified learning rates for training additional 3DGs.
- 3DGStream is implemented on 3DG-S with tiny-cuda-nn for the NTC and evaluated on an NVIDIA RTX 3090 GPU.Initial-frame densification ends at iteration 5000; initialization uses iteration 15000 for N3DV and 10000 for Meet Room.
- For N3DV, the NTC uses hash table size 2^15, feature dimension 4, and 16 resolution levels; Meet Room uses hash table length 2^14.The fully-fused MLP has two hidden layers with 64 neurons each and ReLU activation.
- Additional 3DG training uses learning rates of 0.0024, 0.0375, 0.75, 0.075, and 0.015 for mean, SH coefficients, opacity, scaling, and rotation quaternion.Their proportions follow 3DG-S defaults rather than individually fine-tuned values.
10. SH Rotation
SH rotation updates first-degree spherical-harmonic coefficients after transforming 3DGs, using projection relationships to derive the required rotation matrix.
- Only first-degree SH coefficients require rotation because zeroth-degree SH is rotation-invariant.
- The method derives an SH rotation matrix M from the equivalence between rotating a normal before projection and rotating its projected SH representation.N denotes a normal vector, and P is the projection function.
- Three normal vectors form A = [P(N0), P(N1), P(N2)], providing the projected basis used in the derivation.
- For computational convenience, the chosen basis normals are the three Cartesian unit vectors.
11. More Results
The paper reports quantitative image-quality, ablation, rendering, and storage evaluations, alongside free-view synthesis videos on N3DV scenes.
- Quantitative Results: PSNR is used to compare image quality across all N3DV scenes.The comparison is reported in Table 5.
- Storage: Table 8 details the storage components used by the method.The storage entry in Tables 1 and 2 is decomposed in Table 8.
- Storage: Figure 8 reports the number of additional 3DGs and corresponding storage required for each frame in the flame salmon scene.This evaluates frame-specific growth and its storage requirement.
- Quantitative Results: The ablation study evaluates 3DGStream across all N3DV scenes, while rendering profiling is conducted on the Meet Room dataset.These results are reported in Tables 6 and 7, respectively.
- Qualitative Results: Free-view synthesis videos demonstrate results on various scenes from the N3DV dataset.The videos are provided on the project webpage.
12. More Evaluations
Additional evaluations examine per-frame storage and the trade-off between training efficiency and reconstruction quality as training iterations change.
- Storage Evaluation: Only per-frame NTCs and additional 3DGs need storage beyond the initial 3DGs.This storage organization is detailed in Table 8.
- Storage Evaluation: Initial 3DGs are on the order of 10^5, whereas frame-specific additional 3DGs are on the order of 10^2 in N3DV experiments.Figure 8 tracks how the additional 3DG count changes with frame number.
- Training-Quality Trade-off: Increasing Stage 1 training iterations significantly improves reconstruction quality.Stage 2 results use 250 Stage 1 optimization iterations, and PSNR measures image quality.
- Training-Quality Trade-off: Limiting training iterations enables efficient on-the-fly training at the expense of reduced reconstruction quality.The evaluation explicitly examines this trade-off.