Source-linked AI summary
Robust Dynamic Radiance Fields
Yu-Lun Liu, Chen Gao, Andreas Meuleman, Hung-Yu Tseng, Ayush Saraf, Changil Kim, Yung-Yu Chuang, Johannes Kopf, Jia-Bin Huang
TL;DR
Dynamic radiance-field methods often depend on SfM poses that fail on challenging videos, motivating a more robust reconstruction approach. RoDynRF jointly optimizes camera poses, focal length, and static and dynamic radiance fields, with experiments showing robust reconstruction across challenging videos. The authors nonetheless report that monocular dynamic view synthesis without known poses remains challenging and includes failure cases.
Problem
Existing dynamic view-synthesis methods typically require precise SfM camera poses, but SfM is vulnerable to challenging videos containing low-light noise, motion blur, and dynamic objects.
Method
RoDynRF jointly optimizes camera poses, focal length, and static and dynamic radiance fields for dynamic radiance-field reconstruction from casual monocular videos.
Results
Quantitative and qualitative comparisons demonstrate robust dynamic radiance-field reconstruction across challenging videos and datasets where typical SfM systems fail.
Takeaways & Limitations
The approach reconstructs dynamic radiance fields from casually captured monocular videos without requiring camera poses as input.
Takeaways & Limitations
Robust dynamic view synthesis from monocular video without known camera poses remains challenging, with failure cases reported.
Abstract
from arXiv · showhide
Dynamic radiance field reconstruction methods aim to model the time-varying structure and appearance of a dynamic scene. Existing methods, however, assume that accurate camera poses can be reliably estimated by Structure from Motion (SfM) algorithms. These methods, thus, are unreliable as SfM algorithms often fail or produce erroneous poses on challenging videos with highly dynamic objects, poorly textured surfaces, and rotating camera motion. We address this robustness issue by jointly estimating the static and dynamic radiance fields along with the camera parameters (poses and focal length). We demonstrate the robustness of our approach via extensive quantitative and qualitative experiments. Our results show favorable performance over the state-of-the-art dynamic view synthesis methods.
1. Introduction
Dynamic view synthesis enables novel viewpoints of dynamic scenes but existing unstructured-video methods depend on camera poses that SfM often estimates unreliably. RoDynRF addresses this by jointly optimizing camera parameters and static and dynamic radiance fields, with experiments demonstrating robustness on challenging datasets.
- Motivation: Dynamic view synthesis creates photorealistic novel views of dynamic scenes from arbitrary camera angles and viewpoints.These systems support applications including video stabilization, virtual reality, and view interpolation.
- Limitations of Existing Methods: Existing dynamic view synthesis methods for unstructured videos typically require precise camera poses estimated by SfM systems such as COLMAP.This dependency limits applicability when camera poses are unavailable or inaccurate.
- Limitations of Existing Methods: SfM is vulnerable to low-light noise, motion blur, and dynamic objects, making pose-dependent dynamic view synthesis fragile on challenging videos.NeRF-based methods that jointly optimize camera poses and scene geometry address pose estimation only for strictly static scenes.
- RoDynRF: RoDynRF reconstructs dynamic radiance fields from casual videos without requiring accurate camera poses as input.It optimizes camera poses and two radiance fields that model static and dynamic scene elements.
- RoDynRF: RoDynRF combines coarse-to-fine optimization, epipolar geometry, deformation fields, time-dependent appearance models, and regularization losses.These components are designed to improve pose estimation and consistency during dynamic radiance-field reconstruction.
- Evaluation: Quantitative and qualitative evaluations show robust performance over state-of-the-art methods on challenging datasets where typical SfM systems fail to estimate camera poses.The evaluations include Sintel, Dynamic View Synthesis, iPhone, and DAVIS datasets.
2. Related Work
Prior work spans static view synthesis, dynamic radiance fields, and camera-pose estimation, but dynamic methods commonly depend on precise poses while pose-optimizing NeRF methods are limited to static scenes. RoDynRF combines dynamic scene modeling with simultaneous camera-pose optimization to address this gap.
- Static View Synthesis: Earlier static view-synthesis methods use local warps, proxy geometry, learned features, multiplane images, or voxel-based representations to generate novel views.Some approaches require densely captured images, while others enhance rendering with proxy scene geometry.
- Static View Synthesis: NeRF models scenes with continuous neural radiance fields, whereas explicit voxel representations reduce the days-long training time associated with NeRF.The cited voxel-based methods replace implicit MLP representations with explicit voxels.
- Static View Synthesis: Single-image view-synthesis methods can predict depth, learned features, multiplane images, or layered depth images, but they handle only static scenes.RoDynRF instead targets dynamic scenes from a single monocular video.
- Dynamic View Synthesis: Dynamic view-synthesis techniques address non-rigid scenes using human-body specialization, RGBD data, sparse geometry, stereoscopic transitions, or piecewise-rigid priors.The passage notes that many such systems struggle with complicated geometry or require multiview, time-synchronized capture.
- Dynamic View Synthesis: Recent dynamic NeRF methods produce space-time synthesis results but rely on precise camera poses, limiting them when COLMAP or other SfM systems fail.RoDynRF is presented as handling complex dynamic scenarios without known camera poses.
- Visual Odometry and Camera Pose Estimation: Visual odometry methods estimate camera poses through direct photometric consistency or feature-based techniques, while learning-based systems use self-supervised image-reconstruction losses.Estimating poses from casually captured videos remains challenging.
- Visual Odometry and Camera Pose Estimation: NeRF-based methods that jointly optimize neural scene representations and camera poses remain limited to static sequences.RoDynRF simultaneously optimizes camera poses while modeling dynamic objects.
3. Method
RoDynRF jointly estimates camera parameters with static and dynamic radiance fields, using architectural choices and auxiliary geometric constraints to improve reconstruction robustness.
- Auxiliary Losses: Three auxiliary losses regularize static and dynamic modeling: reprojection, disparity, and scale- and shift-invariant monocular depth losses.Reprojection uses optical-flow correspondences, disparity compares corresponding 3D points in depth, and monocular depth uses MiDaS predictions.
- Framework: RoDynRF jointly optimizes camera poses, shared focal length, and explicit static and dynamic neural voxel radiance fields.The static field reconstructs the static scene and estimates camera parameters, while the dynamic field models moving content.
- Motion Mask Generation: Motion masks combine instance segmentation with epipolar-distance thresholding to exclude dynamic regions from reconstruction and auxiliary-loss calculations.The epipolar component uses a fundamental matrix estimated from optical flow between consecutive frames.
- Camera Pose Estimation: The static reconstruction uses coarse-to-fine voxel resolution, smoothing the optimization landscape and reducing convergence to sub-optimal camera-pose solutions.Voxel resolution progressively increases during training.
- Camera Pose Estimation: Late viewing-direction conditioning prevents the color MLP from bypassing voxel features, supporting consistent geometry and camera-pose estimation under photometric supervision.The viewing direction is fused only in the last color-MLP layer.
- Dynamic Radiance Fields: A coordinate-deformation MLP maps time-varying 3D points from camera space into a canonical voxel space for dynamic radiance-field modeling.It predicts time-dependent 3D offsets that are added to the original coordinates.
4. Experimental Results
Experiments show that the method estimates reliable camera poses and reconstructs higher-quality dynamic scenes across challenging datasets, while ablations identify key design components. Remaining limitations include sensitivity to fast camera motion and changing focal lengths.
- Camera pose estimation: The method estimates camera poses more accurately than existing NeRF-based approaches on MPI Sintel, while global optimization slightly worsens consecutive-frame rotation scores than ParticleSfM.The evaluation also compares favorably with learning-based visual odometry methods.
- Dynamic view synthesis: Using estimated poses, the method reconstructs detailed content and faithful geometry on DAVIS, whereas other methods often produce inconsistent geometry and poor synthesis.Ground-truth motion masks were used to exclude dynamic regions from loss calculations for all NeRF-based comparisons.
- Dynamic view synthesis: On the NVIDIA dynamic view synthesis dataset, the method performs favorably against state-of-the-art methods and remains comparable to COLMAP-pose results without COLMAP poses.Evaluation reports PSNR and LPIPS under the DynamicNeRF protocol.
- Dynamic view synthesis: On the iPhone dataset, the method performs on par with existing methods using masked PSNR and SSIM.The evaluation follows the DyCheck protocol.
- Qualitative comparison: COLMAP fails on 44 of 50 DAVIS sequences, while the method’s jointly learned poses reduce artifacts and improve dynamic novel-view quality for its own and other methods.Other methods benefit from the estimated poses but still generate artifacts in dynamic regions.
- Ablation study: Ablations identify coarse-to-fine voxel upsampling as critical for pose estimation, while viewing-direction fusion, gradient stopping, deformation, and time-dependent MLPs improve rendering.The reported ablation metrics are PSNR, SSIM, and LPIPS on the Playground sequence.
- Failure cases: Fast camera motion can cause flow-estimation errors that produce incorrect poses and geometry, and the shared-intrinsic assumption limits handling of changing focal lengths.These cases are presented as failure cases despite the method’s overall robustness.
5. Conclusions
The paper presents robust dynamic radiance fields for space-time synthesis from casually captured monocular videos without requiring camera poses as input. Extensive quantitative and qualitative comparisons demonstrate accurate reconstruction across challenging videos.
- The method reconstructs dynamic radiance fields for casually captured monocular videos without requiring camera poses as input.