Source-linked AI summary
DynIBaR: Neural Dynamic Image-Based Rendering
Zhengqi Li, Qianqian Wang, Forrester Cole, Richard Tucker, Noah Snavely
TL;DR
Monocular novel-view synthesis remains difficult for long dynamic videos with complex motion and uncontrolled cameras. DynIBaR uses motion-aware volumetric image-based rendering and improves benchmark quality while handling challenging in-the-wild videos.
Problem
Dynamic NeRF methods have limited performance on long videos with complex object motion and uncontrolled camera trajectories, restricting casual in-the-wild use.
Method
DynIBaR aggregates multi-view image features in scene motion-adjusted ray space, using trajectory fields, temporal photometric consistency, and Bayesian motion segmentation.
Results
Over 50% average LPIPS errors are reduced across entire scenes and dynamic-object regions on two dynamic-scene benchmarks, with improved rendering of in-the-wild videos.
Takeaways & Limitations
The framework enables photo-realistic novel views from long in-the-wild dynamic videos with complex scene motion and unconstrained camera trajectories.
Takeaways & Limitations
The method is limited to relatively small viewpoint changes and cannot handle small fast-moving objects because of incorrect initial depth and optical-flow estimates.
Abstract
from arXiv · showhide
We address the problem of synthesizing novel views from a monocular video depicting a complex dynamic scene. State-of-the-art methods based on temporally varying Neural Radiance Fields (aka dynamic NeRFs) have shown impressive results on this task. However, for long videos with complex object motions and uncontrolled camera trajectories, these methods can produce blurry or inaccurate renderings, hampering their use in real-world applications. Instead of encoding the entire dynamic scene within the weights of MLPs, we present a new approach that addresses these limitations by adopting a volumetric image-based rendering framework that synthesizes new viewpoints by aggregating features from nearby views in a scene-motion-aware manner. Our system retains the advantages of prior methods in its ability to model complex scenes and view-dependent effects, but also enables synthesizing photo-realistic novel views from long videos featuring complex scene dynamics with unconstrained camera trajectories. We demonstrate significant improvements over state-of-the-art methods on dynamic scene datasets, and also apply our approach to in-the-wild videos with challenging camera and object motion, where prior methods fail to produce high-quality renderings. Our project webpage is at dynibar.github.io.
1. Introduction
Dynamic NeRF methods struggle with long, unconstrained videos and complex object motion. DynIBaR addresses this by using motion-adjusted feature aggregation with additional motion modeling and temporal regularization.
- Dynamic NeRFs encode spatiotemporally varying scene content with coordinate-based MLPs for monocular dynamic-scene view synthesis.
- Prior methods struggle with long videos, unconstrained camera paths, and complex object motion, limiting casual in-the-wild applications.NSFF targets short forward-facing videos, while HyperNeRF is mainly object-centric and can fail under complex motion.
- DynIBaR aggregates multi-view image features in scene motion-adjusted ray space to reason about spatiotemporally varying geometry and appearance.
- Motion trajectory fields, a temporal photometric loss, and Bayesian IBR-based motion segmentation improve efficiency, temporal coherence, and novel-view quality.
- Over 50% average LPIPS-error reduction is achieved across entire scenes and dynamic-object regions on two dynamic-scene benchmarks.
2. Related Work
Prior novel-view methods combine image-based rendering, geometry, or neural representations, but dynamic scenes remain difficult under monocular input. DynIBaR extends volumetric IBR to dynamic scenes and targets long videos with complex camera and object motion.
- Classic image-based rendering synthesizes novel views by integrating input-image pixels, with methods differing in their dependence on explicit geometry.
- NeRF-based methods encode continuous radiance fields in MLPs, while IBRNet combines image-based rendering with volume rendering without per-scene optimization.
- DynIBaR extends volumetric IBR from static scenes to dynamic scenes, focusing on higher-quality views for long videos with complex camera and object motion.
- Depth- or mesh-based dynamic-scene representations struggle to model complex geometry and view-dependent effects.
- Some monocular dynamic-scene methods work well only for short, 1–2 second, forward-facing videos.
3. Dynamic Image-Based Rendering
DynIBaR performs motion-aware volumetric image-based rendering by aggregating features from nearby views, modeling trajectories across time, and enforcing temporal consistency for dynamic novel-view synthesis.
- The system targets monocular dynamic-video view synthesis, where the goal is rendering a novel viewpoint at any desired time.
- Rather than encoding color and density directly in MLP weights, it uses volumetric image-based rendering to model complex geometry and view-dependent effects.
- 3.1. Motion-adjusted feature aggregation: Motion-adjusted feature aggregation warps sampled 3D points into neighboring source views, extracts features, and aggregates them along each target ray.
- 3.1. Motion-adjusted feature aggregation: Learned motion trajectory fields represent scene motion across frames, enabling correspondence queries through a single MLP query and efficient multi-view aggregation.
- 3.2. Cross-time rendering for temporal consistency: Cross-time rendering warps rays and sampled points to nearby times, aggregates features there, predicts colors and densities, and compares the rendered color with the target pixel.
- 3.3. Combining static and dynamic models: The framework also uses a motion-segmentation module for static–dynamic factorization and regularization combining data, motion-trajectory, and cycle-consistency terms.
4. Implementation details
The evaluation uses benchmark and in-the-wild monocular videos, with COLMAP poses, selected nearby source views, and additional mechanisms for global reasoning, degeneracy handling, and time interpolation.
- Data: The benchmark evaluation uses Nvidia and UCSD dynamic-scene datasets, each containing eight forward-facing scenes and 100–250-frame monocular videos.Held-out images at each time instance are used for evaluation; in-the-wild videos provide more challenging camera motion.
- Source-view selection: Source views for the time-invariant model are selected from a temporal candidate pool and ranked by camera baseline, with Nvs = 16 views retained.Candidates are restricted to a temporal interval around the target frame before baseline ranking.
- Global spatial reasoning: A global spatial coordinate embedding is appended to the ray-transformer input to improve density reasoning at non-surface or occluded points.The embedding complements the time embedding and local image-feature aggregation.
- Robustness: For in-the-wild videos, randomly sampled virtual viewpoints are synthesized from estimated depths and used as additional source images to address motion and camera degeneracies.This technique targets cases with mostly colinear camera and object motion or motion that is too fast to track.
- Time interpolation: The method supports fractional-time rendering by predicting colors and densities at nearby input times, then motion-based splatting and linear blending them.Scene flow derived from motion trajectories determines the splatting and blending.
- Setup: Each scene is reconstructed in Euclidean space with COLMAP poses and 128 coarse-to-fine samples per ray; training a 10-second video takes around two days on eight A100s.Rendering a 768 × 432 frame takes roughly 20 seconds.
5. Evaluation
The evaluation compares DynIBaR with canonical- and scene-flow-based methods using standard image-quality metrics on benchmark datasets, qualitative comparisons, and ablations. DynIBaR improves benchmark scores and preserves details in dynamic and in-the-wild scenes where prior methods often blur or miss content.
- Evaluation protocol: DynIBaR is compared with Nerfies, HyperNeRF, NSFF, and DVS using the same depth, optical-flow, and motion-segmentation inputs.Evaluation reports PSNR, SSIM, and LPIPS over the full scene and moving regions.
- Quantitative evaluation: 2 dB and 4 dB PSNR gains over the second-best methods are reported on the two benchmark datasets for the entire scene.The method also significantly improves all reported error metrics.
- Quantitative evaluation: Over 50% LPIPS error reductions are reported on average across entire scenes and dynamic-object regions.LPIPS is used as an indicator of perceptual similarity with real images.
- Ablation study: The ablation study compares the full system against variants removing temporal consistency, motion aggregation choices, static modeling, and masked reconstruction loss.It is conducted on the Nvidia Dynamic Scene Dataset.
- Qualitative evaluation: On the benchmark datasets, prior dynamic-NeRF methods blur moving-object details such as balloon textures, faces, and clothing, while DynIBaR renders content closest to ground truth.The comparison covers both static and dynamic scene content.
- In-the-wild evaluation: In in-the-wild videos, DynIBaR preserves details such as shirt wrinkles and dog fur, whereas prior dynamic-NeRF methods fail to recover high-quality static and moving content.Explicit depth warping instead produces holes near disocclusions and out-of-field-of-view regions.
6. Discussion and conclusion
The method achieves photo-realistic space-time view synthesis for complex dynamic scenes and improves over prior methods, but remains subject to viewpoint, motion, and source-view limitations.
- The method is limited to relatively small viewpoint changes compared with methods designed for static or quasi-static scenes.
- Small, fast-moving objects may be poorly modeled because initial depth and optical-flow estimates can be incorrect.
- Rendering can fail for dynamic content visible only at distant times, and static content may be unrealistic or blank when too few source-view features are aggregated.
- The approach synthesizes photo-realistic novel views from in-the-wild dynamic videos and improves over prior state-of-the-art methods on dynamic-scene benchmarks.