Source-linked AI summary
Space-time Neural Irradiance Fields for Free-Viewpoint Video
Wenqi Xian, Jia-Bin Huang, Johannes Kopf, Changil Kim
TL;DR
Learning a spatiotemporal irradiance field from one video is difficult because each time has one viewpoint and motion can be confused with appearance. The paper uses monocular depth to constrain geometry and temporal propagation to build a global representation, demonstrating compelling free-viewpoint rendering while remaining limited to observed times and available depth evidence.
Problem
Single-video free-viewpoint rendering must learn dynamic geometry from one viewpoint at each time despite motion–appearance ambiguity.
Method
The method learns a time-conditioned neural irradiance field, uses monocular video depth to constrain geometry, and propagates color and volume density across time.
Results
The method demonstrates compelling free-viewpoint video rendering on casual smartphone videos and evaluates its design choices visually and quantitatively on Sintel.
Takeaways & Limitations
A single casually captured video can support global free-viewpoint rendering of dynamic scenes with preserved motion and texture details.
Takeaways & Limitations
The method renders arbitrary viewpoints only at observed time steps and does not interpolate or extrapolate time.
Abstract
from arXiv · showhide
We present a method that learns a spatiotemporal neural irradiance field for dynamic scenes from a single video. Our learned representation enables free-viewpoint rendering of the input video. Our method builds upon recent advances in implicit representations. Learning a spatiotemporal irradiance field from a single video poses significant challenges because the video contains only one observation of the scene at any point in time. The 3D geometry of a scene can be legitimately represented in numerous ways since varying geometry (motion) can be explained with varying appearance and vice versa. We address this ambiguity by constraining the time-varying geometry of our dynamic scene representation using the scene depth estimated from video depth estimation methods, aggregating contents from individual frames into a single global representation. We provide an extensive quantitative evaluation and demonstrate compelling free-viewpoint rendering results.
1. Introduction
The paper learns a globally consistent spatiotemporal neural irradiance field from a casually captured single video, enabling rendering from novel viewpoints. It resolves motion–appearance ambiguity with monocular video depth and propagates content across time to handle disocclusions.
- Problem: Single-video free-viewpoint rendering targets casually captured videos without multi-camera rigs or dedicated hardware.The learned representation is globally consistent and dynamic, and can later be rendered from novel viewpoints.
- Problem: Existing dynamic-scene view synthesis commonly relies on multiple cameras because conventional reconstruction assumes static scenes.Per-frame depth maps or meshes are typical geometric representations.
- Problem: Monocular video depth enables per-frame depth estimates, but naïve depth warping creates unnatural stretches and holes in disoccluded regions.These artifacts can occur even with perfect depth estimates.
- Approach: Depth supervision constrains time-varying geometry, while temporal propagation of color and volume density fills otherwise unsupervised disoccluded contents.The resulting field can be queried at arbitrary spatiotemporal locations and rendered with standard volume rendering.
- Approach: The method aggregates frame-wise 2.5D representations into one globally consistent spatiotemporal representation using neural implicit functions.Neural implicit representations provide continuous scenes without resolution loss.
- Results: The paper demonstrates compelling free-viewpoint rendering on casual smartphone videos while preserving motion and texture details.The results convey a vivid sense of 3D across various challenging scenes.
2. Related Work
Related work addresses image and video view synthesis, neural implicit representations, video depth estimation, completion, and depth-map fusion. The paper differs by learning a time-conditioned field from monocular video and using dynamic scene depth rather than flow as primary regularization.
- Scope: The method renders arbitrary viewpoints only at observed time steps and does not interpolate or extrapolate time.This defines its temporal scope relative to video view-synthesis methods.
- View synthesis for images: Image-based rendering creates novel views by warping and blending contents from multiple source images, with multi-plane images serving as a popular representation.Recent methods reduce the required inputs to stereo pairs or a single image.
- View synthesis for videos: Video view synthesis must handle time-varying geometry and appearance, so existing methods often require multi-camera capture, special hardware, or synchronous viewpoints.Some methods reduce input views only within specific domains such as performance capture or video re-animation.
- Neural implicit representation: Neural implicit representations model scene properties continuously with multilayer perceptrons and support differentiable training without direct ground-truth 3D supervision.They address limitations of discrete voxel-grid and mesh representations.
- Video depth estimation: Monocular video depth estimation provides depth from a single video, but dynamic scenes and untextured regions remain challenging for existing methods.The paper leverages estimated depth to resolve ambiguity when learning spatiotemporal fields.
- Video completion: Video completion propagates known contents along flow trajectories, whereas this method completes disoccluded novel views within the 3D representation.Its approach operates beyond screen-space inpainting by modeling scene contents across space and time.
- Depth map fusion: The method differs from prior depth-map fusion by avoiding a fixed canonical 3D model and modeling both time-varying geometry and appearance.It uses a dynamic representation rather than restricting fusion to static scenes or geometry alone.
- Concurrent works: Among concurrent monocular-video NeRF extensions, this work directly conditions the dynamic field on time and regularizes training primarily with dynamic scene depth.Other cited methods use canonical deformation or flow-based regularization.
3. Background
NeRF represents radiance and volume density as a continuous function of 3D location and viewing direction. Differentiable volume rendering integrates these quantities along camera rays, allowing optimization from posed images using photometric loss.
- NeRF representation: NeRF models RGB radiance and differential volume density at each 3D location and viewing direction with a multilayer perceptron.Its function maps (x,d) to (c,σ).
- Volume rendering: A pixel color is rendered by integrating radiance modulated by volume density along its camera ray.The ray is parameterized from the camera center through the pixel center.
- Training: The model is trained by minimizing photometric loss between rendered ray colors and corresponding ground-truth image colors.The loss is evaluated over a set of rays.
- Implementation: Continuous volume rendering is approximated by differentiable numerical quadrature over sampled 3D points along each ray.Uniform sampling is followed by transmittance-based sampling that concentrates points near surfaces, using coarse and fine networks.
4. Space-time Neural Irradiance Fields
The method learns a space-time neural irradiance field from RGB-D video, combining color and depth supervision with empty-space and static-scene constraints for novel-view rendering.
- The field maps each spatiotemporal location to emitted color and volume density, using RGB-D frames and calibrated cameras as input.
- Color reconstruction matches rendered frames to the input video, but geometry remains severely under-constrained because each time has only one observed viewpoint.Different geometries can explain the same video through appearance changes, including a planar “flat TV” solution.
- Per-frame estimated scene depth constrains time-varying geometry and resolves the ambiguity between motion and appearance.The method estimates scene depth from learned volume density and compares it with the input depth.
- Empty-space loss penalizes nonzero density before the visible surface, complementing depth reconstruction to constrain geometry around observed surfaces.The loss stops at a small margin ε = 0.05·(sf −sn) before the estimated scene depth.
- Static-scene loss constrains hidden regions by encouraging shared color and density across times at the same spatial locations away from visible surfaces.It propagates partially observed contents across time without explicitly correlating surfaces, reducing vulnerability to misalignment.
- Static constraints are sampled from the union of camera-ray locations across all frames, excluding points close to observed surfaces and randomly jittering selected samples.This sampling strategy targets coverage and efficiency when camera motion is large.
5. Experimental Results
The experiments evaluate novel-view rendering on casually captured dynamic videos and synthetic stereo videos, including baseline comparisons and loss ablations. Results show that depth supervision is consistently beneficial, while static and empty-space losses help depending on scene type.
- Baselines and datasets: The evaluation compares the method with textured meshes, inpainted meshes, and NeRF augmented with a temporal parameter.Qualitative comparisons use the CVD dataset, while quantitative evaluation uses Sintel stereo videos with left-view training and right-view rendering.
- Qualitative comparisons: The CVD experiments present novel-view results from dynamic scenes captured by smartphones using the full model.The comparisons are shown in Figure 5 and include rendered mesh, inpainted mesh, temporal NeRF, and the proposed method.
- Ablation studies: The ablation study measures view synthesis with PSNR on all pixels, PSNR on disoccluded pixels, and SSIM on all pixels.The study varies combinations of the proposed losses and also tests a scene-flow loss derived from optical flow and per-frame depth.
- Ablation studies: Depth loss improves every tested model, while static and empty-space losses improve results depending on scene type.The evaluation aggregates metrics over all frames of Sintel videos rendered from the right viewpoints.
- Ablation studies: The scene-flow loss does not improve quality, whereas Model-2 is slightly better quantitatively than Model-4, the full model.The authors report that the full model usually performs best on real data despite Model-2’s slight quantitative advantage on Sintel.
6. Conclusions
The paper presents an algorithm for learning space-time irradiance fields from single casually captured videos. It validates depth-based geometric constraints and temporal content propagation visually and quantitatively, including challenging handheld cellphone scenes.
- 6. Conclusions: The method learns space-time irradiance fields from single casually captured videos.The paper showcases free-viewpoint rendering of dynamic scenes captured with handheld cellphone cameras.
- 6. Conclusions: Its core contributions are monocular video depth supervision for time-varying geometry and a static scene loss with cross-time sampling.The paper validates these design choices visually and quantitatively on the Sintel dataset.
A. Additional Details
Additional results report that video inpainting alone is insufficient for recovering good disocclusion contents. The appendix also expands qualitative testing and comparisons across ten videos.
- Additional quantitative comparison: Video inpainting is insufficient for good disocclusion contents and produces more artifacts than the proposed approach.The comparison is measured using PSNR and concerns the disoccluded regions produced by novel-view rendering.
- Additional results: The appendix includes implementation details, training details, additional quantitative comparisons, and qualitative results on 10 videos.These results include comparisons with baselines and different loss configurations.
A.1. Implementation Details
The implementation uses empirically selected loss weights and fixed ray and sample-pool batch sizes. Time and spatial inputs are normalized and positionally encoded for training.
- Implementation details: The experiments set α = 1, β = 100, and γ = 10 for the hyper-parameters.These values are used empirically in all experiments.
- Implementation details: Each training step uses 1024 randomly drawn rays and 1024 randomly selected locations from the static-loss sample pool.The time variable is normalized to [−1,1], with four positional-encoding frequency bands applied to time and spatial positions.
A.2. Training Details
The models use a specified MLP training setup, optimized with Adam for 800k iterations; training a 100-frame video takes about 48 hours.
- The model uses 1024 activations for each of the first 8 MLP layers, compared with 256 in the referenced architecture.The remaining architecture follows [48].
- Training uses Adam with β1 = 0.9, β2 = 0.999, and a learning rate of 0.0005.
- The MLP is trained for 800k iterations.
- Training a network on about 100 video frames at 960×540 resolution takes about 48 hours.
A.3. Additional Comparisons to Baseline
The paper quantitatively compares its method with an inpainted mesh baseline on disoccluded regions, reporting PSNR for those pixels only.
- The evaluation compares the proposed method with an inpainted mesh method for handling disoccluded areas.Both methods use the same Sintel ground-truth depth in this comparison.
- Table 2 reports PSNR on disoccluded pixels only.