PaperlayerView exact revision on arXiv

Source-linked AI summary

Monocular Dynamic View Synthesis: A Reality Check

Hang Gao, Ruilong Li, Shubham Tulsiani, Bryan Russell, Angjoo Kanazawa

arXiv:2210.13445v1cs.CV

TL;DR

Monocular dynamic view synthesis is difficult to evaluate because existing protocols can leak effectively multi-view signals into training. The paper quantifies these signals with effective multi-view factors, proposes visibility-aware image and correspondence metrics, and finds substantial performance drops under stricter evaluation, especially for complex motion.

  • Problem

    Existing monocular DVS protocols can include effectively multi-view training signals, while unseen test regions make novel-view evaluation difficult.

  • Method

    The paper introduces effective multi-view factors, co-visibility masked image metrics, correspondence accuracy, and a diverse iPhone dataset for evaluating monocular DVS without camera teleportation.

  • Results

    1-2 dB drop in masked PSNR occurs without camera teleportation, while complex motion produces another 4-5 dB drop in masked PSNR for existing methods.

  • Takeaways & Limitations

    Existing monocular DVS methods have substantial room for improvement under protocols that remove multi-view cues and test complex deformation.

  • Takeaways & Limitations

    EMFs characterize only some difficulty factors, while variable appearance, lighting, and motion complexity remain for future characterization.

Abstract

from arXiv · show

We study the recent progress on dynamic view synthesis (DVS) from monocular video. Though existing approaches have demonstrated impressive results, we show a discrepancy between the practical capture process and the existing experimental protocols, which effectively leaks in multi-view signals during training. We define effective multi-view factors (EMFs) to quantify the amount of multi-view signal present in the input capture sequence based on the relative camera-scene motion. We introduce two new metrics: co-visibility masked image metrics and correspondence accuracy, which overcome the issue in existing protocols. We also propose a new iPhone dataset that includes more diverse real-life deformation sequences. Using our proposed experimental protocol, we show that the state-of-the-art approaches observe a 1-2 dB drop in masked PSNR in the absence of multi-view cues and 4-5 dB drop when modeling complex motion. Code and data can be found at https://hangg7.com/dycheck.

1 Introduction

The paper argues that existing monocular DVS protocols leak multi-view cues through impractical capture patterns, then proposes calibrated metrics and data for realistic evaluation. Under this protocol, state-of-the-art performance drops substantially without multi-view cues and on complex motion.

  • Existing DVS benchmarks use teleporting cameras or quasi-static scenes, creating multi-view cues unavailable in practical single-camera capture.These protocols either switch viewpoints between consecutive frames or fail to represent real-life dynamics.
  • EMFs quantify multi-view signal in monocular sequences from relative camera-scene motion and reveal that current protocols are effectively multi-view.The analysis characterizes camera teleportation as an impractical capture setup.
  • Co-visibility masks restrict PSNR, SSIM, and LPIPS to test pixels seen in the input, while PCK-T evaluates transferred keypoint correspondences.Together, these metrics evaluate rendering quality and inferred deformation without camera teleportation.
  • 1-2 dB masked PSNR and ~5% PCK-T drops occur on existing datasets without camera teleportation.The evaluation compares state-of-the-art methods under reduced multi-view cues.
  • 4-5 dB masked PSNR and ~30% PCK-T drops occur when existing approaches model complex motion in the proposed dataset.The new dataset contains more challenging motion and diverse scenes, exposing substantial room for improvement.

2 Related work

The paper situates monocular dynamic NeRFs among methods for synthesizing novel views of generic dynamic scenes and identifies limitations in their inherited evaluation protocol. Unseen-camera PSNR can include regions absent from training, while multi-camera training introduces effective multi-view conditions.

  • Non-rigid structure-from-motion methods recover dynamic 3D geometry using parametric models or non-parametric depth scans, evaluated with geometry or correspondence accuracy.These methods target accurate surface geometry at each time step.
  • Monocular dynamic NeRFs model time-varying scenes either directly in world space or through canonical-space NeRFs with time-dependent deformation.The paper analyzes methods that synthesize novel views from monocular video.
  • Inherited static-NeRF evaluation uses image metrics such as PSNR on held-out viewpoints, but unseen dynamic views may contain regions absent from training.This makes standard full-image evaluation potentially uninformative for dynamic scenes.
  • Existing approaches address unseen regions by using multiple cameras during training, producing an effectively multi-view setup.The paper introduces sequence-difficulty metrics, a monocular dataset, and a revised evaluation protocol in response.

3 Effective multi-view in a monocular video

Effective multi-view in monocular video lies on a continuous spectrum determined by relative camera–scene motion, rather than being simply single- or multi-camera. The paper quantifies this spectrum with two EMFs, balancing completeness against practical computability.

  • 3.1 Characterizing effective multi-view in a monocular video: Effective multi-view is characterized continuously by relative camera–scene motion, not as a binary property of camera count.This framing distinguishes nominally monocular capture from sequences containing cues similar to multi-camera capture.
  • 3.1 Characterizing effective multi-view in a monocular video: Monocular video can contain effective multi-view cues when camera motion is much faster than scene motion.Under these conditions, the same scene is observed from different viewpoints despite being captured by one camera.
  • 3.2 Quantifying effective multi-view in a monocular video: Full EMF Ω measures the relative magnitude of camera motion to scene motion, providing a theoretically complete but costly characterization.Its denominator represents 3D scene flow, while its numerator represents 3D camera motion over one forward time step.
  • 3.2 Quantifying effective multi-view in a monocular video: Angular EMF ω measures camera angular velocity around a scene look-at point, offering an easier approximation for object-centric captures.It assumes a single look-at point, which often holds for captures involving a single centered subject.
  • 3.2 Quantifying effective multi-view in a monocular video: Full EMF estimation may be noisy or costly because it depends on 3D scene-flow estimates, whereas angular EMF avoids direct scene-motion estimation.Scene flow can be estimated from dense optical flow and depth, or from monocular depth in the general case.
  • 3.2 Quantifying effective multi-view in a monocular video: Larger Ω and ω values indicate that a sequence contains more multi-view cues.The paper recommends reporting angular EMF for simplicity and full EMF when possible.

4 Towards better experimentation practice

Existing DVS datasets often contain effective multi-view cues from teleporting cameras or quasi-static scenes, making monocular evaluation less representative. The paper proposes co-visibility and correspondence metrics plus an iPhone dataset with single-camera training, large-baseline validation, diverse motion, and depth supervision.

  • 4.1 Closer look at existing datasets: Existing datasets operate under an effectively multi-view regime through teleporting camera motion or quasi-static scene motion.Across 23 multi-camera sequences used quantitatively, 22 have teleporting motion; HyperNeRF’s single-camera qualitative sequences have quasi-static motion.
  • 4.1 Closer look at existing datasets: Existing datasets have high EMFs, while the proposed iPhone dataset has smaller EMFs from a single camera and smooth motion.The HyperNeRF dataset reaches ω ≈ 200°/s, whereas the proposed dataset is designed around substantially lower effective multi-view cues.
  • 4.1 Closer look at existing datasets: Alternating frames from multiple physical cameras create teleporting training sequences in Nerfies and HyperNeRF, while NSFF samples from 24 cameras.This protocol supplies multi-view cues unavailable in practical single-camera capture.
  • 4.1 Closer look at existing datasets: Camera teleportation enables full-image evaluation but confounds model efficacy with multi-view signal and avoids unseen-pixel difficulties.Without teleportation, NeRF models struggle to hallucinate unseen regions, and rendering quality also degrades.
  • 4.2 Our proposed metrics: Co-visibility masking evaluates only test pixels observed during training, supporting masked PSNR, SSIM, and LPIPS without camera teleportation.The mask addresses unseen regions in multi-camera validation while retaining image-quality evaluation.
  • 4.2 Our proposed metrics: PCK-T evaluates 2D keypoint correspondences across training frames, directly measuring the quality of inferred scene deformation.This complements image metrics, which do not directly measure deformation accuracy.
  • 4.3 Proposed iPhone dataset: The iPhone dataset contains 14 non-repetitive sequences, single-camera hand-held training, two large-baseline validation cameras, and metric depth.It targets diverse complex motions involving generic objects, humans, and pets; depth supervision and other regularizations benefit DVS training.

5 Reality check: re-evaluating the state of the art

The evaluation disentangles monocular DVS performance from effective multi-view cues by testing canonical methods under non-teleporting captures and on a more challenging iPhone dataset. Results show substantial degradation and persistent difficulty with complex deformation.

  • Experimental setup: The study evaluates NSFF, Nerfies, HyperNeRF, and T-NeRF on existing and proposed datasets using co-visibility masked image and correspondence metrics.The existing Nerfies-HyperNeRF captures comprise 7 sequences, while the iPhone dataset includes multi-camera and single-camera captures.
  • Impact of effective multi-view: All methods generally perform better with camera teleportation than without it, showing that effective multi-view cues inflate prior synthesis quality.NSFF is the exception for PCK-T, plausibly because its additional optical-flow supervision is more accurate without teleportation.
  • Benchmark results without camera teleportation: Without camera teleportation, T-NeRF achieves the best mPSNR and mSSIM, whereas HyperNeRF produces the most photorealistic renderings by mLPIPS but with distorted artifacts.The incorrect shape in the CHICKEN sequence illustrates the mismatch between photorealism and geometric alignment.
  • Ablation study on improving the state of the art: Existing methods perform poorly out-of-the-box on the iPhone dataset, whose diverse, complex motions expose ghosting and visually implausible results.Additional random background compositing, depth supervision, and surface sparsity regularization consistently boost performance in the ablation.
  • Benchmarked results: On complex iPhone deformation, all models struggle to produce valid shapes, and synthesis quality and correspondence accuracy drop considerably relative to Nerfies-HyperNeRF.For Nerfies, the reported decreases are 4.4 dB in mPSNR, 69.6% in mLPIPS, and 40.1% in PCK-T.

6 Discussion and recommendation for future works

The paper formalizes effective multi-view measurement and proposes practical evaluation tools for monocular DVS. It recommends reporting capture difficulty and correspondence quality while identifying limits in the current EMF characterization.

  • Discussion and recommendations: The authors recommend visualizing input sequences, reporting EMFs, evaluating correspondence accuracy, and improving correspondence establishment in future DVS work.They leave characterization of variable appearance, lighting, motion complexity, and other difficulties for future research.
  • Effective multi-view factors: Full EMF Ω and angular EMF ω quantify effective multi-view using relative camera-scene motion and are designed to be scale-agnostic.Ω compares camera motion with scene motion, while ω measures camera angular speed.
  • Full EMF computation: The Full EMF computation estimates 3D scene flow from camera poses, dense optical flow, depth maps, and foreground masks, excluding occluded and background pixels.When metric depth is unavailable, monocular depth is used; unknown depth scale and shift can be aligned with sparse COLMAP points using RANSAC.
  • Effective multi-view factors: In practice, Ω depends on model estimates such as monocular depth, motivating ω as a metric that can avoid external model estimates under practical assumptions.The assumptions include roughly constant object speed, camera fixation on the object, and approximately constant camera-object distance.
  • Co-visibility mask: Co-visibility masking evaluates only test pixels observed in training images, using forward-backward optical-flow consistency and accumulated occlusion maps.RAFT supplies optical flow, and the resulting heatmap records how often each test pixel was observed in training views.

C.2 Masked image metrics

The section defines co-visibility-masked image metrics for evaluating rendered quality only on regions observed during training, then reviews correspondence extraction from dynamic NeRF models. It covers masked PSNR, SSIM, and LPIPS, along with warp-integrate-project correspondence computation and model-specific deformation handling.

  • C.2 Masked image metrics: mPSNR averages per-pixel PSNR scores over the co-visibility-masked region.It extends PSNR by restricting the average to pixels seen during training.
  • C.2 Masked image metrics: mSSIM uses partial convolution so only masked pixels contribute to the structural-similarity result.SSIM is computed at patch level, and the implementation accounts for masked pixels in the final score.
  • C.2 Masked image metrics: mLPIPS masks unseen image regions before computing downsampled masked feature-space distances.Because AlexNet has a receptive field of 1952, small co-visibility masks can make mLPIPS artificially low.
  • D.2 Correspondence readout: Correspondence readout follows a warp-integrate-project pipeline that warps sampled source-ray points, integrates their expected 3D location, and projects it into the target view.The method assumes a known 3D view-to-view warp and uses volume-rendering weights for integration.
  • D.2 Correspondence readout: Nerfies and HyperNeRF compose correspondences through canonical-space deformation, whereas NSFF chains explicit scene-flow predictions across time.T-NeRF directly maps space-time to color and density but makes temporal correspondences difficult to compute.

E Summary of the capture setup and data processing for our iPhone dataset

The iPhone dataset combines multi-camera captures for novel-view evaluation with single-camera captures for correspondence evaluation and training-only sequences. Its processing includes synchronization, calibration, depth filtering, and background point-cloud construction, with manual cleanup required for some regularization inputs.

  • E Summary of the capture setup and data processing for our iPhone dataset: The dataset contains 7 multi-camera captures and 7 single-camera captures, with novel-view synthesis evaluated only on the former.Correspondence evaluation uses all captures, while single-camera sequences lack validation views for novel-view synthesis.
  • E Summary of the capture setup and data processing for our iPhone dataset: Multi-camera captures use one handheld training camera and two stationary validation cameras with a large baseline.The Record3D iPhone app records RGB and depth at each time step, and the validation cameras face inward from distinct viewpoints.
  • E Summary of the capture setup and data processing for our iPhone dataset: Training cameras are intentionally moved in front of test cameras at selected frames to increase evaluation-region coverage.Those test frames are excluded when the training camera is positioned in front of the corresponding test camera.
  • E Summary of the capture setup and data processing for our iPhone dataset: The multi-camera system is synchronized with Adobe Premiere Pro's audio-based functionality and calibrated using SIFT correspondences, depth back-projection, and PnP.Camera poses are aligned across sequences by solving relative SE(3) transformations from matched 2D pixels and 3D points.
  • E Summary of the capture setup and data processing for our iPhone dataset: Single-camera depth is filtered around object edges with a Sobel filter, while background point clouds are completed with TSDF Fusion and manually filtered.Manual filtering is needed because segmentation failures can introduce foreground regions into the background cloud.

F Summary of the implementation details and remaining differences

The implementation aligns major training conditions across T-NeRF, NSFF, Nerfies, and HyperNeRF, including ray counts and view-dependency modeling, while retaining four substantive methodological differences. Re-implementations reproduce prior Nerfies, HyperNeRF, and NSFF results under teleporting training settings.

  • F Summary of the implementation details and remaining differences: Training comparisons align total ray counts, ray undistortion, view-dependency modeling, and shared training recipes across models.The shared recipe uses batch size B = 6144 and N = 2.5 × 10^5 iterations for T-NeRF, Nerfies, and HyperNeRF.
  • F Summary of the implementation details and remaining differences: The NSFF implementation is adapted from a third-party codebase because publicly available training code was unavailable for the Nerfies-HyperNeRF dataset.The authors use HyperNeRF's hyperparameters after confirming that NSFF's defaults are unsuitable for long video sequences.
  • F Summary of the implementation details and remaining differences: Re-implemented Nerfies and HyperNeRF results closely match reported and repository results under teleporting training.The corresponding table reports official, repository, and reproduced results for both models.
  • F Summary of the implementation details and remaining differences: The modified NSFF re-implementation closely reproduces checkpoint-based results on one Yoon et al. sequence under teleporting training.The comparison uses official checkpoints because the original paper lacks per-sequence results.
  • F Summary of the implementation details and remaining differences: Four remaining comparison differences concern static-scene stabilization, sampling and rendering, NeRF coordinates, and flow supervision.Nerfies and HyperNeRF use 384 sampled points, while NSFF uses 256; the methods also differ in coordinate systems and motion supervision.

G Additional results on the impact of effective multi-view

Additional results compare models trained with and without camera teleportation using qualitative comparisons and per-scene quantitative breakdowns. The reported per-scene table distinguishes results computed with and without co-visibility masking.

  • G Additional results on the impact of effective multi-view: Figure 16 provides qualitative comparisons between models trained with and without camera teleportation on the Nerfies-HyperNeRF dataset.
  • G Additional results on the impact of effective multi-view: Table 7 reports per-scene quantitative results on the Nerfies-HyperNeRF dataset, with gray values computed without co-visibility masking.All models in this table are trained under the non-teleporting setting.

H Additional results on per-sequence quantitative performance breakdown

The appendix reports per-sequence quantitative breakdowns and additional qualitative novel-view synthesis results under the non-teleporting setting. These results cover the Nerfies-HyperNeRF and proposed iPhone datasets, with most models using ablation-derived regularizations.

  • Per-sequence quantitative performances are documented for Nerfies-HyperNeRF under non-teleporting capture and for the proposed iPhone dataset.
  • Additional novel-view synthesis results cover Nerfies-HyperNeRF and both multi-camera and single-camera captures from the proposed iPhone dataset.
  • Except NSFF, models on the proposed iPhone dataset use additional regularizations identified through ablation and marked with “++” relative to original models.

J Additional results on inferred correspondence

The appendix adds correspondence evaluations, preprocessing visualizations, and qualitative comparisons across datasets and capture settings. It also distinguishes masked from unmasked image evaluation and documents limitations of validation coverage for single-camera captures.

  • Inferred correspondence: PCK-T evaluates inferred correspondence on single-camera proposed iPhone captures when multi-camera validation is unavailable, with α = 0.05.
  • Capture protocol: The proposed iPhone multi-camera captures intentionally move the training camera in front of test cameras at selected times to increase scene coverage, excluding test frames depicting it.
  • Data preprocessing: Depth preprocessing filters noisy object-edge measurements with a Sobel filter; unfiltered reprojection produces erroneous floaters and excess training noise.
  • Data preprocessing: Keypoint annotations use COCO and StanfordExtra skeletons for humans and quadrupeds, while general objects receive 5 to 15 manually tracked keypoints.
  • Qualitative comparisons: Qualitative comparisons report Ω/ω input-sequence metrics across teleporting and non-teleporting settings, using common co-visibility masks where specified.
  • Qualitative comparisons: Full-image visualizations compare masked and unmasked rendering, while single-camera iPhone captures lack ground-truth validation frames.
Loading 2210.13445v1…