PaperlayerView exact revision on arXiv

Source-linked AI summary

WildRayZer: Self-supervised Large View Synthesis in Dynamic Environments

Xuweiyi Chen, Wentao Zhou, Zezhou Cheng

arXiv:2601.10716v1cs.CV

TL;DR

Dynamic content violates the static-scene assumptions behind novel view synthesis, complicating transient removal and static-background reconstruction. WildRayZer uses analysis by synthesis to derive motion supervision, mask dynamic inputs, and train on newly curated dynamic datasets. Across sparse-view NVS and motion-segmentation benchmarks, it consistently outperforms the evaluated baselines.

  • Problem

    Existing novel view synthesis methods assume static scenes, restricting their use of real-world dynamic videos and large-scale dynamic training data.

  • Method

    WildRayZer derives pseudo motion masks from static-rendering residuals, distills a motion estimator, masks dynamic tokens, and gates rendering supervision.

  • Results

    WildRayZer consistently outperforms each baseline across sparse-view transient-aware NVS and motion-segmentation benchmarks.

  • Takeaways & Limitations

    The results indicate that disentangling camera motion and object motion is feasible in a fully self-supervised setting.

Abstract

from arXiv · show

We present WildRayZer, a self-supervised framework for novel view synthesis (NVS) in dynamic environments where both the camera and objects move. Dynamic content breaks the multi-view consistency that static NVS models rely on, leading to ghosting, hallucinated geometry, and unstable pose estimation. WildRayZer addresses this by performing an analysis-by-synthesis test: a camera-only static renderer explains rigid structure, and its residuals reveal transient regions. From these residuals, we construct pseudo motion masks, distill a motion estimator, and use it to mask input tokens and gate loss gradients so supervision focuses on cross-view background completion. To enable large-scale training and evaluation, we curate Dynamic RealEstate10K (D-RE10K), a real-world dataset of 15K casually captured dynamic sequences, and D-RE10K-iPhone, a paired transient and clean benchmark for sparse-view transient-aware NVS. Experiments show that WildRayZer consistently outperforms optimization-based and feed-forward baselines in both transient-region removal and full-frame NVS quality with a single feed-forward pass.

1. Introduction

WildRayZer extends self-supervised novel view synthesis to dynamic environments, where prior methods’ static-scene assumptions limit scalability and real-world applicability. It estimates motion without ground-truth masks and introduces datasets for large-scale training and evaluation.

  • Motivation: Static-scene assumptions restrict existing novel view synthesis models to static inputs and datasets, limiting scalability in dynamic real-world environments.These models cannot fully exploit in-the-wild videos containing natural dynamic content.
  • WildRayZer: WildRayZer extends RayZer to sparse, unposed dynamic views while removing dynamic objects from final renderings.It reconstructs the scene implicitly from unposed multiview images despite both camera motion and object dynamics.
  • WildRayZer: A static renderer identifies dynamic regions from rendering deviations, which supervise a distilled motion-mask estimator without ground-truth dynamic masks.Pseudo masks combine DINOv3 features and SSIM, with copy-paste augmentation improving robustness.
  • Data and evaluation: Dynamic RealEstate-10K contains over 15K casually captured indoor sequences with moving cameras and objects, enabling systematic training and evaluation.The paired transient-and-clean benchmark supports evaluation of transient-aware novel view synthesis.
  • Results: Experiments show that WildRayZer outperforms prior baselines in novel view synthesis and dynamic motion prediction by a large margin.The reported superiority covers both existing datasets and newly collected benchmarks.

2. Related Work

Related work spans optimization-based and feed-forward novel view synthesis, pose-free and low-3D-bias approaches, and moving-object segmentation. These methods progressively relax pose, static-scene, or representation assumptions, but dynamic in-the-wild synthesis remains challenging.

  • Optimization-based NVS: NeRF and 3D Gaussian Splatting achieve high-quality rendering but rely on accurate poses and static scenes, limiting in-the-wild applicability.Robust variants relax these assumptions through appearance modeling or error down-weighting.
  • Reduced-supervision NVS: Pose-free and explicit-warping approaches reduce supervision requirements but can remain brittle under in-the-wild dynamics.Prior methods include NoPoSplat, SPFSplat, Splatt3R, and related alternatives.
  • Generalizable NVS: Generalizable NVS methods predict views or 3D representations across scenes, often using epipolar geometry, cost volumes, or large transformer models.LVSM reduces hand-crafted 3D inductive bias, while RayZer adds self-supervised pose-free learning for static imagery.
  • Moving Object Segmentation: Moving-object segmentation methods commonly use optical flow, trajectories, or unsupervised video object segmentation rather than the paper’s cross-view motion formulation.Unsupervised video object segmentation may capture salient static entities instead of true movers.

3. Dynamic RealEstate10K

Dynamic RealEstate10K addresses the scarcity of large dynamic NVS datasets by mining diverse real-estate and indoor-interaction videos. Its benchmarks support motion-mask evaluation and sparse-view transient-aware synthesis with controlled transient/clean pairs.

  • Dataset motivation: D-RE10K comprises 15K real indoor sequences mined from in-the-wild videos with moving cameras, humans, pets, and object interactions.The collection intentionally retains transient dynamics that static NVS datasets filter out.
  • Dataset construction: The curation pipeline identifies suitable public videos, filters low-quality clips, and samples candidate frames for dataset construction.Real-estate walkthroughs provide handheld motion, stable lighting, and mostly rigid indoor backgrounds.
  • Evaluation splits: The motion-mask benchmark provides human-verified per-frame annotations for Internet videos, with 25 validation and 74 test sequences.It also reports NVS metrics restricted to static regions for fair comparison.
  • Dataset motivation: Table 1 contrasts large static datasets with typically tiny dynamic datasets and presents D-RE10K as closing this scale gap.The dataset contains diverse transient objects including people, pets, and clutter.
  • Evaluation splits: D-RE10K-iPhone is a 50-sequence paired transient/clean benchmark for sparse-view transient-aware NVS.Tripod-mounted iPhone captures repositioned viewpoints and matched images with and without transient objects.

4. Methodology

WildRayZer adapts RayZer’s self-supervised transformer renderer to dynamic inputs by estimating motion, masking transient tokens, and gating supervision. Pseudo masks combine semantic and appearance discrepancies before cross-frame clustering and pixel refinement.

  • RayZer preliminaries: RayZer reconstructs a scene representation from unposed, uncalibrated static images and renders held-out views without 3D supervision.A camera estimator predicts poses and intrinsics, while an encoder-decoder reconstructs and renders the scene.
  • WildRayZer architecture: WildRayZer adds a motion estimator alongside the camera estimator, scene encoder, and renderer to disentangle transient motion from static 3D structure.An alternating optimization schedule prevents dynamic content from contaminating static scene tokens.
  • Alternating training: During motion-estimator training, a static renderer’s appearance and feature residuals become soft pseudo masks for BCE-with-logits supervision.The renderer and camera-related modules are fixed in this phase; the complementary renderer phase freezes the motion estimator.
  • Pseudo Motion Mask Pipeline: Pseudo-mask construction fuses DINOv3 semantic dissimilarity with SSIM appearance dissimilarity using adaptive weights that shift toward SSIM as rendering improves.DINOv3 features are L2-normalized, and both dissimilarity maps are z-score normalized before fusion.
  • Pseudo Motion Mask Pipeline: DINO patch embeddings are clustered across frames to identify consistently salient motion regions, then masks are upsampled and refined with smoothing, component removal, and GrabCut.Patch-resolution processing reduces computation by roughly 100× while preserving motion boundaries.
  • Dynamic-token masking: Predicted dynamic patches are zeroed before scene encoding, preventing transient tokens from entering the static scene representation.Copy-paste augmentation injects synthetic transient objects and supplies additional transient-mask supervision.

5. Experiments

WildRayZer is evaluated on dynamic-scene benchmarks using sparse-view novel view synthesis, transient-aware metrics, qualitative comparisons, and motion-mask analyses. It consistently outperforms baselines while improving cross-view completion, transient removal, and reconstruction quality.

  • Experimental Setup: The experiments use D-RE10K-Mask for transient-region evaluation and D-RE10K-iPhone for full-image fidelity, with two, three, or four input views and six target views.Metrics include PSNR, SSIM, LPIPS, mIoU, and recall against human-verified motion masks.
  • Main Results: WildRayZer consistently outperforms optimization-based and feed-forward baselines, producing sharper background reconstructions and reliable transient removal.On D-RE10K-iPhone, it also best recovers occluded background revealed across views.
  • Main Results: Qualitative comparisons show cleaner transient removal, better cross-view completion than RayZer + SAV, and stronger preservation of global geometry and fine details.Examples include kitchens and plants across DRE10K-Mask and DRE10K-iPhone.
  • Analysis: Naively masking tokens with off-the-shelf motion estimators produces blurry or occlusion artifacts, indicating that cross-view completion must be learned rather than achieved by masking alone.Co-segmentation over-masks, MegaSAM has diffuse noisy boundaries, and SAV can select the wrong target under sparse views.
  • Analysis: WildRayZer improves both static and transient regions while maintaining robust pose estimation under sparse, dynamic inputs.The static-versus-transient breakdown is conducted on D-RE10K-iPhone with two input views.
  • Ablation Study: Copy–paste augmentation improves cross-dataset motion-mask generalization, raising DAVIS mIoU from 3.4 to 31.0 on eight qualifying sequences.Copy–paste alone does not produce meaningful masks but helps when added after D-RE10K pretraining.
  • Ablation Study: Including DINOv3 features accelerates mask emergence from about 20k to 1.5k steps and raises final D-RE10K mIoU from 29.4 to 39.4.The comparison concerns motion-estimator input modalities.

6. Conclusions

The paper concludes that WildRayZer can disentangle object motion from static 3D structure in a fully self-supervised setting. It consistently outperforms baselines across sparse-view transient-aware NVS and motion-segmentation benchmarks, supported by newly curated dynamic datasets.

  • WildRayZer is a feed-forward self-supervised framework that disentangles object motion from static 3D structure in dynamic environments.
  • Across sparse-view transient-aware NVS and motion-segmentation benchmarks, WildRayZer consistently outperforms each baseline.
  • D-RE10K and D-RE10K-iPhone provide large-scale training data and a benchmark for dynamic NVS under sparse views.

Supplementary Material

The supplementary material documents the full training pipeline, benchmark details, extended visualizations, failure analyses, and additional results across the evaluated datasets.

  • Training Pipeline: Section A details RayZer pretraining, motion-mask learning, masked reconstruction, and joint training with copy–paste augmentation.
  • Benchmark: Section B details the D-RE10K-iPhone benchmark and clarifies its evaluation metrics.
  • Additional Analysis: Section C presents extended qualitative visualizations, motion-mask comparisons, failure analyses, and additional results on D-RE10K, D-RE10K-iPhone, and DAVIS.

A. Training Details

Training proceeds progressively from RayZer pretraining to self-supervised motion-mask learning, masked scene reconstruction, and joint copy–paste training. Pseudo-label filtering, synthetic transient masks, and staged optimization improve stability and supervision quality.

  • RayZer Pretraining: RayZer is pretrained with scene and pose latent representations using 16 × 16 patches from 256 × 256 inputs.The encoder has 12 transformer layers plus eight geometry-specific layers, and the decoder has 12 layers.
  • Motion Mask Training Stage: The motion-mask predictor learns from DINOv3 and SSIM-derived pseudo-labels while the pretrained renderer remains frozen.Pseudo-labels combine semantic and structural rendering differences.
  • Motion Mask Training Stage: Only samples with rendering PSNR > 17 dB supervise motion-mask learning, reducing exposure to noisy pseudo-labels.
  • Masked Latent Scene Reconstruction Stage: Masked latent scene reconstruction freezes the motion predictor and trains the renderer to complete static content across masked regions.Training masks 10% of tokens using a mask-token strategy tailored to novel view synthesis.
  • Joint Training with Copy–Paste Augmentation: Copy–paste augmentation inserts synthetic transient objects into static scenes, providing explicit mask supervision while training the renderer for transient robustness.Objects include animals, vehicles, and people sampled from COCO.
  • Training Objective: The motion-mask objective uses binary cross-entropy with DINOv3 or paste masks as targets and λmask = 1.0.
  • Training Strategy: Progressive multi-stage training is essential because final-stage-only training produces rendering artifacts from inaccurate early motion-mask predictions.

B. Benchmarking Details

The benchmark details section covers additional D-RE10K-iPhone benchmark information and masked image quality metrics, with implementation released as a reference.

  • The section directs readers to Section B.1 for D-RE10K-iPhone benchmark details and Section B.2 for masked image quality metrics.
  • The implementation will be released as a reference.

B.1. D-RE10K-iPhone Benchmark Details

D-RE10K-iPhone is a paired transient/clean benchmark of real-world sequences with humans and vehicles, designed for sparse-view evaluation under controlled viewpoint and illumination conditions.

  • An iPhone mounted on a tripod with a Bluetooth shutter minimizes pose differences between paired images.
  • Pairs with noticeable illumination changes are discarded by comparing static background regions.Examples include sudden occlusions or the reappearance of direct sunlight.
  • For v=2, 3, 4 sparse-view settings, input views span each sequence’s full camera-angle range to evaluate genuine novel view synthesis rather than near-view interpolation.

B.2. Masked Image Quality Metrics

The evaluation uses mask-restricted PSNR, SSIM, and LPIPS to measure image quality separately in transient and static regions, alongside qualitative motion-mask comparisons and failure analysis.

  • Masked image quality metrics: PSNR, SSIM, and LPIPS are computed over real-valued spatial masks to evaluate transient and static scene components separately.For D-RE10K-iPhone, static regions use Mstatic = 1 − Mtransient.
  • Masked image quality metrics: Masked PSNR uses a mask-restricted mean squared error to provide localized pixel-level fidelity within the selected region.
  • Masked image quality metrics: Masked SSIM computes a standard SSIM map with an 11 × 11 Gaussian window before downsampling the mask to SSIM resolution.The stated stability constants are C1 = (0.01)2 and C2 = (0.03)2.
  • Masked image quality metrics: Masked LPIPS applies area-pooled masks to spatial feature-difference maps at each feature level, maintaining perceptual masking across scales.
  • Motion-mask comparisons: WildRayZer’s learned motion masks are reported as most faithful to true motion boundaries among the compared generators without ground-truth motion labels.
  • Failure cases: Failure cases include partial-object masks, under-segmentation, and degraded mask quality when moving objects occupy a large image fraction.

C.3. Additional Visualizations

Additional visualizations show WildRayZer across D-RE10K, unseen DAVIS sequences, and D-RE10K-iPhone, while also documenting representative motion-mask failure cases.

  • The additional results include 12 examples spanning D-RE10K, unseen DAVIS sequences, and D-RE10K-iPhone.
  • On challenging D-RE10K scenes, WildRayZer removes transients while plausibly completing occluded structure across input views.
  • The unseen DAVIS examples demonstrate transfer of learned motion masks and masked rendering to different content and capture conditions.
  • D-RE10K-iPhone examples show sharp static geometry and clean background reconstruction under casual handheld capture.
  • Failure cases include masks that cover only moving object parts, miss small regions, or under-segment large transient objects.
Loading 2601.10716v1…