WildRayZer: Self-supervised Large View Synthesis in Dynamic Environments

Motivation

• Static-scene assumptions restrict existing novel view synthesis models to static inputs and datasets, limiting scalability in dynamic real-world environments.These models cannot fully exploit in-the-wild videos containing natural dynamic content.Source: §1 · p. 1

WildRayZer

• WildRayZer extends RayZer to sparse, unposed dynamic views while removing dynamic objects from final renderings.It reconstructs the scene implicitly from unposed multiview images despite both camera motion and object dynamics.Source: §1 · p. 1, Figure 1 · p. 1
• A static renderer identifies dynamic regions from rendering deviations, which supervise a distilled motion-mask estimator without ground-truth dynamic masks.Pseudo masks combine DINOv3 features and SSIM, with copy-paste augmentation improving robustness.Source: Figure 1 · p. 1, §1 · p. 2

Data and evaluation

• Dynamic RealEstate-10K contains over 15K casually captured indoor sequences with moving cameras and objects, enabling systematic training and evaluation.The paired transient-and-clean benchmark supports evaluation of transient-aware novel view synthesis.Source: §1 · p. 2

Results

• Experiments show that WildRayZer outperforms prior baselines in novel view synthesis and dynamic motion prediction by a large margin.The reported superiority covers both existing datasets and newly collected benchmarks.Source: §1 · p. 2

Optimization-based NVS

• NeRF and 3D Gaussian Splatting achieve high-quality rendering but rely on accurate poses and static scenes, limiting in-the-wild applicability.Robust variants relax these assumptions through appearance modeling or error down-weighting.Source: §2 · p. 2

Reduced-supervision NVS

• Pose-free and explicit-warping approaches reduce supervision requirements but can remain brittle under in-the-wild dynamics.Prior methods include NoPoSplat, SPFSplat, Splatt3R, and related alternatives.Source: §2 · p. 2

Generalizable NVS

• Generalizable NVS methods predict views or 3D representations across scenes, often using epipolar geometry, cost volumes, or large transformer models.LVSM reduces hand-crafted 3D inductive bias, while RayZer adds self-supervised pose-free learning for static imagery.Source: §2 · p. 2

Moving Object Segmentation

• Moving-object segmentation methods commonly use optical flow, trajectories, or unsupervised video object segmentation rather than the paper’s cross-view motion formulation.Unsupervised video object segmentation may capture salient static entities instead of true movers.Source: §2 · p. 2

Dataset motivation

• D-RE10K comprises 15K real indoor sequences mined from in-the-wild videos with moving cameras, humans, pets, and object interactions.The collection intentionally retains transient dynamics that static NVS datasets filter out.Source: §3 · p. 3, Table 1 · p. 3

Dataset construction

• The curation pipeline identifies suitable public videos, filters low-quality clips, and samples candidate frames for dataset construction.Real-estate walkthroughs provide handheld motion, stable lighting, and mostly rigid indoor backgrounds.Source: §3.1 · p. 3, §3 · p. 3

Evaluation splits

• The motion-mask benchmark provides human-verified per-frame annotations for Internet videos, with 25 validation and 74 test sequences.It also reports NVS metrics restricted to static regions for fair comparison.Source: §3.2 · p. 3

Dataset motivation

• Table 1 contrasts large static datasets with typically tiny dynamic datasets and presents D-RE10K as closing this scale gap.The dataset contains diverse transient objects including people, pets, and clutter.Source: Table 1 · p. 3

Evaluation splits

• D-RE10K-iPhone is a 50-sequence paired transient/clean benchmark for sparse-view transient-aware NVS.Tripod-mounted iPhone captures repositioned viewpoints and matched images with and without transient objects.Source: §3.2 · p. 3

RayZer preliminaries

• RayZer reconstructs a scene representation from unposed, uncalibrated static images and renders held-out views without 3D supervision.A camera estimator predicts poses and intrinsics, while an encoder-decoder reconstructs and renders the scene.Source: §4.1 · p. 4

WildRayZer architecture

• WildRayZer adds a motion estimator alongside the camera estimator, scene encoder, and renderer to disentangle transient motion from static 3D structure.An alternating optimization schedule prevents dynamic content from contaminating static scene tokens.Source: §4.2 · p. 4

Alternating training

• During motion-estimator training, a static renderer’s appearance and feature residuals become soft pseudo masks for BCE-with-logits supervision.The renderer and camera-related modules are fixed in this phase; the complementary renderer phase freezes the motion estimator.Source: §4.2 · p. 5

Pseudo Motion Mask Pipeline

• Pseudo-mask construction fuses DINOv3 semantic dissimilarity with SSIM appearance dissimilarity using adaptive weights that shift toward SSIM as rendering improves.DINOv3 features are L2-normalized, and both dissimilarity maps are z-score normalized before fusion.Source: §4.2 · p. 5
• DINO patch embeddings are clustered across frames to identify consistently salient motion regions, then masks are upsampled and refined with smoothing, component removal, and GrabCut.Patch-resolution processing reduces computation by roughly 100× while preserving motion boundaries.Source: §4.2 · pp. 5–6, Figure 3 · p. 5

Dynamic-token masking

• Predicted dynamic patches are zeroed before scene encoding, preventing transient tokens from entering the static scene representation.Copy-paste augmentation injects synthetic transient objects and supplies additional transient-mask supervision.Source: §4.2 · p. 6

Experimental Setup

• The experiments use D-RE10K-Mask for transient-region evaluation and D-RE10K-iPhone for full-image fidelity, with two, three, or four input views and six target views.Metrics include PSNR, SSIM, LPIPS, mIoU, and recall against human-verified motion masks.Source: §5.1 · p. 6

Main Results

• WildRayZer consistently outperforms optimization-based and feed-forward baselines, producing sharper background reconstructions and reliable transient removal.On D-RE10K-iPhone, it also best recovers occluded background revealed across views.Source: §5.2 · p. 6
• Qualitative comparisons show cleaner transient removal, better cross-view completion than RayZer + SAV, and stronger preservation of global geometry and fine details.Examples include kitchens and plants across DRE10K-Mask and DRE10K-iPhone.Source: Figure 4 · p. 7

Analysis

• Naively masking tokens with off-the-shelf motion estimators produces blurry or occlusion artifacts, indicating that cross-view completion must be learned rather than achieved by masking alone.Co-segmentation over-masks, MegaSAM has diffuse noisy boundaries, and SAV can select the wrong target under sparse views.Source: §5.2 · p. 7, §5.3 · pp. 7–8
• WildRayZer improves both static and transient regions while maintaining robust pose estimation under sparse, dynamic inputs.The static-versus-transient breakdown is conducted on D-RE10K-iPhone with two input views.Source: §5.3 · pp. 7–8, Table 3 · p. 8

Ablation Study

• Copy–paste augmentation improves cross-dataset motion-mask generalization, raising DAVIS mIoU from 3.4 to 31.0 on eight qualifying sequences.Copy–paste alone does not produce meaningful masks but helps when added after D-RE10K pretraining.Source: §5.4 · p. 8, Table 5 · p. 8
• Including DINOv3 features accelerates mask emergence from about 20k to 1.5k steps and raises final D-RE10K mIoU from 29.4 to 39.4.The comparison concerns motion-estimator input modalities.Source: §5.4 · p. 9

• WildRayZer is a feed-forward self-supervised framework that disentangles object motion from static 3D structure in dynamic environments.Source: §6 · p. 9
• Across sparse-view transient-aware NVS and motion-segmentation benchmarks, WildRayZer consistently outperforms each baseline.Source: §6 · p. 9
• D-RE10K and D-RE10K-iPhone provide large-scale training data and a benchmark for dynamic NVS under sparse views.Source: §6 · p. 9

Training Pipeline

• Section A details RayZer pretraining, motion-mask learning, masked reconstruction, and joint training with copy–paste augmentation.Source: Supplementary Material · p. 14

Benchmark

• Section B details the D-RE10K-iPhone benchmark and clarifies its evaluation metrics.Source: Supplementary Material · p. 14

Additional Analysis

• Section C presents extended qualitative visualizations, motion-mask comparisons, failure analyses, and additional results on D-RE10K, D-RE10K-iPhone, and DAVIS.Source: Supplementary Material · p. 14

RayZer Pretraining

• RayZer is pretrained with scene and pose latent representations using 16 × 16 patches from 256 × 256 inputs.The encoder has 12 transformer layers plus eight geometry-specific layers, and the decoder has 12 layers.Source: A. Training Details · p. 14

Motion Mask Training Stage

• The motion-mask predictor learns from DINOv3 and SSIM-derived pseudo-labels while the pretrained renderer remains frozen.Pseudo-labels combine semantic and structural rendering differences.Source: A. Training Details · p. 14
• Only samples with rendering PSNR > 17 dB supervise motion-mask learning, reducing exposure to noisy pseudo-labels.Source: A. Training Details · p. 14

Masked Latent Scene Reconstruction Stage

• Masked latent scene reconstruction freezes the motion predictor and trains the renderer to complete static content across masked regions.Training masks 10% of tokens using a mask-token strategy tailored to novel view synthesis.Source: A. Training Details · p. 14

Joint Training with Copy–Paste Augmentation

• Copy–paste augmentation inserts synthetic transient objects into static scenes, providing explicit mask supervision while training the renderer for transient robustness.Objects include animals, vehicles, and people sampled from COCO.Source: A. Training Details · pp. 14–15, Figure 6 · p. 15

Training Objective

• The motion-mask objective uses binary cross-entropy with DINOv3 or paste masks as targets and λmask = 1.0.Source: A. Training Details · p. 15

Training Strategy

• Progressive multi-stage training is essential because final-stage-only training produces rendering artifacts from inaccurate early motion-mask predictions.Source: A. Training Details · p. 15

• The section directs readers to Section B.1 for D-RE10K-iPhone benchmark details and Section B.2 for masked image quality metrics.Source: B. Benchmarking Details · p. 15
• The implementation will be released as a reference.Source: B. Benchmarking Details · p. 15

• An iPhone mounted on a tripod with a Bluetooth shutter minimizes pose differences between paired images.Source: §B.1 · p. 15
• Pairs with noticeable illumination changes are discarded by comparing static background regions.Examples include sudden occlusions or the reappearance of direct sunlight.Source: §B.1 · p. 15
• For v=2, 3, 4 sparse-view settings, input views span each sequence’s full camera-angle range to evaluate genuine novel view synthesis rather than near-view interpolation.Source: §B.1 · p. 15

Masked image quality metrics

• PSNR, SSIM, and LPIPS are computed over real-valued spatial masks to evaluate transient and static scene components separately.For D-RE10K-iPhone, static regions use Mstatic = 1 − Mtransient.Source: §B.2 · pp. 15–16
• Masked PSNR uses a mask-restricted mean squared error to provide localized pixel-level fidelity within the selected region.Source: §B.2 · p. 15
• Masked SSIM computes a standard SSIM map with an 11 × 11 Gaussian window before downsampling the mask to SSIM resolution.The stated stability constants are C1 = (0.01)2 and C2 = (0.03)2.Source: §B.2 · p. 15
• Masked LPIPS applies area-pooled masks to spatial feature-difference maps at each feature level, maintaining perceptual masking across scales.Source: §B.2 · p. 15

Motion-mask comparisons

• WildRayZer’s learned motion masks are reported as most faithful to true motion boundaries among the compared generators without ground-truth motion labels.Source: §C.1 · p. 16

Failure cases

• Failure cases include partial-object masks, under-segmentation, and degraded mask quality when moving objects occupy a large image fraction.Source: §C.2 · p. 16

• The additional results include 12 examples spanning D-RE10K, unseen DAVIS sequences, and D-RE10K-iPhone.Source: §C.3 · p. 16, Figure 9 · p. 18
• On challenging D-RE10K scenes, WildRayZer removes transients while plausibly completing occluded structure across input views.Source: §C.3 · p. 16
• The unseen DAVIS examples demonstrate transfer of learned motion masks and masked rendering to different content and capture conditions.Source: §C.3 · p. 16
• D-RE10K-iPhone examples show sharp static geometry and clean background reconstruction under casual handheld capture.Source: §C.3 · p. 16
• Failure cases include masks that cover only moving object parts, miss small regions, or under-segment large transient objects.Source: Figure 8 · p. 17