PaperlayerView exact revision on arXiv

Source-linked AI summary

RayDer: Scalable Self-Supervised Novel View Synthesis from Real-World Video

Ulrich Prestel, Stefan Andreas Baumann, Nick Stracke, Björn Ommer

arXiv:2605.31535v1cs.CVcs.AIcs.LG

TL;DR

Self-supervised NVS has struggled to scale because multi-network pipelines are difficult to predict and realistic videos contain destabilizing dynamic content. RayDer unifies camera estimation, reconstruction, and rendering in one transformer and treats dynamics as a nuisance state, yielding clean scaling across data and compute while retaining static-scene NVS. Its main scope boundary is that unseen dynamic content may be blurred or loosely interpolated, and full dynamic NVS remains future work.

  • Problem

    Self-supervised NVS seeks to use abundant unlabeled video without expensive camera-pose labels, but multi-network designs and unconstrained dynamic videos have constrained scalable training.

  • Method

    RayDer uses a unified feed-forward transformer for camera estimation, scene reconstruction, and rendering, with a nuisance dynamic state that absorbs time-varying content during training.

  • Results

    RayDer exhibits clean, predictable scaling across three orders of magnitude of data and four model sizes, with compute-optimal performance described by a single power law.

  • Takeaways & Limitations

    Self-supervised NVS can be studied as a single-model scaling problem when network interactions are consolidated and dynamic content is treated as a nuisance factor.

  • Takeaways & Limitations

    Unseen dynamic content can produce blurry mean estimates or loose interpolation, and extending the method to full dynamic 4D NVS is left for future work.

Abstract

from arXiv · show

Self-supervised novel view synthesis (NVS) remains challenging to scale, despite the abundance of video data, largely due to the brittleness of training on realistic videos and the hard-to-predict scaling behavior of multi-network system designs. We introduce RayDer, a unified, feed-forward transformer that consolidates camera estimation, scene reconstruction, and rendering into a single backbone, turning self-supervised NVS into a well-posed single-model scaling problem. A minimal dynamic state, treated as a nuisance factor, absorbs time-varying content and enables stable training on unconstrained real-world video. Importantly, RayDer keeps static-scene NVS as its target task: dynamic content is leveraged purely as scalable supervision, not reconstructed as in dynamic-scene (4D) NVS. Across multiple model sizes and orders of magnitude in data, RayDer exhibits clean power-law scaling with data and compute, and outperforms static-scene data mixtures. On a large number of benchmarks, RayDer achieves strong zero-shot open-set performance competitive with state-of-the-art supervised approaches. Project Page: https://compvis.github.io/rayder

1 Introduction

Self-supervised NVS could exploit abundant video, but realistic dynamic content and multi-network designs make scaling brittle. RayDer addresses both obstacles with a unified backbone and a nuisance dynamic state while retaining static-scene NVS as the target.

  • Challenge: Existing self-supervised NVS systems separate camera estimation, scene representation, and rendering across interacting networks, making scaling difficult to predict and expensive to sweep.Capacity must be allocated across multiple components whose joint behavior becomes brittle as models grow.
  • Challenge: Unconstrained videos contain dynamic content that destabilizes existing methods, preventing direct training on scalable real-world video.Stable learning under these conditions is required before scaling can be meaningfully studied.
  • RayDer: RayDer unifies camera estimation, scene reconstruction, and rendering in a single feed-forward transformer backbone for scalable self-supervised NVS.At fixed parameter counts, the unification improves pose estimation and novel view synthesis quality while enabling predictable scaling.
  • RayDer: RayDer uses a minimal explicit dynamic state as a nuisance factor that absorbs time-varying content during training without changing the static-scene NVS objective.The state is not used at inference time and prevents scene dynamics from corrupting camera-pose representations.
  • Scaling: Across three orders of magnitude in data and four model sizes, RayDer exhibits clean scaling in data and compute, described by a single compute-optimal power-law fit.The resulting behavior is presented as clean and predictable across data, model size, and compute.

2 Related Work

Prior NVS work spans per-scene optimization, feed-forward prediction, pose-free learning, dynamic-scene methods, and unified 3D vision models. These approaches trade off test-time cost, pose dependence, data requirements, and scalability in different ways.

  • Feed-Forward Novel View Synthesis: Per-scene NeRF and 3D Gaussian Splatting methods produce high-quality views but require dense pose captures and scene-specific fitting at test time.Feed-forward models amortize fitting by predicting scene or rendering representations from posed images, but often retain external pose-pipeline dependence.
  • Self-supervised and Pose-Free NVS: Pose-free NVS methods remove or reduce pose supervision through latent scene or pose representations, but some require partial target views or constrained training data.RUST requires partial target views, while other approaches learn from video without pose supervision.
  • Self-supervised and Pose-Free NVS: Dynamic-scene NVS can handle motion but may require multi-view, multi-dynamics data that is difficult to obtain at scale.This data requirement limits scalability for methods such as DyST.
  • Large 3D Vision Models: Large 3D vision models unify multiple geometry tasks in a single transformer backbone, replacing hand-engineered pipelines with scale and broader data.This line motivates architectural consolidation as a route to general 3D understanding across tasks.

3 Scaling Self-Supervised Novel View Synthesis

RayDer addresses the data, system, and quality bottlenecks that hinder scalable self-supervised NVS by combining stable dynamic-video training with a unified transformer and targeted attention and pose-learning designs.

  • Scaling bottlenecks: Existing self-supervised NVS scaling is limited by scarce static-scene video, brittle multi-network pipelines, and pose shortcuts or coarse patches that reduce reconstruction quality.The section frames these as data, system, and quality bottlenecks.
  • Preliminaries and baseline: RayZer uses separate camera-estimation, scene-reconstruction, and rendering ViT subnetworks, forming the baseline pipeline for comparison.Its camera estimator predicts poses and intrinsics, the reconstructor predicts a latent scene representation, and the renderer synthesizes target views.
  • Robust learning from dynamic videos: Dynamic-state prediction absorbs time-varying content separately from camera pose, and state dropout prevents inference-time dependence on the unknown target state.This minimal change eliminates observed training instabilities across experiments, while the state remains a nuisance variable rather than a 4D scene representation.
  • Scalability through network consolidation: RayDer consolidates camera and dynamic-state estimation, reconstruction, and rendering into one shared backbone, simplifying scaling and improving both NVS and camera estimation at fixed parameter count.The model operates in camera-estimation and novel-view-synthesis modes, with heavy computation concentrated in the shared backbone.
  • Improving synthesis quality: Factorized attention enables parallel target prediction and KV caching, reducing per-target compute by ∼7× at a minor quality trade-off.Input tokens attend only to one another, while target tokens attend to themselves and input tokens.
  • Improving synthesis quality: Randomized autoregressive pose learning closes the train-test gap from ordered generation and further improves camera estimation and NVS quality, while the final architecture scales across four model sizes.The final RayDer combines random-order autoregression, parallel-target attention, pose-and-state prediction, and stable training on general video.

4 Experiments

RayDer scales predictably across data, model size, and compute, while large-scale video supervision outperforms curated static-scene mixtures. It achieves strong zero-shot performance and transferable geometry, with limitations on clean lab scenes, unobserved regions, and dynamic content rendering.

  • Scaling Behavior across Data, Model Size, and Compute: More data unlocks higher performance ceilings, while larger models benefit more from additional data when training is sufficiently long.At small data scales, models converge to a common ceiling and larger models can overfit; scaling requires both data and model capacity.
  • Scaling Behavior across Data, Model Size, and Compute: R2 > 0.99: a single power law accurately describes RayDer’s compute-optimal behavior across model and dataset scales.Both compute and data contribute to performance, while irreducible error remains because some regions are occluded.
  • Static-Scene Data Does Not Enable the Same Scaling Regime: Training on more partially dynamic video empirically outweighs the domain-alignment benefit of curated static-scene data.The static-only model significantly underperforms, while adding static data to the larger video corpus provides only marginal gains.
  • Learned Camera Geometry: Transferability and Scaling: RayDer’s learned poses match XFactor and substantially improve over RayZer, indicating transferable camera geometry without dedicated transferability supervision.Continuous rotation and translation errors also decrease predictably with more data and larger models, while probe and cross-scene transfer errors improve together.
  • Open-set Novel View Synthesis: RayDer reaches state-of-the-art or near-state-of-the-art performance across most settings using more than an order of magnitude less training compute than larger competing models.It is trained self-supervised from scratch in a single stage, without pose supervision or pretrained foundation-model weights.
  • Open-set Novel View Synthesis: RayDer underperforms supervised methods on lab datasets such as DTU, primarily in clean, structureless-background regimes.The authors attribute this limitation to a mismatch between the training distribution of unconstrained real-world video and these lab conditions.
  • Closed-Set Static & Supervised Comparison: +2.9dB PSNR: RayDer outperforms supervised LVSM on SpatialVid using pseudo-ground-truth poses.Generating those pseudo-ground-truth poses cost approximately 69k GPU-h, versus approximately 1.2k GPU-h to train RayDer-B in the comparison.
  • Limitations: Unobserved regions are rendered as blurry, low-frequency mean estimates rather than plausible detailed content.On mixed static/dynamic scenes, static geometry remains correct but moving content is not rendered faithfully.

5 Conclusion

RayDer makes self-supervised NVS scalable by combining unified architecture with nuisance-state handling for dynamic video, while retaining static-scene NVS as the target. It shows clean scaling and strong zero-shot performance, motivating extensions toward supervision, generative modeling, 4D NVS, and 3D world models.

  • RayDer consolidates camera estimation, scene reconstruction, and rendering in one feed-forward transformer while preserving the static-scene NVS objective.
  • Explicit dynamic-state handling and architectural unification enable stable learning from generic video and clean scaling across data, model size, and compute.
  • The resulting model achieves strong zero-shot open-set performance competitive with supervised and video diffusion-based systems.
  • Future directions include partial supervision, generative modeling, 4D NVS, and continued scaling toward 3D world foundation models.

A Extended Exploration Details

The exploration evaluates RayDer across diverse video datasets, zero-shot benchmarks, model configurations, and training conditions, while documenting instability in selected baseline runs. Evaluation emphasizes unseen-data NVS and camera estimation.

  • RayZer trainings in the reported setting consistently diverged, whereas Config A results use selected runs that did not diverge.
  • RayDer is trained on diverse and curated video datasets containing dynamic and mostly static scenes.The datasets are SA-V and SpatialVid-HQ.
  • Zero-shot evaluation measures NVS on RealEstate-10k and camera estimation on DL3DV-10k.
  • The exploration covers ablations with extended NVS and camera-estimation metrics summarized in Table A.1.

B Implementation Details

Implementation reporting covers the hyperparameter tables, scaling configuration relationships, and hardware-dependent training-speed measurement. These details define how model variants and infrastructure comparisons should be interpreted.

  • Table B.2 reports the main exploration hyperparameters, which are identical for variants trained on SA-V and SV-HQ.
  • SCALING-S is derived from Config H and differs only in its training dataset.
  • The RayZer-setting model matches Jiang et al.’s depth, width, and view count and is trained only on DL3DV.
  • Training speed is measured on 4× Nvidia H200 nodes with GPUs power-limited to 500W and NDR200 interconnect.Other H200 setups may be faster.

B.1 Architecture Details

RayDer uses a unified transformer with role-conditioned attention, autoregressive view interactions, direct camera prediction, pixel-space supervision, and differentiable camera parameterizations. The implementation also addresses training stability and video-specific intrinsics.

  • Camera estimation: Camera poses are predicted pointwise from individual camera tokens rather than through a canonical-view relative MLP head.
  • Output heads: Output heads use RMSNorm followed by single linear layers for camera pose, intrinsics, and dynamic-state prediction.
  • Architecture: The transformer combines intra-view and global attention, with neighborhood attention added in shallow local high-resolution layers.
  • Attention masking: Causal attention makes target views independent during training and inference, while inference target views attend to all input views.
  • Token roles: Token conditioning distinguishes camera-estimation and NVS roles across view, camera, and state tokens.
  • Training supervision: RayDer trains end-to-end with pixel-space reconstruction loss on randomly partitioned context and target views, optionally adding perceptual loss.
  • Camera parameterization: Camera extrinsics use a 6D twist mapped to SE(3) through the exponential map, providing a differentiable minimal parameterization.
  • Camera intrinsics: Per-view focal lengths use f = exp(θf) + ϵf with ϵf = 10^-6, and per-view intrinsics accommodate zooming in some training videos.

B.2.1 Scaling Power Laws

The appendix fits power laws to NVS performance across compute and dataset scale, finding simple forms that describe the observed trends accurately across metrics and benchmarks.

  • Fitting procedure: Power laws are fit by first selecting the compute Pareto frontier separately for each dataset size, then fitting the target function to those models.Compute is measured in GFLOP, with points beginning at 50k training steps.
  • Fitting procedure: Standard power laws consistently provide good fits when modeling performance over compute for a fixed dataset size.The fitted form uses an irreducible metric component and a compute-dependent power-law term.
  • Joint scaling model: The shared compute-and-data power law omits offset constants because C0 and D0 tend toward zero without improving fit quality.More complex alternatives with cross terms or dataset-modulated compute terms were generally less effective.
  • Quantitative fits: MSE(C, D) ≈0.0033 + 200 · C−0.40 + 2.6 · D−0.60^2.82 achieves R2 = 0.997.The corresponding LPIPS and 1 − SSIM fits also report R2 = 0.997 with their metric-specific coefficients and exponents.
  • Model selection: An alternative formulation predicted worse performance with additional data at very low compute, a behavior the authors could not reproduce.Other explored forms generally produced bad or less optimal fits, with one LPIPS-specific exception.
  • Robustness: The identical compute-data power-law form remains accurate on additional, harder, noisier evaluation sets and metrics beyond RE10K.This supports that the scaling trend is not specific to a single benchmark.

B.2.2 Training Data Details

The appendix details video datasets, preprocessing, frame sampling, pose evaluation, and exploratory analyses of camera and dynamic-state behavior.

  • Datasets: Training uses original SpatialVid, its HQ subset, and SA-V videos, selected partly for diversity, review processes, and open licensing.These datasets support exploration of open-set data for self-supervised NVS.
  • Datasets: The 1% and 10% SpatialVid subsets are nested, fixed shard selections whose HQ-to-non-HQ ratio matches the full dataset.Using identical subsets across runs controls the data-scaling comparison.
  • Preprocessing: Videos are converted to high-bitrate H.264 at 6 fps to unify codecs and improve training throughput without extracting all frames as images.The preprocessing reduces frame-rate variation while avoiding a data-loading bottleneck.
  • Frame sampling: Training samples randomized chunks at an average 2 fps, with locally perturbed frame times; snippets shorter than 4 seconds are discarded.The standard setting uses 8 frames, requiring at least 4 seconds of source video.
  • Pose evaluation: Camera quality is assessed by probing frozen camera-estimator features into relative SE(3) trajectories and reporting t@α and R@α for α ∈ {10, 20, 30} degrees.Evaluation uses frame distances of 1 and 24, with pose normalization to [−1, 1].
  • Dynamic state: Matching dynamic state and camera pose yields the sharpest reconstructions, while mismatches can produce blurry frames because the nuisance state competes with pose.The state is intended for training stability and is discarded at inference, rather than treated as a disentangled scene representation.
  • Observed behavior: RayDer fails on some DTU settings when evaluation scenes differ substantially from the training distribution, especially for larger view transforms.The camera-estimation stage is particularly vulnerable in that setting.
  • Observed behavior: Unobserved scene regions receive blurry averaged patches in novel views.This behavior is shown as a qualitative failure case rather than a reported metric.

C.2 Further Failure Cases/Limitations

The appendix documents distribution-shift failures, unobserved-region blur, and unstable RayZer training on dynamic-video datasets.

  • Distribution shift: RayDer performs poorly on DTU in the SEVA setting, plausibly because object-centric scenes with black-and-white backgrounds lie outside its open-set real-world training distribution.Other evaluation settings closer to training conditions perform better.
  • Novel-view artifacts: Novel views contain blurred artifacts in regions absent from all context views, along with blur in fine details and objects near the camera.The authors attribute the unobserved-region effect to the regression objective and note similar behavior in related methods.
  • RayZer stability: RayZer training on SpatialVid or SA-V is unstable, with early convergence followed by divergence or stalled training despite many configuration changes.The instability depends strongly on batch size and view distance, and stable configurations for true NVS were not found.
  • RayZer stability: Divergence produces gradient spikes and sharp PSNR drops, whereas stalled training skips most steps after gradient norms exceed a threshold.Disabling step skipping prevents stalling but still leads to degeneracies such as divergence.
  • RayZer stability: Smaller batch sizes generally cause faster divergences, and the learned camera space can collapse to a degenerate SE(3)-interpolation solution.The reported main-table runs use the best evaluation PSNR obtained before stalling or divergence.

D Additional Evaluations

Additional evaluations show strong SSIM across many open-set settings, weaker LPIPS without perceptual supervision, and implementation details affecting reported comparisons.

  • Open-set evaluation: RayDer achieves near-state-of-the-art SSIM across several datasets despite using only self-supervised MSE reconstruction loss.The evaluated RayDer-L-5762 model uses λperc = 0 and no perceptual supervision.
  • Open-set evaluation: LPIPS results are weaker, consistent with the absence of perceptual loss during training.The appendix reports LPIPS across a broad set of datasets and evaluation settings.
  • Implementation: The reported results use a corrected implementation after fixing bugs including unsynchronized DDP gradients without gradient accumulation.The authors observed similar instabilities with and without these fixes, while the corrected implementation generally gave the best results.
  • Table conventions: Evaluation tables identify split abbreviations for ReconFusion, ViewCrafter, and SEVA, and list datasets including LLFF, DTU, CO3D, WRGBD, M360, and T&T.Kaleido uses 512² resolution instead of 576², and diffusion-based or multi-dataset checkpoints are marked separately.

E Additional Samples

The supplementary section provides qualitative novel-view synthesis examples across static and dynamic training settings, including sparse-view interpolation and comparisons with prior self-supervised NVS.

  • Additional Samples: The supplementary material also includes video visualizations of interpolated fly-throughs and comparisons with E-RayZer on DL3DV-10K.E-RayZer is identified as the primary prior self-supervised NVS method trained on a mixture of static-scene datasets.
  • Additional Samples: RayDer-B samples trained on DL3DV-10k compare ground-truth images with generated novel views using the benchmark’s official even-indexed inputs.The ground-truth images appear above the generated views.
  • Additional Samples: RayDer-L produces zero-shot open-set samples on WildRGBD and DL3DV-10k with sparse input views.These examples evaluate generalization using few context images.
  • Additional Samples: On DL3DV-10k, RayDer synthesizes smooth intermediate views by interpolating between predicted camera poses from sparse context images.The figure specifically illustrates zero-shot view interpolation.
  • Additional Samples: RayDer-L trained on dynamic data is evaluated zero-shot for static-scene NVS on RealEstate10k, WildRGBD, and LLFF under sparse-view settings.The examples use two input images for RealEstate10k and WildRGBD, and three for LLFF.
Loading 2605.31535v1…