PaperlayerView exact revision on arXiv

Source-linked AI summary

RayZer: A Self-supervised Large View Synthesis Model

Hanwen Jiang, Hao Tan, Peng Wang, Haian Jin, Yue Zhao, Sai Bi, Kai Zhang, Fujun Luan, Kalyan Sunkavalli, Qixing Huang, Georgios Pavlakos

arXiv:2505.00702v1cs.CV

TL;DR

3D Vision models commonly depend on ground-truth geometry and camera poses, limiting the move toward scalable learning from unposed imagery. RayZer trains a feed-forward multi-view model without 3D supervision by disentangling cameras and scenes and using ray structure as its only 3D prior. It achieves novel-view synthesis performance comparable to or better than pose-supervised oracle methods, while relying on assumptions about camera intrinsics and unordered, potentially noisy image sets.

  • Problem

    3D Vision models still rely heavily on ground-truth geometry and camera-pose labels, which are costly to obtain and may be imperfect.

  • Method

    RayZer predicts cameras, reconstructs a latent scene representation, and renders target views using self-predicted poses, with a transformer architecture guided only by ray structure.

  • Results

    RayZer achieves novel-view synthesis performance comparable to or better than oracle methods that use pose labels during training and testing.

  • Takeaways & Limitations

    The results support the feasibility of training a large multi-view 3D Vision model without 3D geometry or camera annotations.

  • Takeaways & Limitations

    RayZer assumes shared x- and y-axis focal lengths, shared intrinsics across views, and a principal point at the image center, while unordered image sets may be limited in scale and noisy.

Abstract

from arXiv · show

We present RayZer, a self-supervised multi-view 3D Vision model trained without any 3D supervision, i.e., camera poses and scene geometry, while exhibiting emerging 3D awareness. Concretely, RayZer takes unposed and uncalibrated images as input, recovers camera parameters, reconstructs a scene representation, and synthesizes novel views. During training, RayZer relies solely on its self-predicted camera poses to render target views, eliminating the need for any ground-truth camera annotations and allowing RayZer to be trained with 2D image supervision. The emerging 3D awareness of RayZer is attributed to two key factors. First, we design a self-supervised framework, which achieves 3D-aware auto-encoding of input images by disentangling camera and scene representations. Second, we design a transformer-based model in which the only 3D prior is the ray structure, connecting camera, pixel, and scene simultaneously. RayZer demonstrates comparable or even superior novel view synthesis performance than ``oracle'' methods that rely on pose annotations in both training and testing. Project: https://hwjiang1510.github.io/RayZer/

1. Introduction

RayZer addresses the reliance of 3D Vision models on imperfect ground-truth geometry and pose labels by training a multi-view model with self-supervision. It disentangles camera and scene representations, uses ray structure as its only 3D prior, and achieves novel-view synthesis performance comparable to or better than pose-supervised oracle methods.

  • Motivation: 3D Vision models often rely on time-consuming and imperfect ground-truth geometry and camera-pose annotations, motivating learning without 3D supervision.The paper frames this reliance as limiting scalability and effectiveness.
  • RayZer: RayZer takes unposed, uncalibrated multi-view images, predicts cameras and a scene representation, and renders novel views in a feed-forward pipeline.Training uses self-predicted camera poses to render target views, with photometric supervision from input images.
  • Self-supervised framework: Splitting images into input and target sets disentangles scene reconstruction from camera estimation and prevents trivial solutions that are not 3D-aware.The target set supplies photometric self-supervision when its estimated poses render the scene predicted from the input set.
  • Model design: RayZer uses transformers without explicit 3D representations or hand-crafted rendering equations; its only 3D prior is the ray structure linking cameras, pixels, and scenes.Predicted camera poses become pixel-aligned Plücker ray maps that guide scene reconstruction and let camera and scene representations regularize each other.
  • Results: RayZer achieves novel-view synthesis performance comparable to or better than oracle methods using pose labels during training and testing.The authors report that noisy COLMAP annotations can limit oracle-model performance.

2. Related Work

Prior large 3D Vision models use architectural priors, latent representations, or optimization, but commonly retain pose-supervision or inference constraints. RayZer instead explores feed-forward self-supervised learning from unposed multi-view imagery.

  • Large-scale 3D Vision Models: Large 3D Vision models either incorporate multiview-stereo and epipolar biases or use transformer architectures that omit explicit architectural 3D priors.Latent representations and learned rendering functions improve scalability, but related models still require pose supervision or accurate inference-time camera annotations.
  • Comparison: RayZer explores self-supervised training to avoid the ground-truth pose requirements retained by related large 3D models.This positions RayZer as a model-level alternative to supervised scaling and per-scene optimization.
  • Self-supervised 3D Representation Learning: RayZer differs from prior self-supervised multi-view methods by explicitly estimating camera poses before conditioning latent scene reconstruction.The cited comparison contrasts this order with RUST's inverse pipeline.
  • Self-supervised 3D Representation Learning: RayZer uses explicit pose representations and geometrically interpolates predicted poses for novel-view synthesis, whereas RUST uses latent pose representations.The comparison states that latent pose representations make scene-pose disentanglement challenging and are not explicitly 3D-aware.
  • Optimization-based Unsupervised SfM, SLAM, and NVS: Unlike optimization-based SfM, SLAM, and NVS methods, RayZer is a feed-forward parametrized model that learns priors from large-scale training data.Traditional optimization workflows are described as requiring dense views, slow speed, and sensitivity to hyperparameters.

3. Preliminaries

RayZer uses a latent set scene representation whose 3D awareness is learned rather than explicitly encoded. A learned renderer maps the scene representation and a ray to the property of the corresponding pixel.

  • Latent set scene representation: The latent set representation compresses scene information into tokens rather than explicitly encoding a 3D structure.Its 3D-awareness properties are fully learned, unlike classical explicit, implicit, and hybrid 3D representations.
  • Latent set scene representation: The scene representation is denoted z ∈ R^n×d, where n is the number of tokens and d is the latent feature dimension.The representation is a set of latent scene tokens.
  • Rendering: A learned rendering network R_θ maps the latent scene representation z and a ray r to a rendered property v for the corresponding pixel.The rendered property can be RGB values.
  • Rendering: LVSM groups rays from the same image patch and decodes them jointly for improved efficiency and performance.

4. RayZer

RayZer predicts cameras and a scene representation from unposed, uncalibrated multi-view images, then renders novel views. Its self-supervised design uses predicted camera rays to condition reconstruction and photometrically supervise rendering.

  • Self-supervised learning framework: RayZer takes unposed, uncalibrated multi-view images and predicts camera intrinsics, per-view poses, and a scene representation for novel view synthesis.The model targets static scenes and requires no 3D geometry or camera-pose annotations during training.
  • Self-supervised learning framework: The training data are split into non-overlapping subsets IA and IB: IA predicts the scene, while IB supplies rendered targets for photometric supervision.The subsets are randomly sampled during training, and the loss includes a perceptual-loss weight λ.
  • RayZer model architecture: RayZer estimates camera parameters first, converts them into pixel-aligned Plücker ray maps, and uses those rays to condition scene reconstruction.The ray representation captures 2D ray-pixel alignment and 3D ray geometry, providing fine-grained camera information to the reconstruction stage.
  • RayZer model architecture: The model uses pure self-attention transformers, with image patch tokens carrying spatial and image-index positional embeddings before camera estimation.The camera estimator updates learnable per-view camera tokens using full self-attention across the input views.
  • RayZer model architecture: Camera prediction uses one canonical reference view, relative SE(3) poses for other views, and a shared-intrinsics focal-length parameterization.The intrinsic assumptions are equal x- and y-axis focal lengths, shared intrinsics across views, and a principal point at the image center.
  • RayZer model architecture: A second transformer encoder predicts a latent set scene representation from IA and its predicted rays, after which predicted rays for IB drive target-image rendering.Rendered images are decoded at patch level and reshaped into the final 2D image; training compares them with IB using self-supervised photometric loss.

5. Experiments

RayZer is evaluated across scene-level and object-level datasets for novel view synthesis, against oracle, supervised, and self-supervised baselines. Results show competitive performance without pose labels, while ablations and pose analyses examine data ordering, learned pose geometry, and architectural choices.

  • Experimental Setup: RayZer is evaluated on DL3DV, RealEstate, and Objaverse using novel view synthesis quality, with dataset-specific input and target view counts.The evaluation uses 16/8 views for DL3DV, 5/5 for RealEstate, and 12/8 for Objaverse.
  • Main Results: RayZer performs on par with oracle methods on RealEstate and DL3DV and can outperform them on glasses and white-wall cases where COLMAP struggles.The visual comparisons attribute these cases to limitations of COLMAP-based pose annotations.
  • Experimental Setup: The baselines include oracle models using ground-truth poses during training and inference, PF-LRM with camera supervision, and RUST through design ablations.RayZer uses 24 transformer layers, while GS-LRM and LVSM use 16 layers for parameter-count fairness; PF-LRM uses 24.
  • Main Results: Without 3D labels, RayZer matches LVSM overall, outperforms it on DL3DV and RealEstate10k, and performs slightly worse on Objaverse.Objaverse uses perfect Blender pose annotations, whereas DL3DV and RealEstate use COLMAP annotations that may be imperfect.
  • Main Results: Training with unordered image sets performs worse than training with continuous video frames, with the difference arising at the pose-estimation stage.The paper suggests plentiful video resources may offer better scaling than limited, noisy unordered image sets.
  • Analysis and Ablation Study: Interpolated predicted poses test 3D awareness, while ablations show benefits from Plücker ray maps, latent scene representations, and pose-first processing.Explicit 3DGS training does not converge; removing ray maps degrades performance, and predicting poses after the scene degrades performance.

6. Conclusion

RayZer is introduced as a self-supervised large multi-view 3D vision model trained without 3D geometry or camera annotations. It achieves comparable or better novel-view synthesis than methods using pose labels in training and inference.

  • RayZer is trained with zero 3D supervision, including no 3D geometry or camera annotations.
  • Table 6 evaluates self-supervised pre-training for pose estimation by reading latent camera tokens with a supervised two-layer MLP and comparing supervised training from scratch.
  • Table 7 ablates RayZer’s scene representation, Plücker ray-map conditioning, and explicit-camera conversion choices.
  • RayZer achieves comparable or better novel-view synthesis performance than prior methods using pose labels during training and inference.

A. Experimental Details

The experiments use rendered Objaverse videos and a shared training protocol for RayZer and all baselines. Additional implementation details include transformer-based rendering from spatial, scene, and pose tokens.

  • Objaverse Data Details: Objaverse is rendered as continuous videos with cameras on a unit sphere and about 70 frames spanning azimuth 0° to 360°.Elevations are randomly sampled between -20° and 60° for each shape instance.
  • RayZer and all baselines are trained with the same protocol using 32 A100 GPUs and a total batch size of 256.Training uses a 3000-iteration warmup, cosine learning-rate decay, 50,000 steps for baselines, and gradient clipping above norm 1.0.
  • Table 8 studies the effects of canonical-view selection and frame-sampling curriculum for training on continuous video frames.
  • More Model Details: The rendering decoder receives spatial tokens, scene tokens, and a pose token, then uses updated spatial tokens to regress pixel values.

B. RayZer Training with Continuous Inputs

RayZer supports multi-view inputs from continuous video frames or unordered image sets. For continuous frames, it improves self-supervision through middle-frame canonical selection and curriculum-based frame sampling.

  • RayZer accepts multi-view inputs sampled from continuous video frames or unordered image sets.
  • Canonical View Selection: The middle timestep is selected as the canonical view because zero-mean pose-MLP initialization gives it smaller pose-data variance than first-frame selection.
  • Curriculum: Training difficulty increases through a curriculum that samples frames with progressively larger distance ranges.The curriculum uses dataset-specific frame-sampling ranges for DL3DV, RealEstate, and Objaverse.
  • Ablation: Removing either canonical-view selection or frame-sampling curriculum degrades performance in Table 8.The ablation supports both techniques for training on continuous video frames.

C. More Results

Additional results examine RayZer’s failure cases and visual comparisons. RayZer generally performs on par with GS-LRM and LVSM while requiring no camera-pose annotations.

  • Failure Case Pattern: RayZer can fail on fine-grained geometry, complicated materials, and occlusions.Examples include complicated plant geometry, stacked glasses, and specular reflections on silver teapots.
  • More Comparisons: RayZer generally performs on par with GS-LRM and LVSM while remaining self-supervised and requiring no camera-pose annotations.
  • More Visualization: Figures 9–11 provide additional visual comparisons against ground-truth novel views.

D. More Discussion

RayZer’s strong novel-view synthesis may combine pose estimation with 3D-aware video-frame interpolation, especially when camera baselines are large. The authors identify improved disentanglement of these factors as an important future direction.

  • More Discussion: RayZer may jointly learn actual camera poses and 3D-aware video-frame interpolation, helping explain strong synthesis despite imperfect pose estimation.The authors conjecture this joint learning behavior to explain the gap between synthesis quality and fine-tuned pose accuracy.
  • More Discussion: On RealEstate’s small camera baselines, RayZer mainly focuses on pose estimation, whereas larger-baseline datasets also benefit from interpolation cues.RealEstate pose estimation is reported as accurate; DL3DV and Objaverse make pose estimation harder under self-supervision.
  • More Discussion: Improving disentanglement between interpolation and pose estimation is an important future direction for learning real-world SE(3) camera poses.The authors contrast continuous video frames, which encourage local pose smoothness, with unordered image sets, which discard that prior.
  • More Discussion: The visual-compression figures compare RayZer and oracle methods on DL3DV, while separate figures place target novel views above RayZer-rendered views.Figures 9–11 show this target-versus-rendered layout for DL3DV, RealEstate, and Objaverse.
Loading 2505.00702v1…