PaperlayerView exact revision on arXiv

Source-linked AI summary

One View Is Enough! Monocular Training for In-the-Wild Novel View Generation

Adrien Ramanana Rahary, Nicolas Dufour, Patrick Perez, David Picard

arXiv:2603.23488v2cs.CV

TL;DR

Monocular novel-view synthesis depends on scarce multi-view supervision. OVIE trains on single images using depth-derived pseudo-views and, on unseen benchmarks, rivals or surpasses prior methods.

  • Problem

    Monocular novel-view synthesis depends on scarce multi-view supervision, while common purpose-built datasets cover only a narrow slice of visual content.

    Source: §1 · p. 2

  • Method

    OVIE uses monocular-depth-derived pseudo-views and masked losses to train a feed-forward model that requires no geometry at inference.

    Source: §1 · pp. 2–3, §3.3 · p. 7

  • Results

    On two unseen benchmarks, OVIE is competitive across both; it rivals or outperforms in-domain geometry-free methods on RealEstate10K and surpasses all baselines on DL3DV.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The results support acquiring 3D-aware view-synthesis capability from internet-scale 2D images, including for domains where multi-view capture is impossible.

    Source: §5 · p. 15

    Its textural realism objective assumes local texture statistics remain consistent across views.

    Source: §3.3 · p. 8

Abstract

from arXiv · show

Monocular novel-view synthesis has long required multi-view image pairs for supervision, limiting training data scale and diversity. We argue it is not necessary: one view is enough. We present OVIE, trained entirely on unpaired internet images. We leverage a monocular depth estimator as a geometric scaffold at training time: we lift a source image into 3D, apply a sampled camera transformation, and project to obtain a pseudo-target view. To handle disocclusions, we introduce a masked training formulation that restricts geometric, perceptual, and textural losses to valid regions, enabling training on 30 million uncurated images. At inference, OVIE is geometry-free, requiring no depth estimator or 3D representation. Trained exclusively on in-the-wild images, OVIE outperforms prior methods in a zero-shot setting, while being 600x faster than the second-best baseline. Code and models are publicly available at https://github.com/AdrienRR/ovie.

1 Introduction

OVIE trains monocular novel-view synthesis from single in-the-wild images by using depth-based pseudo-views as training targets, then generates images directly from a source image and target camera pose. It generalizes to unseen benchmarks and runs at over 100 FPS.

Motivation

  • Monocular novel-view synthesis could support spatial reasoning and virtual camera exploration, but multi-view datasets are scarce and narrow, while synthetic and video data introduce domain-gap or dynamic-scene problems.Existing models generalize within their training domains but fail elsewhere.Source: §1 · p. 2

Training paradigm

  • Depth-estimated point clouds transformed to sampled camera poses create partial pseudo-novel views, turning any single image into a source-target training pair without multi-view capture.Pseudo-targets are faithful in visible regions but sparse where geometry is occluded or disoccluded.Source: §1 · p. 2

Model and losses

  • OVIE maps only a source image and target pose directly to pixels, with observed-region reconstruction, masked perceptual supervision, and PatchGAN adversarial loss for realistic unseen-region textures.At inference, it needs no depth estimator, 3D representation, or per-scene optimization.Source: §1 · pp. 2–3

Results

  • On unseen RealEstate10K, OVIE rivals or outperforms in-domain geometry-free monocular methods; on DL3DV, it surpasses all baselines in a more balanced out-of-domain comparison.The model was trained without multi-view supervision on 30 million in-the-wild images.Source: §1 · p. 3

Data and scale

  • Training on single-image collections removes multi-view data requirements; analysis finds data scale matters more than diversity, while broader coverage yields marginal gains and metric depth provides metric-scale awareness.The training paradigm uses pseudo-views from monocular depth estimation and metric translations, avoiding scale-calibrated supervision at test time.Source: §1 · p. 3

2 Related Work

OVIE is positioned as feed-forward, scene-level monocular view synthesis trained without posed multi-view data, using metric depth for training-time pseudo-pairs but not at inference.

Problem setting

  • OVIE targets scene-level novel-view synthesis from one arbitrary-scene image and a target pose, generalizing across scenes and domains in one feed-forward pass.This distinguishes it from per-scene optimization, multi-view feed-forward methods, and largely object-centric single-image reconstruction.Source: §2 · pp. 3–4

Paired multi-view supervision

  • Prior paired-data methods learn from posed multi-view collections, while OVIE trains on 30 million unconstrained single images without multi-view data or posed images.The cited multi-view benchmarks cover a narrow visual domain that OVIE's training images reach beyond.Source: §2 · p. 4

Unpaired NVS

  • Unposed 3D-aware generators require slow, per-image GAN inversion and remain category-limited; Infinite Nature-Zero is closer, but retains depth warping at inference and domain-specific training.OVIE instead trains on unconstrained single images without a category prior, canonical pose distribution, or test-time optimization.Source: §2 · p. 4

Monocular depth estimation

  • Metric depth, unlike affine-invariant relative depth, recovers absolute scale, enabling OVIE to construct training pseudo-pairs with true metric translations without manual annotation.The passage contrasts relative estimators such as MiDaS, DPT, Depth Anything, Marigold, and MoGe with metric estimators such as ZoeDepth, UniDepth, and MoGe-2.Source: §2 · p. 5

Warping-based NVS

  • Unlike warping-based methods that depend on test-time depth and inherit its failures, OVIE requires no depth estimate at inference.Prior approaches warp explicit 3D representations and inpaint disocclusions; Infinite Nature uses posed videos, while MultiDiff's output quality depends on depth accuracy.Source: §2 · p. 5

3 Method

OVIE trains a pose-conditional novel-view model on unpaired images using depth-based pseudo-targets and masked supervision. At inference, it uses only a source image and target pose, with no depth estimator or 3D representation.

3.1 Overview

  • At inference, the depth estimator and projection pipeline are discarded; the model predicts a novel view from a source image and target pose in one forward pass.It requires no 3D data structures, point clouds, or warped inputs.Source: §3.1 · p. 6

3.2 Annotation-Free Training Pair Construction

  • A pretrained depth network lifts each image into a point cloud; sampled camera poses produce sparse pseudo-target views and binary masks that mark valid reprojected pixels.When MoGE-2 provides metric-scale depth, the generated pairs also supervise real-world scale changes.Source: §3.2 · pp. 6–7

3.3 Training Objective

  • Masked reconstruction, LPIPS, and patch-level P-DINO losses supervise valid regions; MSE yielded more stable convergence and better high-frequency detail than MAE or Charbonnier in experiments.Masking the prediction and pseudo-target before perceptual feature extraction suppresses invalid gradients from unobserved regions.Source: §3.3 · p. 7
  • A PatchGAN adversarial loss sharpens high-frequency detail, using source images as real samples because incomplete pseudo-targets are unreliable and local texture statistics are assumed consistent across views.The total objective combines reconstruction, perceptual, and adversarial terms.Source: §3.3 · pp. 7–8

3.4 Model Architecture

  • A convolutional encoder-decoder surrounds pose-conditioned Transformer blocks; the relative pose is encoded as translation and a unit quaternion, then applied through AdaLN.The pose representation omits camera intrinsics, which are rarely available at deployment.Source: §3.4 · p. 8

4 Experiments

OVIE is evaluated on qualitative generalization, benchmark comparisons, loss and data ablations, and inference throughput. The experiments test whether training on unpaired internet images yields robust novel-view synthesis and practical interactive performance.

4.1 Experimental Setup

  • OVIE trains on 30 million unpaired internet images, using MoGe-2 metric-depth predictions to generate pseudo-pairs without ground-truth multi-view pairs, depth annotations, or camera poses.No part of the training sets overlaps with the evaluation benchmarks.Source: §4.1 · pp. 8–9

4.2 Qualitative Results

  • OVIE generalizes across indoor, outdoor, object-centric, and painted imagery, producing geometrically consistent views, accurate camera-following, convincing inpainting, and correctly scaled parallax.Its metric scale awareness emerges from training on metric pseudo-pairs and requires no test-time scale calibration.Source: §4.2 · p. 11

4.3 Comparison with State of the Art

  • On DL3DV, OVIE outperforms all baselines on every metric; on RealEstate10K, it beats two of three in-domain baselines and remains competitive with VIVID.RealEstate10K gives the baselines an in-domain advantage, whereas DL3DV is out-of-domain for all methods.Source: §4.3 · p. 11, Table 1 · p. 12

4.4 Ablation Studies

  • Loss ablations show that learned perceptual and adversarial terms improve perceptual quality, while L2 reconstruction outperforms robust alternatives across nearly all reported metrics.Removing all learned losses raises PSNR and SSIM but degrades LPIPS and FID; removing P-DINO, LPIPS, or GAN worsens FID, and removing GAN particularly harms DL3DV. L2 beats L1 and Charbonnier on all RealEstate10K metrics and most DL3DV metrics; Charbonnier leads only on DL3DV FID by 0.2 points.Source: §4.4 · pp. 11–12

4.5 Data Scaling and Diversity

  • Performance improves consistently as training data scales from 3K to 30M images, and mixing diverse sources modestly improves or matches the best single-source results at a fixed 2M-image budget.The mixed model reaches FID 7.08 on RealEstate10K and 14.2 on DL3DV; scale contributes larger gains than changing composition, while diversity adds a complementary benefit.Source: §4.5 · pp. 12–13, Figure 5 · p. 13, Table 3 · p. 14

4.6 Towards an Interactive Navigation Model

  • 116 FPS (8.6 ms) lets OVIE generate views in a single forward pass, over 600× faster than the next-best method while exceeding its perceptual quality.On one H100 GPU, VIVID runs at 0.19 FPS, GeoGPT at 0.17 FPS, and PhotoNVS at 0.024 FPS; keyboard-driven camera controls enable interactive scene exploration.Source: §4.6 · p. 14, Figure 6 · p. 14

5 Conclusion · A Summary of Supplementary Material · B Additional Quantitative Results

OVIE uses monocular depth to generate pseudo-pairs from 30 million unlabeled images, matching or exceeding models trained on specialized multi-view benchmarks. The supplement adds quantitative analyses and technical and qualitative material.

5 Conclusion

  • Using pseudo-pairs from 30 million unlabeled images, OVIE matches or exceeds models trained on specialized multi-view benchmarks.The conclusion presents monocular depth estimation as a scalable, domain-agnostic source of geometric supervision.Source: §5 · p. 15
  • The paper identifies data scale, rather than architectural complexity, as the primary driver of view-synthesis performance and points to applications where multi-view capture is impossible.It also states that the framework supports real-time inference and can enable 3D-aware applications in historical archives and artwork.Source: §5 · p. 15

B Additional Quantitative Results

A Summary of Supplementary Material

  • Supplementary technical material details camera sampling, geometry-aware reprojection, pseudo-view hyperparameters, architecture, pose conditioning, and training.The document identifies these topics across Sections C and D.Source: A Summary of Supplementary Material · p. 20
  • Additional qualitative results cover out-of-distribution inputs including paintings, comparisons to pseudo-ground-truth targets, and RealEstate10K comparisons with ground truth and baselines.The supplement also provides animated navigations and interactive recordings showing continuous generation from one initial frame.Source: A Summary of Supplementary Material · p. 20
  • The supplement compares OVIE with InfiniteNature-Zero [30], noting that it shares single-view training but requires geometry-based depth estimation and sky segmentation at inference.That inference requirement is why InfiniteNature-Zero was omitted from the paper’s baselines.Source: A Summary of Supplementary Material · p. 20

B.1 Effect of Dataset Size on SSIM and LPIPS

SSIM and LPIPS both improve consistently as training dataset size increases, extending the data-scaling trends observed for PSNR and FID.

B.1 Effect of Dataset Size on SSIM and LPIPS

  • SSIM and LPIPS both improve consistently with more training data, as shown in Figure 7.The trends confirm those observed for PSNR and FID in Figure 5 of the main paper.Source: §B.1 · p. 20

B.2 Throughput analysis · C Camera Sampling Details

OVIE is reported to be 600× faster than the next-fastest method while improving SSIM and LPIPS on DL3DV; the appendix also details metric-scale camera sampling for training pseudotargets.

B.2 Throughput analysis

  • 600× faster than the next-fastest method, OVIE also achieves improved SSIM and LPIPS on DL3DV.Figure 8 complements the main paper’s improved PSNR and FID results in Figure 6.Source: §B.2 · p. 20, Figure 8 · p. 21
  • SSIM and LPIPS on RealEstate10K [80] improve monotonically as dataset volume increases.Source: Figure 7 · p. 21

C Camera Sampling Details

  • Training pseudotargets are generated by reprojecting the input scene’s 3D point cloud into newly sampled camera viewpoints.Source: C Camera Sampling Details · p. 21
  • A monocular depth estimator predicts absolute depth and horizontal field of view, enabling reconstruction of the point cloud at true metric scale.Source: C Camera Sampling Details · p. 21
  • Metric-scale geometry lets camera-pose sampling use scene units such as meters, so displacements and distances scale consistently with each scene.Source: C Camera Sampling Details · p. 21
  • A routing module stochastically assigns each batch element to one of six sampling methods using fixed prior weights, and each method outputs a world-to-camera extrinsic matrix [R | t] ∈R3×4.Source: C Camera Sampling Details · p. 21
  • The section denotes continuous uniform and normal distributions as U and N, and defines ˆy = [0, 1, 0]⊤ and ˆz = [0, 0, 1]⊤ as canonical up and forward unit vectors.Source: C Camera Sampling Details · p. 22

C.1 Sampling methods. … D Implementation Details

The appendix specifies how plausible target viewpoints are sampled and constructed, then describes depth-based reprojection and visibility handling. Its hyperparameters and model settings are summarized in Table 4.

C.1 Sampling methods.

  • Identity and point-cloud-scaled translation are among the six viewpoint strategies, with translation depth clamped to keep points from passing behind the camera.The translation has no rotation and is sampled using the point cloud’s per-axis standard deviation.Source: §C.1 · p. 22
  • Pure rotation samples a forward direction within a field-of-view bound, while the hybrid strategy independently samples and composes rotation and translation.Pure rotation fixes translation to zero; the hybrid composes the rotation with the sampled translation.Source: §C.1 · p. 22
  • Normal-derived sampling places the camera along an estimated surface normal; frontal-hemisphere sampling perturbs an anchor and limits angular deviation.Normal-derived batches without a valid normal fall back to identity; both methods use log-uniform distance sampling.Source: §C.1 · pp. 22–23

C.2 Look-at construction.

  • The look-at construction derives forward, right, and up camera axes from the camera position and target, then forms the rotation matrix from those axes.The right axis is the normalized cross product of the canonical up vector and forward vector; the true up axis is their cross product.Source: §C.2 · p. 23

C.3 Geometry-aware reprojection.

  • Geometry-aware reprojection maps each depth-estimated 3D source point into target-image coordinates using the target camera pose and intrinsic matrix.The projection is qi ∼ K(Rpi + t).Source: §C.3 · p. 23
  • Reprojection removes points facing away from the camera and uses a z-buffer to keep the nearest projected point at each pixel; unfilled pixels remain black.Points are culled when their normal dotted with the direction to the target camera is nonpositive.Source: §C.3 · p. 23
  • A visibility mask marks populated valid pixels for use in perceptual-loss computation.Source: §C.3 · p. 23

C.4 Hyperparameters.

  • Table 4 summarizes the sampling hyperparameters and model settings.Source: §C.4 · p. 23

D.1 Architecture · D.2 Camera Conditioning via AdaLN

OVIE combines a convolutional encoder–decoder with a ViT-B bottleneck, while camera extrinsics condition transformer blocks through adaLN-Zero.

D.1 Architecture

  • OVIE uses an 8×-downsampling convolutional encoder, a ViT-B bottleneck, and a symmetric convolutional decoder.The input images are 256×256.Source: §D.1 · p. 23
  • Three ResNet stages with GroupNorm and SiLU compress each 256×256 input into a 32×32×512 feature map.Source: §D.1 · p. 23
  • The feature map is patchified into 1,024 1×1-patch tokens and linearly projected to 768 dimensions.Source: §D.1 · p. 23
  • A 12-layer ViT-B with 12 heads, RMSNorm, and SwiGLU processes the tokens before symmetric decoding ends in a 1×1 convolution and Sigmoid.Source: §D.1 · p. 23

D.2 Camera Conditioning via AdaLN

  • Camera extrinsics p ∈ R^7, comprising translation and a rotation quaternion, are linearly projected to a conditioning embedding that predicts dimension-wise scale, shift, and residual gates for MSA and SwiGLU in each block.The embedding conditions the ViT-B bottleneck via adaLN-Zero.Source: §D.2 · p. 24
  • The conditioning MLP’s final linear layer is zero-initialized, so the conditioning path contributes nothing at the start of training.Source: §D.2 · p. 24

D.3 Optimization and Training · E Additional Qualitative Results · E.1 Out-of-distribution novel views

D.3 describes OVIE’s optimization and training losses, including its adversarial-training safeguards; E.1 shows that it synthesizes novel views from out-of-domain artistic images such as paintings.

D.3 Optimization and Training

  • OVIE trains for 2,000,000 steps with batch size 512, using AdamW, gradient clipping, cosine learning-rate decay, warmup, and EMA generator weights.The peak and minimum learning rates are 2 × 10−4 and 2 × 10−5; warmup lasts about 12.5k steps, and EMA decay is 0.999.Source: §D.3 · p. 24
  • The training objective combines L2 reconstruction, LPIPS, and P-DINO perceptual loss from a pretrained DINOv3-ViT-B/16 model.The LPIPS and P-DINO weights are 1.0 and 0.5, respectively.Source: §D.3 · p. 25
  • A PatchGAN discriminator sharpens high-frequency detail, using a frozen DINO-S/8 backbone instead of DINO-S/16 to reduce adversarial patch artifacts.Its convolutional head uses kernel size 9 and spectral normalization; discriminator and generator updates use hinge and non-saturating losses, respectively.Source: §D.3 · p. 25
  • Adversarial gradients are dynamically weighted against reconstruction gradients, with the adaptive weight capped at 10,000 and the adversarial penalty scaled by 0.75.The adaptive weight uses gradients at the decoder’s last convolutional layer, and ϵ = 10−6 stabilizes its computation.Source: §D.3 · p. 25
  • Discriminator updates and adversarial penalties are delayed until 37.5% and 40% of training, respectively; ablations and supplementary experiments use 250,000 steps with warmup duration unchanged.These delays are intended to prevent early collapse.Source: §D.3 · p. 25

E.1 Out-of-distribution novel views

  • OVIE synthesizes novel views from out-of-domain, non-realistic source images such as paintings, a setting standard multi-view-dependent methods cannot train on feasibly.Figure 9 illustrates these results.Source: §E.1 · p. 25

E.2 Comparison between training pseudo-targets and generated views … E.5 Real-time interactive navigation clips

The supplementary sections show OVIE’s pseudo-targets and generated views, qualitative baseline comparisons, trajectory clips, and real-time interactive navigation. They also explain the pseudo-target grid pattern and describe how navigation is controlled from the initial image.

E.2 Comparison between training pseudo-targets and generated views

  • Figures 10–13 show training pseudo-targets alongside views generated from the source image at the target camera pose.Source: §E.2 · p. 25
  • The pseudo-target grids arise because back-projected source pixels retain a regular layout whose spacing becomes visible at novel viewpoints, varying with depth and angle.Source: §E.2 · p. 26

E.3 Comparison to baseline methods

  • Figures 14–16 qualitatively compare GeoGPT, PhotoNVS, VIVID, and OVIE on RealEstate10K.Source: §E.3 · p. 26

E.4 Side-by-side navigation clips

  • Supplementary GIFs show RealEstate10K navigation sequences, with ground truth beside views generated using only the sequence’s first image.Source: §E.4 · p. 26

E.5 Real-time interactive navigation clips

  • Supplementary MP4 screen recordings showcase OVIE’s real-time interactive navigation capabilities.Source: §E.5 · p. 26
  • Mouse and keyboard inputs produce incremental camera position and rotation changes; OVIE updates camera extrinsics and generates images on the fly from the initial source image alone.Source: §E.5 · p. 27

F Additional Comparison to InfiniteNature-Zero

InfiniteNature-Zero is a relevant single-view-training comparison, but it uses geometry at inference and, despite an evaluation setup favorable to it, has weaker perceptual results than OVIE on both benchmarks.

F Additional Comparison to InfiniteNature-Zero

  • InfiniteNature-Zero learns perpetual view generation from single-view image collections, but its render-refine-repeat pipeline relies on monocular depth estimation and sky segmentation at inference.Like OVIE, it requires no posed multi-view training data, making it conceptually relevant despite its geometry-dependent approach.Source: F Additional Comparison to… · p. 27
  • For evaluation, the public InfiniteNature-Zero checkpoint generates each target independently from the source rather than autoregressively, accessing the ground-truth source and its depth at every step.The passage describes this as an inherent advantage over OVIE and other geometry-free methods.Source: F Additional Comparison to… · p. 27
  • On RealEstate10K, InfiniteNature-Zero’s LPIPS/FID are 0.395/28.8 versus OVIE’s 0.279/6.74; on DL3DV, they are 0.472/45.7 versus 0.464/13.6, respectively.InfiniteNature-Zero has the highest FID among compared methods on both benchmarks. Figure 17 shows its outputs are blurrier; on RealEstate10K, its PSNR/SSIM are 19.4/0.642 versus OVIE’s 18.8/0.602, which the passage attributes to these metrics’ insensitivity to perceptual sharpness.Source: F Additional Comparison to… · p. 27, Figure 17 · p. 36

Related papers

Loading 2603.23488v2…