PaperlayerView exact revision on arXiv

Source-linked AI summary

One View Is Enough! Monocular Training for In-the-Wild Novel View Generation

Adrien Ramanana Rahary, Nicolas Dufour, Patrick Perez, David Picard

arXiv:2603.23488v2cs.CV

TL;DR

Monocular novel-view synthesis is constrained by scarce multi-view supervision, motivating OVIE’s use of monocular depth to create pseudo-pairs from single internet images. Trained on 30 million in-the-wild images, OVIE generalizes strongly out of domain and supports real-time, geometry-free inference.

  • Problem

    Monocular novel-view synthesis relies on scarce posed multi-view datasets that cover only a narrow portion of the visual world.

  • Method

    OVIE uses monocular depth to lift a source image into 3D, reproject it under sampled camera transformations into pseudo-novel views, and train with masked losses on valid regions.

  • Results

    OVIE trained on 30 million unpaired in-the-wild images matches or exceeds models trained on specialized multi-view benchmarks and achieves over 100 FPS inference.

  • Takeaways & Limitations

    Single-image collections can provide scalable geometric supervision for 3D-aware view synthesis, including domains where multi-view capture is impossible and real-time applications.

  • Takeaways & Limitations

    The training approach assumes local texture statistics remain consistent across views and relies on monocular depth estimates to generate pseudo-pairs.

Abstract

from arXiv · show

Monocular novel-view synthesis has long required multi-view image pairs for supervision, limiting training data scale and diversity. We argue it is not necessary: one view is enough. We present OVIE, trained entirely on unpaired internet images. We leverage a monocular depth estimator as a geometric scaffold at training time: we lift a source image into 3D, apply a sampled camera transformation, and project to obtain a pseudo-target view. To handle disocclusions, we introduce a masked training formulation that restricts geometric, perceptual, and textural losses to valid regions, enabling training on 30 million uncurated images. At inference, OVIE is geometry-free, requiring no depth estimator or 3D representation. Trained exclusively on in-the-wild images, OVIE outperforms prior methods in a zero-shot setting, while being 600x faster than the second-best baseline. Code and models are publicly available at https://github.com/AdrienRR/ovie.

1 Introduction

OVIE addresses monocular novel-view synthesis without multi-view supervision by creating pseudo-pairs from individual images using monocular depth. Trained on 30 million in-the-wild images, it generalizes strongly while retaining efficient geometry-free inference.

  • Multi-view supervision is scarce and domain-limited, causing models trained on available datasets to fail outside their training domains.Purpose-built benchmarks cover a narrow visual slice, while synthetic and video data introduce domain gaps or violate static-scene assumptions.
  • Monocular depth converts each single image into a pseudo-novel-view training pair by lifting it into 3D, transforming it, and reprojecting it.The resulting target is partial but faithful in non-occluded regions, eliminating the need for multi-view capture.
  • Masked reconstruction, perceptual, and texture losses restrict supervision to valid pseudo-target regions, allowing training despite disocclusions.The method applies masking throughout its loss formulation and adds adversarial texture synthesis for unobserved regions.
  • 30 million in-the-wild images support strong zero-shot generalization: OVIE is competitive on RealEstate10K and surpasses all baselines on DL3DV.Both benchmarks were unseen during training; on RealEstate10K, OVIE rivals or outperforms in-domain geometry-free monocular methods.
  • Experiments indicate that data scale matters more than diversity, while broader coverage can provide marginal gains.The contribution statement presents pseudo-novel views as a scalable, domain-agnostic training paradigm.
  • Over 100 FPS makes the geometry-free model more than 600× faster than the next fastest baseline.The feed-forward design maps a source image and target camera pose directly to a synthesized image without an intermediate 3D representation.

2 Related Work

Prior monocular novel-view methods either rely on posed multi-view data, learn category-restricted 3D generative representations, or use expensive large generative models. OVIE instead remains geometry-free at inference while using metric depth only as an offline training scaffold.

  • Per-scene methods optimize representations for one scene, whereas monocular scene-level NVS seeks one-pass generalization across arbitrary scenes and domains.The target setting takes a single image and target camera pose as input.
  • The dominant feed-forward paradigm requires posed, static multi-view collections to learn geometric correspondences.Geometry-free methods synthesize directly in pixel space but share this training-data bottleneck.
  • Unposed 3D-aware generators learn implicit, tri-plane, or mesh representations, but applying them to real inputs requires slow, per-image GAN inversion.Their use is also confined to the category distribution represented in training data.
  • Large generative-prior methods incur expensive iterative inference and still require posed multi-view fine-tuning for geometric consistency.That fine-tuning reintroduces domain restrictions.
  • Metric depth provides absolute scale, enabling pseudo-pairs with true metric translations without manual annotation.Relative-depth methods instead produce affine-invariant predictions.
  • OVIE belongs to geometry-free monocular NVS but avoids posed multi-view training by constructing pseudo-pairs with metric depth.The distinction is between training-time geometric scaffolding and inference-time representation.

3 Method

OVIE constructs partial pseudo-targets from unpaired images by lifting them with frozen monocular depth, sampling camera transformations, and reprojecting point clouds. A pose-conditioned pixel-space model learns from masked geometric and perceptual losses plus adversarial texture supervision, then discards the geometric scaffold at inference.

  • The framework generates training pairs on the fly from unpaired image collections using a frozen monocular depth estimator.This enables pose-conditional image-to-image learning from pseudo-supervision rather than captured pairs.
  • Given a source image and relative camera transformation, the model predicts the corresponding novel view directly in pixel space.The objective is to match the true appearance from the target viewpoint.
  • Annotation-Free Training Pair Construction: The training pipeline lifts each image into a point cloud, samples a plausible pose, and reprojects it into a sparse pseudo-target with a validity mask.The mask identifies valid pixels and excludes disocclusions, occlusion boundaries, backface-culled regions, and out-of-frame content.
  • At inference, OVIE needs only a source image and target pose, with no depth estimator, point cloud, warped input, or per-scene optimization.The model therefore reduces novel-view synthesis to one forward pass.
  • Annotation-Free Training Pair Construction: Metric depth supplies pseudo-pairs with true metric camera changes, providing scale-grounded geometric supervision.OVIE uses metric-scale estimates to express sampled translations in real-world units.
  • Training Objective: All loss terms account for missing pseudo-target content through masking, while MSE reconstruction was found more stable and better at preserving high-frequency detail.Masked perceptual losses operate on both prediction and target before feature extraction.
  • Training Objective: The objective combines reconstruction, perceptual, and adversarial terms to enforce geometric fidelity, semantic coherence, and perceptual realism.LPIPS and P-DINO provide feature-based supervision, while PatchGAN sharpens high-frequency detail using the source as a real-image proxy.
  • A convolutional encoder-decoder with pose-conditioned Transformer blocks maps source features and a 7D pose vector to the synthesized view.The pose vector contains 3D translation and a unit quaternion, and AdaLN injects it into Transformer blocks.

4 Experiments

OVIE is evaluated for qualitative generalization, benchmark performance, ablations, data scaling, and inference throughput. Across these experiments, it combines strong out-of-domain results with efficient single-pass generation and supports broad scaling benefits from unpaired internet data.

  • 4.2 Qualitative Results: OVIE demonstrates generalization across indoor, outdoor, object-centric, and non-photographic imagery, including paintings that lack true multi-view supervision.These examples cover distributions substantially different from typical novel-view synthesis data.
  • 4.2 Qualitative Results: OVIE produces sharp, geometrically consistent novel views, follows target camera poses, and convincingly inpaints unobserved regions.Qualitative comparisons report accurate parallax and pose adherence, while concurrent methods may ignore target poses or generate inconsistent geometry.
  • 4.3 Comparison with State of the Art: On DL3DV, OVIE outperforms all baselines on every metric in the fair out-of-domain comparison.All compared methods face the same domain shift, making this the more balanced benchmark setting.
  • 4.4 Ablation Studies: Removing learned losses raises PSNR and SSIM but degrades LPIPS and FID, showing that pixel-level metrics alone can favor blurry predictions.Without learned losses, PSNR/SSIM reach 19.6 dB / 0.627 on RealEstate10K and 15.7 dB / 0.441 on DL3DV, while FID reaches 62.1 / 111.0.
  • 4.4 Ablation Studies: 34.2 points: removing the adversarial loss increases DL3DV FID by 34.2, versus 6.28 points on RealEstate10K.The ablation suggests adversarial training is especially important for out-of-domain generalization.
  • 4.5 Data Scaling and Diversity: Training-scale experiments show consistent improvement as the dataset grows, with scale contributing more than changing dataset composition at fixed scale.The experiments span 3K to 30M images; diverse mixing provides a complementary gain.
  • 4.6 Towards an Interactive Navigation Model: 116 FPS: OVIE runs at 116 FPS on DL3DV, over 600× faster than the next-best approach while exceeding its perceptual quality.The single forward pass takes 8.6 ms, compared with 0.19 FPS for VIVID, the next fastest listed method.

5 Conclusion

OVIE addresses the scarcity of multi-view supervision by using monocular depth to train on internet-scale unpaired images. It combines strong generalization with real-time inference, supporting 3D-aware applications from single images.

  • 30 million unlabeled images enable OVIE to match or exceed models trained on specialized multi-view benchmarks.
  • Data scale, rather than architectural complexity, is identified as the primary driver of view-synthesis performance.
  • OVIE’s framework targets domains where multi-view capture is impossible, including historical archives and artwork.
  • The supplementary document adds quantitative, camera-sampling, implementation, and qualitative analyses, including out-of-distribution images and RealEstate10K comparisons.

B.1 Effect of Dataset Size on SSIM and LPIPS

On RealEstate10K, both SSIM and LPIPS improve monotonically as the training dataset grows.

  • Both SSIM and LPIPS improve consistently with increasing training-data volume.

B.2 Throughput analysis

OVIE offers a favorable quality-throughput tradeoff on DL3DV. Its SSIM and LPIPS improvements accompany substantially faster inference than competing methods.

  • 600× faster inference than the next fastest method accompanies improved SSIM and LPIPS scores on DL3DV.
  • Figure 8 uses FPS on the horizontal axis, with upper-right preferred for SSIM and lower-right preferred for LPIPS.
  • Bubble size represents parameter count in the quality-versus-inference comparison.
  • During training, monocular depth and estimated horizontal field of view reconstruct a metric-scale point cloud for pseudo-target generation.

C.1 Sampling methods.

OVIE samples diverse, plausible camera transformations and reprojects monocular-depth point clouds into target views. Visibility filtering and z-buffering produce valid partial targets for masked training.

  • Sampling methods: Six sampling strategies range from identity and pure translation to rotation, hybrid, normal-derived, and frontal-hemisphere transformations.
  • Sampling methods: Frontal-hemisphere sampling jitters an anchor point, perturbs viewing direction within a bounded angle, and places the camera at a log-sampled distance.
  • Sampling methods: A look-at rotation is formed from forward, right, and true-up vectors computed from the camera position and target point.
  • Geometry-aware reprojection: Reprojection maps each 3D point into image coordinates using the camera intrinsics and sampled rotation and translation.
  • Geometry-aware reprojection: Backface culling removes points facing away from the camera, while z-buffering keeps the minimum-depth point when projections collide.
  • Geometry-aware reprojection: A visibility mask marks populated pixels and is applied when computing perceptual losses.

D.1 Architecture

OVIE uses a convolutional encoder, ViT-B bottleneck, and symmetric convolutional decoder, with camera pose conditioning injected into transformer blocks through adaLN-Zero.

  • Architecture: OVIE compresses 256×256 input images into a 32×32×512 feature map, patchifies them into 1024 tokens, and processes them with a ViT-B bottleneck.The encoder uses three ResNet stages, while the bottleneck uses D=768 dimensions, 12 heads, RMSNorm, and SwiGLU.
  • Camera Conditioning via AdaLN: Camera extrinsics p ∈R7 are projected into a D-dimensional conditioning embedding through a single linear layer.The extrinsics contain translation and quaternion rotation parameters.
  • Camera Conditioning via AdaLN: The conditioning embedding modulates both MSA and SwiGLU sub-layers using dimension-wise scale, shift, and residual-gate parameters.These parameters are regressed by a two-layer MLP from the camera-conditioning embedding.
  • Camera Conditioning via AdaLN: The conditioning MLP’s final linear layer is zero-initialized, so the conditioning path contributes nothing at the start of training.

D.3 Optimization and Training

OVIE combines reconstruction, perceptual, and adversarial objectives during training, with delayed adversarial updates and safeguards against unstable gradients.

  • Optimization: Training uses 2,000,000 steps, global batch size 512, AdamW, gradient clipping at 1.0, and cosine learning-rate decay from 2 × 10−4 to 2 × 10−5.A linear warmup precedes decay for approximately 12.5k steps.
  • Loss: The objective combines L2 reconstruction, LPIPS, and P-DINO perceptual losses, with λLPIPS=1.0 and λP-DINO=0.5.P-DINO features are extracted from a pretrained DINOv3-ViT-B/16 model.
  • Adversarial Training: A PatchGAN discriminator is used to sharpen high-frequency detail and reduce adversarial patch artifacts through a frozen DINO-S/8 backbone.The discriminator uses batch normalization, a kernel-size-9 convolutional head, spectral normalization, hinge loss, and the generator uses a non-saturating loss.
  • Adversarial Training: λadv = 0.75 scales the generator’s adversarial penalty, while λmax = 10,000 limits the adaptive weight to prevent gradient explosion.The adaptive weight uses gradients of the decoder’s last convolutional layer.
  • Adversarial Training: Adversarial discriminator updates begin at 37.5% of training and adversarial penalties at 40%, preventing early collapse.
  • Out-of-Domain Evaluation: Artistic-domain examples are treated as out-of-domain results because training on such domains would be infeasible with standard multi-view-dependent methods.

E.2 Comparison between training pseudo-targets and generated views

During training, OVIE learns from depth-lifted pseudo-targets, while at inference it generates novel views from the source image and target pose; continuous navigation updates pose incrementally from the initial image.

  • Comparison between training pseudo-targets and generated views: Training pseudo-targets are created by depth-lifting source images to sampled poses, and generated views are rendered at the corresponding poses for comparison.
  • Comparison between training pseudo-targets and generated views: Pseudo-targets contain grid-like patterns because back-projected source pixels preserve the point cloud’s regular spatial structure.The visible spacing varies with depth and viewing angle.
  • Comparison between training pseudo-targets and generated views: Qualitative comparisons on RealEstate10K evaluate GeoGPT, PhotoNVS, VIVID, and OVIE.
  • Comparison between training pseudo-targets and generated views: Supplementary trajectory demonstrations compare ground-truth sequences with views generated using only the first image of each sequence.
  • Comparison between training pseudo-targets and generated views: Interactive navigation maps mouse and keyboard actions to incremental camera-position and rotation changes, continuously generating views from the initial source image.

F Additional Comparison to InfiniteNature-Zero

InfiniteNature-Zero is a geometry-dependent single-view baseline evaluated independently per target view; OVIE achieves better reported perceptual metrics and sharper qualitative outputs despite that baseline’s inference advantage.

  • Method: InfiniteNature-Zero uses a geometry-based render-refine-repeat pipeline with monocular depth estimation and sky segmentation at inference.
  • Evaluation Protocol: For fair evaluation, InfiniteNature-Zero generates each target view independently from the source frame, accessing the source image and depth estimate at every step.This gives it an inherent advantage over OVIE and other geometry-free methods.
  • Quantitative Comparison: 0.279 LPIPS and 6.74 FID on RealEstate10K are reported for OVIE, versus 0.395 LPIPS and 28.8 FID for InfiniteNature-Zero.
  • Quantitative Comparison: 0.464 LPIPS and 13.6 FID on DL3DV are reported for OVIE, versus 0.472 LPIPS and 45.7 FID for InfiniteNature-Zero.
  • Qualitative Comparison: InfiniteNature-Zero produces blurrier outputs than OVIE, while its PSNR and SSIM remain comparable to OVIE’s.
Loading 2603.23488v2…