PaperlayerView exact revision on arXiv

Source-linked AI summary

MetaView: Monocular Novel View Synthesis with Scale-Aware Implicit Geometry Priors

Yufei Cai, Xuesong Niu, Hao Lu, Kun Gai, Kai Wu, Guosheng Lin

arXiv:2607.12000v2cs.CV

TL;DR

Monocular novel view synthesis must balance geometric consistency, camera control, and generalization under large viewpoint changes. MetaView combines implicit geometry priors with minimal explicit 3D cues, outperforming existing methods across challenging datasets.

  • Problem

    Existing monocular novel view synthesis methods struggle to balance spatial consistency with generalization under large viewpoint changes.

  • Method

    MetaView combines implicit geometry priors with minimal explicit 3D cues, including scale cues, for diffusion-based synthesis without explicit 3D reconstruction.

  • Results

    MetaView consistently outperforms reconstruction-based and purely implicit methods, with superior generalization and geometric consistency under challenging large viewpoint changes.

  • Takeaways & Limitations

    MetaView provides precise viewpoint control and large-view generation while bridging explicit geometric constraints and implicit generative modeling.

  • Takeaways & Limitations

    MetaView can fail in complex out-of-domain or dynamic 4D scenes and may provide imprecise viewpoint control in expansive distant scenes.

Abstract

from arXiv · show

Current visual generation models are capable of producing high-quality content, yet they lack a coherent perception of the spatial structure. Existing generative novel view synthesis methods typically introduce explicit geometry priors, which enforce spatial consistency but inherently restrict generalization in large view changes. In contrast, recent interactive generative methods favor implicit scene modeling, offering greater flexibility at the cost of precise camera control and geometry consistency. In this paper, we propose MetaView, a diffusion-based monocular novel view synthesis framework that enables rendering under large view changes from a single image. Our key insight is to combine implicit geometry modeling with minimal yet essential explicit 3D cues: we incorporate implicit geometry priors from a feed-forward geometry perception network to regularize structure without imposing restrictive reconstruction pipelines, while leveraging metric depth to anchor the generation to a metric scale. This design allows MetaView to achieve both geometry consistency and precise controllability. Extensive experiments demonstrate that, under challenging monocular large viewpoint changes, MetaView significantly outperforms existing methods and exhibits superior generalization. Our code is publicly available at https://github.com/KlingAIResearch/MetaView.

1 Introduction

MetaView addresses the tension between spatial consistency and generalization in monocular novel view synthesis under large viewpoint changes. It combines implicit geometry priors with minimal explicit cues for camera control and scale-aware generation without explicit reconstruction.

  • Problem: Existing novel view synthesis methods struggle to balance consistency with generalization under large viewpoint changes.They use either explicit 3D representations or purely implicit spatial geometry learning.
  • Approach: MetaView enables precise camera control and high-fidelity synthesis without explicit 3D reconstruction.The framework moves beyond the dichotomy between reconstruction-based pipelines and purely implicit generation.
  • Approach: MetaView uses hierarchical features from a feed-forward geometry perception network as implicit priors to regularize relative scene geometry.It also encodes camera intrinsics and extrinsics with a modified RoPE and injects metric scale cues to address scale ambiguity.
  • Evaluation: The method introduces Dense Matching Distance to assess spatial alignment more accurately than low-level metrics.Experiments cover DL3DV, RealEstate10K, and Sekai-Real-Walking-HQ.
  • Results: MetaView significantly outperforms reconstruction-based and purely implicit methods and exhibits strong generalization across diverse scenarios.The reported experiments evaluate monocular novel view synthesis under extreme variations.

2 Related Works

Related work spans diffusion-based visual generation, implicit geometry estimation, and novel view synthesis. Existing NVS methods range from NeRF and 3DGS to diffusion-based approaches, which improve controllability or efficiency but retain limitations in generalization, geometric consistency, or 3D awareness.

  • Diffusion Models: Diffusion models learn complex high-dimensional distributions through iterative denoising, while latent diffusion performs denoising in lower-dimensional latent spaces with cross-attention conditioning.Latent diffusion balances generation efficiency and visual fidelity.
  • Geometry Estimation: Traditional geometry estimation uses handcrafted features and joint optimization, whereas DUSt3R predicts camera poses and depth maps through feed-forward implicit transformer modeling.The passage describes traditional systems as robust in general scenes but limited in robustness and scalability.
  • Novel View Synthesis: NeRF represents scenes with implicit neural networks and volume rendering, while 3D Gaussian Splatting improves reconstruction efficiency and rendering quality.NeRF marked a breakthrough in novel view synthesis, followed by work on rendering fidelity and optimization speed.
  • Diffusion-Based NVS: Diffusion-based NVS methods use diffusion priors, 2D coordinate warping, or explicit 3D caches, but face limited generalization, geometric consistency, or 3D awareness.ZeroNVS performs 3D distillation; other methods guide generation through conditioned inpainting.

3 Method

MetaView synthesizes a target view from a source image and relative camera pose by combining metric-scale explicit cues with implicit geometry priors in a pretrained MM-DiT diffusion model. It injects these signals through camera-aware RoPE and geometry-token attention while avoiding handcrafted 3D reconstruction operations.

  • Problem formulation: MetaView synthesizes a novel view Xgen from a single source image Xsrc under a target camera pose T defined relative to the source.The input condition includes the source image, camera intrinsics K, and relative extrinsics T=(R,t), whose 6-DoF parameterization specifies target position and orientation.
  • DiT backbone: The framework builds on pretrained Qwen-Image-Edit, a text-and-image-conditioned T2I model using the MM-DiT architecture and flow matching.Its image and text tokens are processed through stacked Multi-Modal Attention blocks with cross-modal self-attention.
  • Explicit 3D cues: Camera-aware RoPE encodes K and T, while metric depth z from DepthAnything3-Metric anchors generation to a consistent spatial scale.Metric-scale alignment is necessary because projection becomes ambiguous when world geometry and camera pose use mismatched scales.
  • Explicit 3D cues: MetaView injects sparse downsampled depth cues into latent-token attention without depth lifting or other handcrafted reconstruction operations.This reduces additional explicit 3D inductive biases while preserving an explicit measure of spatial scale.
  • Implicit geometry priors: Intermediate multi-level features from DepthAnything3 are projected into geometry tokens and concatenated with source and generated image tokens in self-attention.These features provide implicit relative geometry priors while remaining compatible with the DiT token representation.
  • Geometry-guided generation: Geometry tokens encode relative geometric priors, are anchored by downsampled metric depth, and guide Xgen conditioned on the relative transformation T.The resulting interaction enables geometry-aware generation while preserving the framework’s implicit scene modeling.

4 Experiments

Experiments evaluate MetaView across multiple datasets and challenging viewpoint changes, using DMD to assess extrapolative novel-view synthesis where conventional metrics are inadequate. Results show improved spatial awareness, camera and scale control, component effectiveness, and cross-domain generalization.

  • Evaluation Metrics: DMD measures average optical-flow displacement between matched points and is proposed for arbitrary viewpoint variations where PSNR, SSIM, and related metrics fail.Reported DMD values are normalized to 0∼100.
  • Main Results: MetaView produces coherent results under arbitrary viewpoint changes while maintaining precise camera-pose and scene-scale control, unlike implicit and reconstruction-based baselines.Implicit methods exhibit scale drifting, while reconstruction-based methods degrade through blurriness and distortions under large viewpoint variations.
  • Ablation Study: Ablating geometry tokens preserves approximate viewpoint orientation but harms fine-grained relative consistency, causing missing objects, contour discrepancies, and incorrect object arrangements.Ablations are conducted on DL3DV-medium, covering geometry tokens, the z-axis in RoPE, and the pretrained backbone.
  • Metric Analysis: DMD aligns with human preference and varies monotonically with increasing viewpoint changes, whereas PSNR, SSIM, and LPIPS can reward blurry artifacts or show nonmonotonic trends.This evaluation targets extrapolated regions requiring plausible completion rather than strict pixel alignment.
  • Generalization: Frozen pretrained-backbone parameters preserve semantic priors, enabling consistent novel-view synthesis across human, animal, cartoon, watercolor, and AI-generated scenes.These results demonstrate robustness across diverse out-of-domain visual styles.

5 Conclusion

MetaView is a diffusion-based monocular novel view synthesis framework that enables precise viewpoint control and large-view generation without explicit 3D reconstruction. It bridges explicit geometric constraints and implicit generative modeling to address structural inconsistency and scale drifting.

  • Conclusion: MetaView achieves precise viewpoint control and large-view generation without explicit 3D reconstruction.The framework is diffusion-based and designed for monocular novel view synthesis.
  • Conclusion: MetaView bridges explicit geometric constraints with implicit generative modeling.This design targets the complementary limitations of explicit and implicit approaches.
  • Conclusion: The framework addresses structural inconsistency and scale drifting by incorporating implicit geometry priors.These priors guide fine-grained structural synthesis.

Supplementary Material · A Implementation Details · A.1 Baseline Implementation

The supplementary implementation details describe the evaluation setups for ViewCrafter and HY-World, including their official models, video-generation configurations, camera-pose estimation, and interpolation procedures. All generated images are resized to 960×528 before evaluation to reduce resolution-related sensitivity in low-level metrics.

  • A.1 Baseline Implementation: ViewCrafter uses official open-source code and pre-trained weights for single-image 25-frame video generation at 1024×576.Camera poses and depth are estimated with VIPE [18].
  • A.1 Baseline Implementation: ViewCrafter obtains reconstruction-module camera poses and depth information through VIPE [18] estimation.The setup uses the single-image input setting.
  • A.1 Baseline Implementation: ViewCrafter applies the official pose interpolation method using the relative camera pose of the target viewpoint.This procedure is part of the official-code evaluation setup.
  • A.1 Baseline Implementation: HY-World uses its official distilled model to generate 77-frame videos at 832 × 480 with 4 inference steps.The input prompt is fixed to "A static scene".
  • A.1 Baseline Implementation: HY-World derives camera poses from VIPE [18] test-set estimates and applies uniform interpolation across the 77 video frames.The interpolation produces a continuous sequence of pose variations.
  • A.1 Baseline Implementation: 960×528 is the common evaluation resolution for all generated images.Images are resized before evaluation to ensure fair comparison and mitigate low-level metrics’ sensitivity to image resolution.

A.2 Data Curation · A.3 View Overlap Computation · B Effect of Geometry Tokens

The paper curates metrically aligned, filtered RGB data, samples views using depth-based overlap rather than frame intervals, and shows that geometry tokens receive strong attention during synthesis. These procedures target reliable geometry and difficulty-aware evaluation under large viewpoint changes.

  • A.2 Data Curation: RGB data come from DL3DV, RealEstate10K, and Sekai-Real-Walk-HQ, with VIPE and DepthAnything3-Metric aligning estimated depth scale to camera poses.All reported values are in meters.
  • A.2 Data Curation: The curation pipeline removes scenes with anomalous depth values caused by cuts, overexposure, or underexposure.Invalid depth includes NaN or Inf values.
  • A.2 Data Curation: Large viewpoint changes can cause incomplete reprojection, making some reconstruction-based methods fail to generate valid novel views.This behavior is illustrated by a zooming case in Fig. A.
  • A.3 View Overlap Computation: Training and testing sample view pairs by valid view overlap because monocular NVS becomes practically ill-posed when the source view provides no effective reference.Overlap is measured using depth-based reprojection of source pixels into the target image plane.
  • A.3 View Overlap Computation: During training, the maximum frame interval is 40 and source-target overlap must be at least 30%; DL3DV evaluation is divided into easy, medium, and hard subsets by overlap.A source pixel is valid when its reprojected depth and target depth differ by less than 10%.
  • A.3 View Overlap Computation: Continuous zooming can retain high overlap despite large frame intervals and pose variation, so such pairs are assigned to the easy subset.This explains why baseline methods can still perform sub-optimally on the easy set.
  • B Effect of Geometry Tokens: Across all layers, generation tokens Xgen allocate significantly higher attention to geometry token G than to other components.Fig. B visualizes attention weights averaged across all heads, with brighter regions indicating stronger attention.

C Limitations & Discussion

MetaView has limitations in complex out-of-domain scenes, expansive distant scenes, and dynamic environments. These limitations affect extrapolation, scale-based viewpoint control, and applicability beyond static settings.

  • Limitations & Discussion: MetaView may fail to extrapolate in complex out-of-domain scenarios when their scene semantics are absent from the foundation model’s pre-trained priors.The limitation is attributed to the capabilities and semantic coverage of the foundation model.
  • Limitations & Discussion: In expansive distant scenes, challenging scale estimation can cause imprecise viewpoint control.The issue arises where accurate scale estimation is inherently difficult.
  • Limitations & Discussion: MetaView is currently restricted to viewpoint transformations within static environments.The supplied passage ends mid-sentence after introducing this restriction.

D More Results

Additional qualitative visualizations show MetaView results on the DL3DV, RealEstate10K, and Sekai-Real-Walking-HQ test sets.

  • Additional qualitative results: MetaView is qualitatively evaluated on the DL3DV, RealEstate10K, and Sekai-Real-Walking-HQ test sets.The visualizations are presented in Fig. C, Fig. D, and Fig. E, respectively.
Loading 2607.12000v2…