PaperlayerView exact revision on arXiv

Source-linked AI summary

MoCam: Unified Novel View Synthesis via Structured Denoising Dynamics

Haofeng Liu, Yang Zhou, Ziheng Wang, Zhengbo Xu, Zhan Peng, Jie Ma, Jun Liang, Shengfeng He, Jing Li

arXiv:2605.12119v2cs.CVcs.GR

TL;DR

Novel-view synthesis relies on geometric priors that can become sparse and erroneous under large view changes, while appearance cues are misaligned with novel views. MoCam stages geometry before appearance in diffusion, and the authors report better results than static conditioning approaches, including with severely flawed point clouds.

  • Problem

    Existing view-synthesis methods rely on geometric priors that become sparse, incomplete, and erroneous under large view changes, while appearance cues are misaligned with novel views.

    Source: §1 · p. 2

  • Method

    MoCam structures diffusion denoising to use geometric priors early for alignment, then appearance priors later for refinement across static and dynamic view synthesis.

    Source: §1 · p. 3

  • Results

    MoCam significantly outperforms static conditioning approaches and produces geometrically coherent, photorealistic results even with point clouds containing severe holes or distortions.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The same generation principle applies to single-image 3D view synthesis and video 4D re-camera.

    Source: §1 · p. 3

    The rendered scaffold can contain disocclusion holes and geometric distortions, especially far from the original camera path, and is unsuitable as a final output.

    Source: §3.2 · p. 6

Abstract

from arXiv · show

Generative novel view synthesis faces a fundamental dilemma: geometric priors provide spatial alignment but become sparse and inaccurate under view changes, while appearance priors offer visual fidelity but lack geometric correspondence. Existing methods either propagate geometric errors throughout generation or suffer from signal conflicts when fusing both statically. We introduce MoCam, which employs structured denoising dynamics to orchestrate a coordinated progression from geometry to appearance within the diffusion process. MoCam first leverages geometric priors in early stages to anchor coarse structures and tolerate their incompleteness, then switches to appearance priors in later stages to actively correct geometric errors and refine details. This design naturally unifies static and dynamic view synthesis by temporally decoupling geometric alignment and appearance refinement within the diffusion process. Experiments demonstrate that MoCam significantly outperforms prior methods, particularly when point clouds contain severe holes or distortions, achieving robust geometry-appearance disentanglement.

1 Introduction

MoCam coordinates geometry and appearance through staged diffusion conditioning: rendered geometry anchors early structure, and source appearance refines later details. The same principle supports both single-image 3D view synthesis and video 4D re-camera.

1 Introduction

  • Under large view changes, reconstructed geometric priors become sparse, incomplete, or erroneous, and existing pipelines propagate those flaws or statically fuse geometry and appearance with degraded structure and texture.The introduction identifies depth maps and point clouds reconstructed from monocular inputs as vulnerable geometric priors.Source: §1 · p. 2
  • Geometric scaffolds align target trajectories but contain holes and distortions, while high-fidelity source appearance is misaligned with novel views, so simultaneous conditioning creates signal conflict.Early appearance cues can cause geometric drift, whereas late geometry can bake structural errors into the output.Source: §1 · pp. 2–3
  • MoCam conditions on rendered scaffolds early to establish global structure, then uses source appearance to correct geometric errors and fill disocclusions as denoising progresses.Early scaffold conditioning deliberately tolerates geometric incompleteness; later refinement uses stabilized geometry without destabilizing overall structure.Source: §1 · pp. 2–3
  • MoCam's stage-wise conditioning applies across input modalities, unifying single-image 3D view synthesis and video 4D re-camera through separate geometric alignment and appearance refinement.The approach targets the shared challenge of synthesizing views under unreliable geometry.Source: §1 · p. 3
  • Despite severe point-cloud holes or distortions, MoCam produces geometrically coherent, photorealistic results and significantly outperforms static conditioning approaches.The introduction describes this as robust geometry-appearance disentanglement and state-of-the-art robustness under sparse, inaccurate geometry.Source: §1 · p. 3

2 Related Works

Prior novel-view synthesis methods trade geometric alignment against visual fidelity, while MoCam uses stage-wise denoising guidance to separate geometry anchoring from appearance refinement. The related work motivates this approach through limitations in optimization-based reconstruction, video generation, and static diffusion conditioning.

Optimization-Based Novel View Synthesis

  • NeRF, 3DGS, and dynamic 4D-representation [39] and deformation-field methods require dense multi-view video and costly per-scene optimization.With monocular input, reconstruction quality and appearance fidelity degrade; MoCam instead uses generative video-model priors without per-scene optimization.Source: §2 · p. 3

Generative Novel View Synthesis

  • Video diffusion enables view sequences, but camera-conditioned generation [1] [2] [19] can lack geometric accuracy, while render-then-inpaint methods [16] [43] [44] inherit scaffold errors.Gen3C uses a spatiotemporal 3D cache and TrajectoryCrafter [44] adds Ref-DiT conditioning, yet sparse, inaccurate rendered geometry can bake errors into generation. MoCam decouples geometry and appearance over denoising to mitigate error propagation and improve stability under large camera motions.Source: §2 · p. 4

Conditioning Mechanisms in Diffusion Models

  • Static conditioning methods such as Control-Net, T2I-Adapter, and IP-Adapter [42] retain guidance across timesteps, whereas TSM [52] and DMP explore dynamic conditioning.MoCam hands off stage-wise between a geometrically aligned but flawed scaffold and a view-disaligned but visually rich reference video, matching each signal to the denoising stage where it is most effective.Source: §2 · p. 4

3 MoCam

MoCam generates novel views by first anchoring geometry with a rendered point-cloud scaffold, then switching to the source video to refine appearance and correct scaffold errors. Its structured denoising dynamics align these complementary conditions with different stages of generation.

3.1 Preliminary: Video Generative Models

  • The denoising timestep progresses from global structure to local detail, motivating MoCam’s stage-wise conditioning strategy.The method uses this progression to align different condition signals with different stages of generation.Source: §3.1 · p. 6

3.2 Scaffold Generation

  • Depth estimation and inverse perspective projection build a dynamic 3D point-cloud scaffold, rendered along the target camera trajectory as a spatially aligned but artifact-prone guide.Monocular input can leave disocclusion holes and geometric distortions, especially in views far from the original camera path.Source: §3.2 · p. 6

3.3 Stage-Wise Dual-Conditioning Diffusion

  • The scaffold supplies spatial geometry and motion, while the source video supplies high-fidelity appearance and object dynamics; their camera-motion signals can conflict.The scaffold condition is concatenated with initial noise along the frame dimension for spatially explicit guidance, and the source video is conditioned similarly to transfer content and texture.Source: §3.3 · p. 7
  • MoCam uses the scaffold when t > T_switch to anchor global geometry, then switches to the source when t ≤ T_switch to refine appearance and correct scaffold errors.This handover aims to prevent scaffold flaws from persisting into final high-fidelity synthesis.Source: §3.3 · pp. 7–8

4 Experiments

Across in-the-wild and multi-view evaluations, MoCam delivers strong perceptual quality alongside geometric consistency. Ablations support its stage-wise strategy: scaffold conditioning anchors early geometry, then reference conditioning corrects errors and refines appearance.

4.1 Evaluation on In-the-wild Benchmark

  • FVD-V 255.16 vs. 289.37, and MoCam achieves the lowest pose errors in single-view 3D reconstruction comparisons.The results support maintaining perceptual fidelity and precise camera control under the geometric ambiguity of single-view reconstruction.Source: Table 1 · p. 9, §4.1 · pp. 9–10
  • For single-view reconstruction, MoCam uses limited scaffold geometry to anchor plausible structure before refining appearance, producing coherent, detailed results despite sparse point clouds.GEN3C and TrajCrafter show structural distortions, while ReCamMaster fails to infer correct 3D layouts without explicit geometry.Source: §4.1 · p. 9
  • In 4D re-camera examples, MoCam preserves subject structure and renders high-fidelity textures under view changes, unlike scaffold-based methods with geometric degradation and ReCamMaster’s inconsistent, chaotic outputs.Examples include a distorted cat body during orbiting and a collapsed human arm; MoCam preserves the cat’s volume and the person’s limbs.Source: §4.1 · p. 10
  • MoCam leads on most 4D re-camera metrics, including background consistency, subject consistency, and imaging quality, while retaining the lowest rotation error and competitive translation error.The comparison covers diverse scenes and camera trajectories on the OpenVid dataset.Source: §4.1 · p. 10, Table 2 · p. 11

4.2 Evaluation on Multi-view Video Benchmark

  • On the iPhone multi-view benchmark, MoCam significantly outperforms other methods on PSNR, SSIM, LPIPS, and FVD.The reported gains in LPIPS and FVD are particularly noteworthy; qualitative examples also show finer clothing details and more accurate facial structure than blurry or distorted results from other methods.Source: Table 3 · p. 11, §4.2 · pp. 12–13

4.1 Evaluation on In-the-wild Benchmark

  • As orbit magnitude increases from 30 to 90 degrees, MoCam maintains consistently high scores while competing scaffold-based and implicit methods deteriorate through propagated errors or geometric drift.Early geometry anchoring prevents drift, and late appearance refinement corrects sparse scaffold errors; Fig. 7 shows MoCam retaining coherent subjects as degradation increases.Source: §4.1 · p. 11, Figure 6 · p. 11, Figure 7 · p. 12

4.3 Ablation Studies

  • MoCam significantly outperforms Scaffold-Only, Scaffold-Early, and Static-Both, showing that sequential conditioning is needed to anchor geometry and then correct scaffold flaws with reference appearance.Scaffold-Only has IQ 0.4807; Static-Both has Rotation Error 2.71 and Translation Error 11.01, while Scaffold-Early avoids the worst point-cloud artifacts but produces blurry or generic details without later reference conditioning.Source: §4.3 · pp. 13–14
  • MoCam produces comparable high-fidelity results on Wan2.1 and remains coherent with perturbed depth inputs, indicating robustness across backbones and reconstruction noise.The depth-error examples show inaccurate-scaffold distortions being corrected in the final output.Source: §4.3 · p. 14

5 Conclusion

MoCam unifies 3D/4D view synthesis by temporally separating early geometric scaffold anchoring from late appearance-based error correction, addressing sparse and erroneous geometry.

5 Conclusion

  • MoCam’s stage-wise denoising prevents point-cloud flaws from propagating while retaining geometric control and robustness to imperfect monocular reconstructions.The framework anchors a scaffold early and corrects errors at later stages.Source: §5 · p. 14
  • Future work may explore joint scaffold-video refinement.Source: §5 · p. 14

A Comparison with 3D-based method

MoCam outperforms ViewCrafter in single-view 3D reconstruction, with better visual quality and structure accuracy and fewer geometry distortions. The comparison is reported qualitatively in Fig. 11 and quantitatively in Table 4.

A Comparison with 3D-based method

Related papers

Loading 2605.12119v2…