Source-linked AI summary
MoCam: Unified Novel View Synthesis via Structured Denoising Dynamics
Haofeng Liu, Yang Zhou, Ziheng Wang, Zhengbo Xu, Zhan Peng, Jie Ma, Jun Liang, Shengfeng He, Jing Li
TL;DR
Novel view synthesis must reconcile spatially aligned but unreliable geometry with visually rich yet misaligned appearance signals. MoCam stages these signals through diffusion, anchoring geometry early and correcting it with appearance later, yielding robust results across static and dynamic view synthesis.
Problem
Novel view synthesis lacks a reliable way to combine sparse, distorted geometric priors with high-fidelity but geometrically misaligned appearance priors.
Method
MoCam temporally decouples conditioning by anchoring coarse geometry early in denoising, then using appearance signals later to correct errors and refine details.
Results
MoCam generates geometrically coherent and photorealistic results despite severe point-cloud holes or distortions, significantly outperforming static conditioning approaches.
Takeaways & Limitations
The same stage-wise generation principle supports both single-image 3D view synthesis and video 4D re-camera under unreliable geometry.
Takeaways & Limitations
MoCam relies on monocularly reconstructed scaffolds that can contain disocclusion holes and geometric distortions, especially for views far from the original camera path.
Abstract
from arXiv · showhide
Generative novel view synthesis faces a fundamental dilemma: geometric priors provide spatial alignment but become sparse and inaccurate under view changes, while appearance priors offer visual fidelity but lack geometric correspondence. Existing methods either propagate geometric errors throughout generation or suffer from signal conflicts when fusing both statically. We introduce MoCam, which employs structured denoising dynamics to orchestrate a coordinated progression from geometry to appearance within the diffusion process. MoCam first leverages geometric priors in early stages to anchor coarse structures and tolerate their incompleteness, then switches to appearance priors in later stages to actively correct geometric errors and refine details. This design naturally unifies static and dynamic view synthesis by temporally decoupling geometric alignment and appearance refinement within the diffusion process. Experiments demonstrate that MoCam significantly outperforms prior methods, particularly when point clouds contain severe holes or distortions, achieving robust geometry-appearance disentanglement.
1 Introduction
MoCam addresses the conflict between spatially aligned but incomplete geometric scaffolds and visually rich but misaligned appearance signals by temporally decoupling them through structured denoising. It unifies static and dynamic view synthesis while improving robustness to severe geometric holes and distortions.
- Motivation: Geometric scaffolds align target trajectories but suffer holes and distortions, whereas source images and videos provide high-fidelity appearance yet lack novel-view geometric alignment.The introduction identifies these complementary signals as difficult to combine simultaneously.
- Method: MoCam uses structured denoising dynamics to condition early timesteps on rendered scaffolds for coarse structural anchoring, then later stages on appearance for high-frequency refinement and error correction.This progression matches diffusion models’ distinct representational needs across denoising phases.
- Unified framework: The same stage-wise conditioning framework supports single-image 3D reconstruction and video 4D re-camera, independently of the input modality.MoCam first establishes geometry and then refines appearance within a unified generation principle.
- Results: MoCam achieves robust geometry-appearance disentanglement and generates geometrically coherent, photorealistic results despite severe point-cloud holes or distortions.The method significantly outperforms static conditioning approaches under these challenging geometric conditions.
- Results: Late-stage appearance conditioning enables active geometric error correction and state-of-the-art robustness under sparse and inaccurate geometry.The contribution frames this capability as advancing controllable view synthesis.
2 Related Works
Prior novel view synthesis methods reconstruct 3D or 4D representations from posed images, while newer approaches use pretrained image or video diffusion models for single-image and trajectory-based synthesis. Extending these methods to dynamic 4D re-camera introduces additional challenges involving temporal dynamics and viewpoint changes.
- Optimization-Based Novel View Synthesis: NeRF represents static scenes as continuous volumetric functions, enabling highly photorealistic novel view synthesis.3D Gaussian Splatting has more recently achieved comparable prominence in this area.
- Optimization-Based Novel View Synthesis: Single-view 3D reconstruction methods use pretrained image diffusion models to synthesize novel views from single images.These methods address view synthesis without requiring the full posed-image reconstruction pipeline described for classical approaches.
- Video and Dynamic View Synthesis: Video generative models support smooth camera trajectories by maintaining temporal consistency across synthesized views.ViewCrafter uses video diffusion to generate high-fidelity view sequences along camera paths.
- Video and Dynamic View Synthesis: Dynamic-scene 4D video re-camera is more complex because generation must jointly handle temporal dynamics and viewpoint changes.This challenge arises when extending trajectory-based view synthesis beyond static scenes.
3 MoCam
MoCam uses structured denoising dynamics to coordinate scaffold-based geometric alignment and source-based appearance refinement across generation stages. It anchors global structure early, then switches conditions to correct geometric inaccuracies and synthesize high-frequency details.
- Scaffold Construction: MoCam renders a dynamic point-cloud scaffold along the target camera trajectory to provide spatially aligned motion and structural guidance.The scaffold is derived from monocular depth and inverse perspective projection, but is intended as an initial guide rather than the final output.
- Dual Conditioning: The scaffold supplies geometry and motion, while the source video supplies high-fidelity appearance, textures, and object dynamics.The two conditions are complementary but can carry conflicting camera-movement signals when combined statically.
- Structured Denoising Dynamics: MoCam switches from scaffold conditioning when t > T_switch to source conditioning when t ≤ T_switch during denoising.This time-dependent context implements the proposed structured denoising dynamics.
- Stage-wise Generation: Early denoising anchors global structure and target motion, whereas later denoising refines appearance and corrects inherited geometric inaccuracies.The later source-conditioned stage can inpaint and correct scaffold flaws before decoding the clean target latent into the output video.
4 Experiments
Across single-view 3D reconstruction, 4D re-camera, and iPhone evaluation, MoCam delivers strong perceptual quality, pose accuracy, and geometric-appearance fidelity. Qualitative and ablation results show that structured denoising preserves coherent geometry while refining appearance and correcting scaffold errors.
- 3D Reconstruction Qualitative Results: MoCam anchors plausible geometry from sparse scaffolds before refining appearance, producing coherent and detailed single-view reconstructions despite extreme point-cloud sparsity.GEN3C and TrajCrafter exhibit structural distortions, while ReCamMaster fails to infer correct 3D layouts without explicit geometry.
- 3D Reconstruction Quantitative Results: 255.16 vs. 289.37: MoCam achieves lower FVD-V than competitors and the lowest pose errors in single-view 3D reconstruction.The results support simultaneous perceptual fidelity and precise camera control under geometric ambiguity.
- 4D Re-Camera Results: MoCam achieves the highest scores across most 4D re-camera metrics, notably background consistency, subject consistency, and imaging quality, while preserving geometric structure.Scaffold-based competitors follow target motion but suffer severe degradation, including distorted bodies and collapsed limbs.
- iPhone Dataset: MoCam significantly outperforms other methods on PSNR, SSIM, LPIPS, and FVD on the iPhone dataset, with especially strong gains in perceptual metrics.Qualitative results show more accurate facial structure and clothing than the blurry or distorted outputs of competing methods.
- Ablation and Generalization: MoCam’s structured denoising dynamics outperform all ablation variants by sequencing geometric and appearance conditioning rather than applying either persistently or simultaneously.Scaffold-Only collapses with IQ 0.4807, while Static-Both reaches Rotation Error 2.71 and Translation Error 11.01; MoCam also remains coherent under perturbed depth and across Wan2.1 and Wan2.2.
5 Conclusion
MoCam is a unified 3D/4D view synthesis framework that addresses sparse and erroneous geometry through structured denoising dynamics. Its temporal separation of geometry and appearance enables early scaffold anchoring and late-stage error correction without sacrificing geometric control.
- MoCam addresses sparse and erroneous geometry in unified 3D/4D view synthesis through structured denoising dynamics.
- Temporal decoupling anchors geometry early and corrects errors late, preventing point-cloud flaws from propagating without sacrificing geometric control.
- This stage-wise design remains robust to imperfect reconstruction, which is inevitable in monocular settings.
A Comparison with 3D-based method
MoCam outperforms ViewCrafter in single-view 3D reconstruction, preserving correct shapes and view coherence while achieving better visual quality and structure accuracy.
- Qualitative and quantitative comparison: Compared with ViewCrafter, MoCam maintains correct shapes and view coherence, avoiding distortions such as incorrect tires and text through structured denoising dynamics.The comparison covers qualitative and quantitative single-view 3D reconstruction results.
- Qualitative and quantitative comparison: MoCam achieves better visual quality and structure accuracy than ViewCrafter on single-view 3D reconstruction.The quantitative comparison is reported on the OpenVid dataset using BC, SC, IQ, RotErr, and TransErr.