Consistent View Synthesis with Pose-Guided Diffusion Models

1. Introduction

• Earlier single-image view-synthesis methods can produce high-quality novel views but are limited to a small range of camera motion.Source: §1 · p. 1
• Long-term view synthesis generates a video from one image and a camera-pose sequence, allowing later views to move far from the input viewpoint.The described examples include cameras moving into rooms absent from the input image.Source: §1 · p. 2
• Large camera movements require hallucinating substantial unseen content while keeping views consistent with the input and previously generated views.Source: §1 · p. 2
• Prior approaches use depth-dependent warping and refinement or geometry-free transformers, but consistent, high-quality long-term results remain challenging.The passage notes that warp-and-refine approaches depend on monocular depth accuracy, while the cited transformer methods have consistency challenges.Source: §1 · p. 2
• Epipolar attention uses camera-pose-derived epipolar lines to constrain associations between input- and output-view features in the diffusion UNet.The epipolar attention layers are added after each self-attention layer.Source: §1 · p. 2
• On Realestate10K and Matterport3D, the model generates realistic content in unseen regions and consistent views across viewpoints.The paper reports quantitative and qualitative evaluation on both datasets.Source: §1 · p. 2

2. Related Work

• Existing novel-view approaches generally interpolate among input views or extrapolate only within limited camera movement; PixelSynth [45] and SE3DS [19] progressively build point clouds, while GeoGPT [47] avoids monocular depth.LoR [44] reduces temporal flickering, but producing high-quality distant views remains challenging.Source: §2 · p. 2
• The overview places epipolar-attention layers after UNet self-attention layers; camera poses define epipolar-line constraints for source-to-target feature attention.The UNet denoises a target-view image from the source image, camera poses, and source-view features.Source: Figure 2 · p. 3
• The paper positions its diffusion model as a way to generate consistent, realistic sequences of novel views.Source: §2 · p. 3
• Diffusion models support varied image-generation tasks, but their lack of 3D camera-pose controllability prevents direct application to view synthesis.Source: §2 · p. 3
• Unlike object-centric 3DiM [73], this work targets long-term generation for realistic, complex scene-centric scenes and reports quality gains from explicit epipolar constraints.Source: §2 · p. 3
• Epipolar attention is a form of cross-attention that uses geometric constraints to compute dependencies between source- and target-view pixels.Source: §2 · p. 3

3. Methodology

• The framework autoregressively generates a sequence of novel views from an input image and camera poses.The goal is a sequence of images, with each viewpoint generated in turn.Source: §3 · p. 3

3.1. Diffusion Model

• The forward process gradually adds Gaussian noise to real images, and the learned backward process maps noise back to images.The noise schedule is chosen so that the final forward-process state approximates a Gaussian distribution.Source: §3.1 · pp. 3–4
• The UNet predicts the noise term for the backward process and is trained with a mean-square noise-prediction loss.The method follows DDPM training for noise prediction and uses the improved-DDPM variational objective for the covariance term.Source: §3.1 · p. 4

3.2. Pose-Guided Diffusion Model

• Epipolar attention reweights source-to-target cross-view affinities using camera-derived epipolar-line weights before producing target-view features.The source-view encoder supplies features, while the epipolar constraint reduces candidate source pixels for each target pixel.Source: §3.2 · p. 4

3.3. Consistent Long-Term View Synthesis

• Autoregressive inference uses prior frames as conditioning, with stochastic source-frame selection to improve temporal consistency.Fixing all backward-process noises produces artifacts; fixing noise only through an early step and resampling later alleviates the issue while improving consistency and maintaining realism.Source: §3.1 · p. 4, §3.3 · p. 5

4.1. Experimental Setup

• Experiments use RealEstate10K and Matterport3D, compare against GeoGPT, LoR, and SE3DS, and generate 20-frame sequences for evaluation.Long-term evaluation uses last-frame FID and KID for image quality and flow-warping error for temporal consistency.Source: §4.1 · pp. 5, 7

4.3. Long-Term View Synthesis

• The pose-guided model maintains image quality over time and produces less temporal flickering than existing approaches in long-term view synthesis.Figure 6 compares last-frame FID and flow-warping error across generated video lengths; the reported comparison contrasts the model with GeoGPT and LoR.Source: Figure 6 · p. 7, §4.3 · p. 7

4.4. Ablation Study

• The Concat baseline fails to generate high-quality long-term views compared with Cross-view attention, while epipolar constraints help estimate source–target dependencies.The ablation results are presented in Table 3; the proposed attention produces realistic novel views.Source: §4.4 · pp. 7–8, Table 3 · p. 8
• RealBasicVSR yields less flickering, while pose-guided diffusion super-resolution produces higher-quality novel views and is used in the experiments.The comparison uses the same 64 × 64 sequences as input and evaluates super-resolved 256×256 videos.Source: §4.4 · p. 8

Scale variation

• The method cannot handle scene scales that vary dramatically across videos, including scenes significantly larger than those in the Re10K training data.The authors identify scale normalization or data augmentation as possible requirements for handling such cases.Source: §5 · p. 8, Figure 7 · p. 8

Inference time

• Inference is time-consuming because predicting one novel view requires 250 backward-process steps.The authors plan to explore diffusion-model acceleration methods.Source: §5 · p. 8

• The pose-guided diffusion model generates novel-view videos from a single image under massive camera motion.Source: §6 · p. 8
• Its generated views are realistic even when viewpoints are far from the input view.Source: §6 · p. 8
• The generated views remain consistent across various viewpoints.Source: §6 · p. 8

• The supplement presents qualitative results for existing methods and the proposed method.Source: §A.1 · p. 12
• It provides technical details of the proposed approach to complement the main paper.Source: §A.1 · p. 12
• It describes experiment details, including dataset preprocessing, metric computation, and reproduction of existing approaches.Source: §A.1 · p. 12

Comparing with previous SOTA methods

• The supplement presents single-image view-synthesis results from GeoGPT [47], LoR [44], SE3DS [19], and the proposed model.Source: §A.2 · p. 12

Generating diverse outputs

• The method produces multiple realistic videos from the same inputs.Source: §A.2 · p. 12

Ablation study

• Baseline videos from the final few frames show significantly degraded per-frame quality relative to the proposed method.The comparison uses 64×64 diffusion outputs followed by the same super-resolution model to produce 256 × 256 results.Source: §A.2 · p. 12

Long-range view interpolation

• The method is also applied to interpolate between two far-away viewpoints, using stochastic conditioning and comparison with FiLM [43].The experiment uses the 1st, 11th, and 21th frames as inputs to produce a 21-frame video.Source: §A.2 · p. 12

Failure cases

• Reported failure cases include scenes with scales unlike the training data and rare camera-pose sequences such as going up or down stairs.Source: §A.2 · p. 12

Flickering caused by super-resolution

• Flickering in the generated video mainly comes from the per-frame super-resolution stage.An x-t slice visualization is provided for low- and high-resolution videos.Source: §A.2 · p. 12

Epipolar line computation

• A target-view point is projected onto the source-view image plane using the relative camera pose, and the projected point and camera origin define its epipolar line.Source: §A.3 · p. 12
• The method weights source-view pixels by distance to the epipolar line and uses the resulting matrix for epipolar attention.The weight map is visualized in Figure 9; when the line does not intersect the target view, attention falls back to common cross-attention.Source: §A.3 · pp. 12–13, Figure 9 · p. 13

X-T slice visualization

• The x-t slice visualization attributes most flickering to the per-frame super-resolution stage.Source: Figure 8 · p. 13

Stochastic frame sampling

• Stochastic source-view sampling is used during both training and inference; inference-only stochastic sampling gives inferior results.The authors hypothesize this is due to a mismatch in relative-pose distributions between training and inference.Source: §A.3 · p. 13

Super-resolution

• For high-resolution output, the system upsamples the low-resolution target view and concatenates it with a noised high-resolution target view for UNet denoising.Features from the high-resolution source view are also used; the authors report improved temporal consistency when taking the source view as input.Source: §A.3 · p. 13

Training details

• The proposed method is implemented in PyTorch and trained using eight Nvidia V100 GPUs.Source: §A.3 · p. 13

A.4. Experiment Details

• Re10K and MP3D test sets each contribute 500 videos to evaluation, with videos shorter than 200 frames excluded from Re10K.The processed Re10K split contains 61,986 training and 1,763 testing videos; MP3D contains 6,097 training and 1,062 test videos.Source: §A.4 · pp. 13, 15
• The super-resolution model concatenates an upsampled low-resolution target view with a noised high-resolution target view, then denoises using source-view features.The target view is bilinearly upsampled, and the UNet is trained to denoise the high-resolution target.Source: Figure 10 · p. 14
• The pose-guided diffusion model’s UNet produces 64 × 64 novel views, with further architecture and hyperparameter details referenced in Table 5.The supplied table passage identifies training details but does not provide their values.Source: Figure 11 · p. 14, Table 5 · p. 14
• Evaluation uses LPIPS, FID/KID, and RAFT-based flow warping error Ewarp.AlexNet computes LPIPS, InceptionV3 computes FID/KID, and pre-trained RAFT computes Ewarp.Source: §A.4 · p. 15
• Baseline evaluations use official implementations or released parameters, with dataset-specific training and image resizing for GeoGPT, LoR, and SE3DS.GeoGPT Re10K outputs are center-cropped and resized to 256 × 256; the passages specify distinct training or pretrained-model protocols for the other evaluations.Source: §A.4 · p. 15