Source-linked AI summary
MVDiffusion: Enabling Holistic Multi-view Image Generation with Correspondence-Aware Diffusion
Shitao Tang, Fuyang Zhang, Jiacheng Chen, Peng Wang, Yasutaka Furukawa
TL;DR
MVDiffusion addresses error accumulation and inconsistency in multi-view image generation, where prior methods often process views iteratively through warping and inpainting. It generates views in parallel with a pretrained diffusion model augmented by correspondence-aware attention, achieving state-of-the-art performance on panorama and depth-to-image tasks while remaining limited by sampling cost and memory use.
Problem
Prior multi-view text-to-image methods often generate views autoregressively through warping and inpainting, causing accumulated errors and poor handling of loop closure.
Method
MVDiffusion simultaneously generates multiple views with branches of a pretrained Stable Diffusion model connected by correspondence-aware attention.
Results
MVDiffusion achieves state-of-the-art performance on panoramic image generation and multi-view depth-to-image generation.
Takeaways & Limitations
The method supports high-resolution photorealistic panoramas from arbitrary texts, perspective-to-360-degree extrapolation, and scene-mesh texturing.
Takeaways & Limitations
Generating high-quality images requires at least 50 sampling steps, while parallel denoising makes the model memory-intensive and limits scalability to many images.
Abstract
from arXiv · showhide
This paper introduces MVDiffusion, a simple yet effective method for generating consistent multi-view images from text prompts given pixel-to-pixel correspondences (e.g., perspective crops from a panorama or multi-view images given depth maps and poses). Unlike prior methods that rely on iterative image warping and inpainting, MVDiffusion simultaneously generates all images with a global awareness, effectively addressing the prevalent error accumulation issue. At its core, MVDiffusion processes perspective images in parallel with a pre-trained text-to-image diffusion model, while integrating novel correspondence-aware attention layers to facilitate cross-view interactions. For panorama generation, while only trained with 10k panoramas, MVDiffusion is able to generate high-resolution photorealistic images for arbitrary texts or extrapolate one perspective image to a 360-degree view. For multi-view depth-to-image generation, MVDiffusion demonstrates state-of-the-art performance for texturing a scene mesh.
1 Introduction
MVDiffusion addresses inconsistency and error accumulation in multi-view text-to-image synthesis by generating views simultaneously with correspondence-aware cross-view interactions. It achieves state-of-the-art performance across panorama and depth-to-image generation tasks.
- Motivation: Autoregressive multi-view generation conditions each image on the previous one through warping and inpainting, accumulating errors and failing to handle loop closure.Large viewpoint variations and complex scenes further challenge reliance on the previous image.
- Approach: MVDiffusion generates multiple views simultaneously using branches of a pretrained perspective-image text-to-image model.The approach uses Stable Diffusion and adds correspondence-aware attention between UNet blocks.
- Approach: Correspondence-aware attention facilitates cross-view interactions and enforces multi-view consistency while the original Stable Diffusion weights remain frozen.Freezing the pretrained weights is intended to preserve the model’s generalization capability.
- Results: MVDiffusion achieves state-of-the-art performance on panorama generation and multi-view depth-to-image generation.It supports high-resolution photorealistic panoramas from arbitrary per-view texts, perspective-to-360-degree extrapolation, and scene-mesh texturing.
2 Related Work
Prior work develops diffusion-based image and 3D content generation, but multi-view methods commonly rely on local, iterative processing. MVDiffusion instead takes a holistic approach to consistent multi-view generation.
- Diffusion-based generation: Diffusion and latent diffusion models provide efficient foundations for high-resolution text-conditioned image synthesis.Latent diffusion reduces computation and memory by operating in a compressed latent space.
- Panorama generation: Existing 360-degree generation methods do not necessarily produce true panoramas.The supplied related-work passage identifies this limitation without specifying the methods’ exact failure modes.
- MVDiffusion: MVDiffusion’s multi-branch architecture generates views in parallel and inserts correspondence-aware attention after each UNet block.The figure identifies FFN as Feed-Forward Network and elaborates on CAA mechanisms.
- Scene and multi-view generation: Prior scene-generation methods use image warping and inpainting along trajectories or between views.These approaches include SceneScape and Text2Room.
- Scene and multi-view generation: Because each generated image relies on local context, iterative methods accumulate errors and produce less favorable results.MVDiffusion is introduced as a holistic alternative using camera poses and text prompts.
3 Preliminary
MVDiffusion builds on latent diffusion models, which encode images into a compressed latent space, denoise conditionally with a UNet, and decode the result into high-resolution images.
- Latent Diffusion Models: Latent Diffusion Models form the foundation of MVDiffusion and contain a VAE, denoising network, and condition encoder.The VAE uses encoder E and decoder D, while the denoising network is εθ and the condition encoder is τθ.
- Latent representation: High-resolution images are mapped into a lower-dimensional latent space by the VAE encoder and reconstructed by the decoder.Stable Diffusion uses a down-sampling factor f = 8.
- Training objective: The latent diffusion objective trains the denoising network to predict noise from noisy latents under conditioning.The supplied equation states the training objective, while the surrounding text defines the noisy latent and time sampling.
- Conditional denoising: The time-conditional UNet incorporates optional text, image, or user-specified conditions through cross-attention.The timestep t is uniformly sampled from 1 to T.
- Sampling: At sampling time, reverse denoising produces latent samples that the decoder converts into high-resolution images in one forward pass.Advanced samplers can further accelerate sampling.
4 MVDiffusion: Holistic Multi-view Image Generation
MVDiffusion generates corresponding views in parallel with a shared multi-branch UNet and correspondence-aware attention. The design supports panorama generation, perspective-to-panorama extrapolation, and depth-to-image scene texturing.
- Architecture: MVDiffusion runs multiple Stable Diffusion branches simultaneously and connects them with correspondence-aware attention to enforce multi-view consistency.The system applies when pixel-to-pixel correspondences are available.
- Correspondences: Pixel correspondences come from planar tomography for panoramas and depth-based unprojection and projection for geometry texturing.The two tasks use separate generation pipelines, with interpolation added for depth-to-image generation.
- Panorama generation: The panorama generation module produces eight 512 × 512 perspective images with 90° horizontal fields of view and 45° overlap.The views are generated through simultaneous denoising with a frozen pretrained Stable Diffusion model.
- Correspondence-aware attention: The multi-branch UNet feeds noisy latents through shared weights and places a CAA block after each UNet block.CAA operates concurrently on multiple feature maps and exchanges information across views.
- Correspondence-aware attention: CAA aggregates information from corresponding target pixels and their local neighborhoods through cross-attention.The implementation uses K = 3, giving a neighborhood of 9 points.
- Correspondence-aware attention: CAA adds frequency-encoded 2D displacement information and uses bilinear interpolation for non-integer target locations.Zero initialization of final layers preserves the pretrained Stable Diffusion functionality.
- Task-specific modules: Panorama extrapolation generates seven target views from one perspective condition, while depth-to-image generation produces key images that interpolation densifies.The depth-to-image generation module uses a depth-conditioned Stable Diffusion model and CAA-equipped multi-branch UNet.
- Training: Training freezes the pretrained Stable Diffusion network and trains the added CAA blocks, while depth-to-image training uses two stages.The two-stage procedure first fine-tunes the UNet on ScanNet and then trains the added CAA and image-condition parameters.
5 Experiments
MVDiffusion is evaluated on panorama generation and multi-view depth-to-image generation using image-quality and multi-view-consistency metrics. Across qualitative and quantitative comparisons, it preserves image quality while improving border and sequence consistency over autoregressive, inpainting, and standard diffusion baselines.
- Evaluation setup: The experiments evaluate panorama generation and multi-view depth-to-image generation using image quality and multi-view consistency metrics.Image quality uses FID, IS, and CS; consistency uses an overlapping-PSNR ratio.
- Panoramic image generation: Matterport3D provides 10,912 panoramic images from 90 buildings, split into 9,820 training and 1,092 evaluation panoramas.The panorama task generates perspective crops covering the panoramic field of view, with consistency required in overlapping regions.
- Panoramic image generation: MVDiffusion addresses left-right border inconsistency found in Text2Light and Stable Diffusion panorama baselines by enforcing explicit correspondence-based consistency.Inpainting baselines also show error accumulation through gradual style changes across generated image sequences.
- Panoramic image generation: Despite training only on indoor scenes, MVDiffusion generates outdoor panoramas and diverse scenes beyond its training domain.The reported generalization is attributed to preserving the pretrained Stable Diffusion weights by freezing them during CAA training.
- Panoramic image generation: MVDiffusion can extrapolate a complete panorama from one perspective image in image-and-text-conditioned generation.Figure 5 presents an example of whole-scene extrapolation from a single perspective image.
- Multi-view depth-to-image generation: The depth-to-image experiments use ScanNet scenes and convert depth sequences into RGB images while preserving geometry and multi-view consistency.The evaluation includes 590 non-overlapping test sequences of 12 images, followed by fusion into a cohesive scene mesh.
- Multi-view depth-to-image generation: For multi-view depth-to-image generation, MVDiffusion achieves PSNR ratios of 0.67 and 0.76 for the two evaluated tasks, compared with a real-image upper limit of 1.0.The comparison uses an original Stable Diffusion model without correspondence attention as the lower-limit reference.
6 Conclusion
MVDiffusion simultaneously generates consistent multi-view images with correspondence-aware attention, achieving state-of-the-art performance while retaining a frozen pretrained diffusion model. Its main limitations are computational cost, memory requirements, and potential misuse for disinformation.
- Conclusion: MVDiffusion simultaneously generates consistent multi-view images using correspondence-aware attention integrated into each Stable Diffusion UNet block.The frozen pretrained model preserves generalization capability while the attention mechanism recognizes pixel-to-pixel correspondences.
- Conclusion: MVDiffusion achieves state-of-the-art performance in panoramic image generation and multi-view depth-to-image generation.The paper reports these results across both evaluated multi-view generation tasks.
- Limitations: At least 50 sampling steps and parallel denoising create computational and memory demands that limit scalability to applications requiring many images.The paper identifies long sampling trajectories and memory-intensive parallel denoising as bottlenecks, including for long virtual tours.
- Broader impact: MVDiffusion can support detailed environment generation for video games, virtual reality, and movie scenes, while potentially enabling disinformation.The broader-impact discussion connects the method to script-based content production and notes a misuse risk.
A Network Architecture of correspondence-aware attention block
The correspondence-aware attention block follows a vision-transformer-like architecture augmented with zero convolutions and GELU activation.
- Architecture: The correspondence-aware attention block is architecturally similar to vision transformers.It additionally incorporates zero convolutions and GELU activation.
- Architecture: Zero convolutions and GELU activation are included in the correspondence-aware attention block.The design follows the use of zero convolutions suggested in ControlNet.
- Notation: C, H, and W denote channel number, height, and width, respectively.
B Implementation details of MVDiffusion
MVDiffusion is trained from Stable-diffusion-v2 on 512 × 512 perspective images and performs parallel denoising of eight images with DDIM during inference.
- Training: The generation model is built upon Stable-diffusion-v2 and trained on 512 × 512 perspective images for 10 epochs.Training uses AdamW with batch size 4, learning rate 2e−4, and four A6000 GPUs.
- Inference: During inference, DDIM with a step size of 50 performs parallel denoising on eight generated images.
B.2 Implementation details of baselines
The appendix documents the correspondence-aware attention architecture and MVDiffusion implementation, while baseline procedures cover panorama and depth-to-image settings with parallel eight-image inference.
- Architecture: Figure 9 presents the architecture of the correspondence-aware attention block.
- Panorama baselines: Baseline panorama generation includes combining perspective prompts for Text2Light and fine-tuning Stable Diffusion on 9,820 panorama images.The Stable Diffusion panorama baseline keeps VAE layers frozen while fine-tuning its UNet.
- Depth-to-image implementation: The depth-to-image implementation fine-tunes Stable-diffusion-2-depth on ScanNet perspective images before training correspondence blocks and convolution layers.Inference uses DDIM with step size 50 for parallel denoising on eight images.
B.4 Implementation details of baselines
The baselines generate an initial image, then warp it into subsequent frames and use diffusion-based inpainting to fill missing regions.
- RePaint generates the first image with depth-conditioned Stable Diffusion-v2, then applies latent warping and RePaint inpainting to subsequent images.
- Depth-conditioned ControlNet warps generated images to the current frame and uses Stable-inpainting to fill holes with depth information.Its inpainting model is fine-tuned on the training dataset while keeping the VAE fixed.
B.5 Visualization results
The visualization results cover panorama generation, image-and-text-conditioned extrapolation, out-of-distribution scenes, baseline comparisons, depth-to-image outputs, and interpolated frames.
- Figures 10–17 provide supplementary panorama outputs and border-rotation visualizations for comparing left–right consistency, quality, and alignment.The visualizations include Stable Diffusion (panorama) and Text2light outputs.
- Figures 18–19 show MVDiffusion extrapolating a complete scene from one perspective image and a text description.
- Figure 20 illustrates generalization from training on MP3D indoor data to out-of-training-distribution scenes such as outdoor environments.
- Figures 21–26 present additional depth-to-image results alongside depth-conditioned ControlNet and RePaint baselines.
- Figure 27 uses left and right keyframes to generate middle frames with the interpolation module while maintaining consistency across the sequence.