PaperlayerView exact revision on arXiv

Source-linked AI summary

MV-Adapter: Multi-view Consistent Image Generation Made Easy

Zehuan Huang, Yuan-Chen Guo, Haoran Wang, Ran Yi, Lizhuang Ma, Yan-Pei Cao, Lu Sheng

arXiv:2412.03632v1cs.CV

TL;DR

Existing multi-view generation methods require invasive T2I-model changes and full fine-tuning, with computational and image-quality challenges. MV-Adapter adds multi-view generation through a plug-and-play adapter with duplicated and parallel attention and unified camera and geometry conditioning. It achieves 768-resolution multi-view generation on SDXL and supports derivative models and arbitrary viewpoints.

  • Problem

    Existing multi-view methods modify pre-trained T2I models and require full fine-tuning, posing computational challenges and image-quality difficulties amid scarce high-quality 3D data.

    Source: §1 · p. 2

  • Method

    MV-Adapter is a plug-and-play adapter with duplicated self-attention layers, parallel attention, and a unified encoder for camera and geometric conditions.

    Source: §1 · p. 2

  • Results

    MV-Adapter achieves multi-view generation at 768 resolution on SDXL and is reported to adapt to T2I derivatives and produce consistent reference-image results with parallel attention.

    Source: §1 · p. 2, §5.3 · p. 8

  • Takeaways & Limitations

    The adapter supports camera- and geometry-guided applications, including 3D and texture generation, and can be extended to arbitrary viewpoints.

    Source: §1 · p. 3

    The stated future-work scope includes extending the approach to scene-level multi-view generation and learning zoom, lighting, physical, or temporal properties.

    Source: §A.4 · p. 22

Abstract

from arXiv · show

Existing multi-view image generation methods often make invasive modifications to pre-trained text-to-image (T2I) models and require full fine-tuning, leading to (1) high computational costs, especially with large base models and high-resolution images, and (2) degradation in image quality due to optimization difficulties and scarce high-quality 3D data. In this paper, we propose the first adapter-based solution for multi-view image generation, and introduce MV-Adapter, a versatile plug-and-play adapter that enhances T2I models and their derivatives without altering the original network structure or feature space. By updating fewer parameters, MV-Adapter enables efficient training and preserves the prior knowledge embedded in pre-trained models, mitigating overfitting risks. To efficiently model the 3D geometric knowledge within the adapter, we introduce innovative designs that include duplicated self-attention layers and parallel attention architecture, enabling the adapter to inherit the powerful priors of the pre-trained models to model the novel 3D knowledge. Moreover, we present a unified condition encoder that seamlessly integrates camera parameters and geometric information, facilitating applications such as text- and image-based 3D generation and texturing. MV-Adapter achieves multi-view generation at 768 resolution on Stable Diffusion XL (SDXL), and demonstrates adaptability and versatility. It can also be extended to arbitrary view generation, enabling broader applications. We demonstrate that MV-Adapter sets a new quality standard for multi-view image generation, and opens up new possibilities due to its efficiency, adaptability and versatility.

1 INTRODUCTION

Existing multi-view methods face computational and image-quality challenges from full fine-tuning of T2I models. MV-Adapter adds multi-view capability through a plug-and-play adapter, achieving 768-resolution generation on SDXL and supporting varied conditions and viewpoints.

1 INTRODUCTION

  • Existing multi-view methods fine-tune T2I models, creating computational challenges at high resolution and risking image-quality degradation amid scarce high-quality 3D data.The cited methods process at least n view images simultaneously during training; advanced methods still struggle at 512 resolution.Source: §1 · p. 2
  • MV-Adapter is a plug-and-play approach that enhances T2I models and derivatives while preserving the base model’s network structure and feature space.Updating fewer parameters is presented as faster and more memory-efficient, while constraining optimization to help preserve pre-trained knowledge and mitigate overfitting.Source: §1 · p. 2
  • Duplicated self-attention layers and parallel attention let the adapter model geometric knowledge while inheriting priors from pre-trained self-attention.The duplicated layers create multi-view attention and image cross-attention; their output projections are initialized to zero.Source: §1 · p. 2
  • A unified condition encoder integrates camera parameters and geometric information into spatial-map representations.The paper connects this guidance to applications in 3D generation and texture generation.Source: §1 · pp. 2–3
  • 768-resolution multi-view generation is achieved on Stable Diffusion XL (SDXL).The adapter also supports customized or controllable generation on base-model derivatives, camera and geometry guidance, and arbitrary viewpoints.Source: §1 · pp. 2–3
  • The paper presents the adapter-based approach as a framework for efficient learning, versatile 3D and texture applications, arbitrary-view generation, and decoupled learning of new knowledge.The proposed decoupling is also framed as potentially informative for modeling physical or temporal knowledge.Source: §1 · p. 3

2 RELATED WORK

The section situates MV-Adapter among advances in text-to-image diffusion, model derivatives, and multi-view generation. It contrasts prior multi-view methods’ extensive model updates with an adapter designed to work with pre-trained T2Is and their derivatives.

2 RELATED WORK

  • Diffusion advances improved text conditioning, text-image alignment, and image quality; Stable Diffusion XL is described as a two-stage cascade model with improved high-frequency detail.The section also describes Stable Diffusion as performing diffusion in an autoencoder’s latent space.Source: §2 · p. 3
  • T2I derivatives address efficiency and controllable generation through distillation, personalization, and spatial-control methods, often using adapters or fine-tuning.The paper positions its approach as adhering to the non-intrusive principle for compatibility with these extensions.Source: §2 · p. 3
  • Prior multi-view methods extend T2I models using large-scale 3D datasets, but typically update many parameters, alter pre-trained feature spaces, and limit compatibility with derivatives.Examples include MVDream’s camera embeddings and 3D self-attention (Shi et al., 2023b), SPAD’s epipolar constraints, and Era3D’s row-wise attention (Li et al., 2024).Source: §2 · p. 3

3 PRELIMINARY

Multi-view diffusion models extend T2I attention across views to promote consistency, but attending to all view pixels is computationally intensive. Geometric attention and row-wise attention restrict cross-view interactions to reduce that cost.

3 PRELIMINARY

  • Multi-view diffusion models add multi-view attention to T2Is to generate images consistent across viewpoints.Some methods extend self-attention to include pixels across all multi-view images.Source: §3 · p. 3
  • Dense multi-view self-attention concatenates features from n views and attends across all pixels, capturing global dependencies at high computational cost.Epipolar attention restricts attention to patches along epipolar lines to mitigate that cost.Source: §3 · p. 4
  • For orthographic views at 0° elevation, epipolar lines align with image rows, motivating row-wise self-attention after the original T2I self-attention layers.The displayed process first applies self-attention and then multi-view attention across corresponding rows in different views.Source: §3 · p. 4

4 METHOD

MV-Adapter plugs into personalized or distilled T2Is, combining a condition guider with decoupled attention; its added layers learn multi-view consistency and support camera-, geometry-, and image-conditioned generation.

4.1 CONDITION GUIDER

  • The condition guider encodes camera raymaps or geometry maps (position and normals), then adds multiscale convolutional features to U-Net scales; it can also accept depth or pose maps.Raymaps encode each spatial location’s ray origin and direction; position maps provide cross-view point correspondences, while normals encode orientation and fine geometric details.Source: §4.1 · pp. 4–5

4.2 DECOUPLED ATTENTION

  • Decoupled attention retains original spatial self-attention and adds duplicated multi-view and optional image cross-attention layers in parallel, with zero-initialized outputs to preserve the base feature space.The duplicated layers are initialized from pretrained spatial self-attention weights, while their output projections are initialized to zero.Source: §4.2 · pp. 5–6
  • Multi-view attention uses row-wise self-attention for object generation, row- and column-wise attention for texturing, and full self-attention for arbitrary-view generation.Texture generation uses four evenly spaced views at 0° elevation plus top and bottom views.Source: §4.2 · p. 6
  • For image conditioning, the frozen pretrained T2I U-Net encodes a clear reference image at timestep t = 0; multiscale self-attention features enter denoising through decoupled image cross-attention.The extracted features provide fine-grained subject information for controlling generated content.Source: §4.2 · p. 6

4.3 TRAINING AND INFERENCE

  • Training freezes the pretrained T2I and optimizes only MV-Adapter on reference-image, text, and n-view tuples; it randomly drops image features and shifts log-SNR by log(n).The log-SNR shift increases with the number of generated views.Source: §4.3 · p. 6

5 EXPERIMENTS

Experiments evaluate MV-Adapter across camera- and geometry-guided generation, adapter efficiency and adaptability, and downstream 3D reconstruction and texturing.

5.1 CAMERA-GUIDED MULTI-VIEW GENERATION

  • On camera-guided text- and image-to-multiview tasks, MV-Adapter achieves higher visual fidelity and consistency with conditions than the compared methods.The evaluations use 1,000 Objaverse prompts for text-to-multiview generation and the GSO dataset for image-to-multiview generation.Source: §5.1 · pp. 6–7, Table 1 · p. 7, Table 2 · p. 7

5.2 GEOMETRY-GUIDED MULTI-VIEW GENERATION

  • MV-Adapter integrates with different T2I derivative models, and its geometry-guided generation is evaluated on representative models and extensions.The community-model evaluation includes personalized and distilled models, as well as ControlNet.Source: §5.1 · p. 6, §5.2 · p. 7, Figure 5 · p. 7

5.4 APPLICATIONS

  • Text- and image-conditioned texturing produces high-quality textures faster than the compared methods, with backprojection and inpainting used to fill uncovered UV-map regions.The paper reports the comparison in Figure 10 and Table 3 and uses view-coverage analysis to identify uncovered areas.Source: §5.2 · pp. 7–8, §5.4 · pp. 9–10, Table 3 · p. 9

5.3 ABLATION STUDY

  • Adapter training reduces SDXL training costs versus full training, while adaptability and attention ablations favor MV-Adapter’s compatibility and parallel design.MVDream struggles to align images with text and style when tested with T2I derivatives; serial attention produces artifacts and inconsistent details compared with the parallel setting.Source: §5.3 · p. 8, Table 4 · p. 9, Table 5 · p. 9

5.4 APPLICATIONS

  • MV-Adapter’s SDXL-based model significantly outperforms Era3D on 3D reconstruction, while its SD2.1-based model is comparable.The paper uses generated multi-view images and normal maps for reconstruction; Table 6 reports the comparison.Source: §5.4 · p. 9, Table 6 · p. 9
  • For arbitrary-view generation, eight anchor views cover the object, and the four nearest views guide generation of each new target view.The nearest anchors are selected by clustering viewpoints based on spatial orientation.Source: §5.4 · p. 10

6 CONCLUSION

MV-Adapter is presented as a plug-and-play approach to multi-view generation that preserves the base model’s feature space while modeling 3D geometric knowledge. The conclusion emphasizes its efficiency, adaptability, and versatility across models, conditions, and applications.

6 CONCLUSION

  • MV-Adapter enhances T2I models and derivatives without compromising quality or altering their original feature space.The conclusion characterizes it as a versatile, plug-and-play adapter.Source: §6 · p. 10
  • Duplicated self-attention layers and parallel attention let the adapter model 3D geometric knowledge; a unified encoder integrates camera parameters and geometry.The encoder represents this information in spatial maps.Source: §6 · p. 10
  • Evaluations highlight efficiency, adaptability, and versatility, with applications including 3D object generation and texture generation.The conclusion describes MV-Adapter as an efficient and flexible solution for multi-view image generation.Source: §6 · p. 10

A.1 BACKGROUND

The background describes latent-space diffusion in Stable Diffusion and Stable Diffusion XL, including their denoising networks and architectural differences.

A.1 BACKGROUND

  • Stable Diffusion and SDXL perform diffusion in the latent space of a pretrained autoencoder.An encoded image is perturbed by forward diffusion, and the denoising network learns to predict the added noise using an MSE loss.Source: §A.1 · p. 15
  • Stable Diffusion’s denoising UNet has paired down- and up-sampling blocks and a middle block, with self-attention and cross-attention layers connected serially.The conditioning input c denotes the texts.Source: §A.1 · p. 15
  • SDXL uses a UNet backbone three times larger than Stable Diffusion’s and adds a refinement denoiser to improve visual fidelity.Source: §A.1 · p. 15

A.2 IMPLEMENTATION DETAILS

The implementation trains MV-Adapter on rendered 3D data with SD2.1 and SDXL backbones, then evaluates text-, image-, and geometry-conditioned generation against task-specific baselines. The setup also tests transfer to varied community models and extensions.

Dataset

  • 70,000 Objaverse samples train six-view models; the camera-guided views use fixed azimuths, while geometry-guided data adds top and bottom views.Training uses a filtered high-quality subset with Cap3D captions; five random reference views are also rendered.Source: §A.2 · pp. 15–16

Training

  • SD2.1 and SDXL models train for 10 epochs on eight A100 GPUs, with a 5 × 10^-5 learning rate and condition dropout.The noise schedule shifts by log(n), where n = 6 views.Source: §A.2 · p. 16

Inference

  • Inference uses 50-step DDPM sampling; text-only generation uses guidance scale 7.0, while image-conditioned generation sets image and text guidance to 3.0.Camera and geometry conditions receive no classifier-free guidance because they are never dropped during training.Source: §A.2 · p. 16

Comparison with baselines

  • Text-to-multiview tests compare MVDream and SPAD on 1,000 Objaverse prompts using FID, IS, and CLIP Score.Image-to-multiview tests use 100 GSO assets and report PSNR, SSIM, and LPIPS; texture tests compare six methods on 200 Objaverse models.Source: §A.2 · p. 16

Community models and extensions

  • Evaluation spans personalized, distilled, and controllable community models across varied styles and subjects, testing transfer beyond a single T2I derivative.The implementation also evaluates texture quality with FID and KID and records generation time.Source: §A.2 · pp. 17–18, Table 7 · p. 17

A.3.1 MV-ADAPTER VS. MULTI-VIEW LORA

Multi-view LoRA can interfere with personalized model adaptations, whereas MV-Adapter separates its attention layers from the base model’s spatial attention. The reported experiments show more robust consistency when transferring across models and integrating personalized LoRAs.

Limitation of multi-view LoRA

  • Multi-view LoRA layers can overlap with personalized LoRA layers, creating a trade-off between multi-view consistency and fidelity to a style or subject.The passage attributes this difficulty to the non-orthogonality of the learned transformations.Source: §A.3.1 · p. 18

Decoupled attention

  • MV-Adapter trains decoupled attention layers without fine-tuning the original spatial self-attention, preserving the base feature space and improving compatibility.The trained layers do not overlap with those in the original T2I model.Source: §A.3.1 · p. 18

Experiments

  • Across two rank-64, alpha-32 LoRA settings, multi-view LoRA often loses consistency after a base-model switch or new LoRA integration; MV-Adapter maintains it with personalized models.The experiments target attention layers alone or multiple module types; the comparison is shown in Figures 12–14.Source: §A.3.1 · p. 18

Conclusion

  • Compared with LoRA, decoupled attention is reported as more robust and adaptable for extending T2I models, with greater compatibility across pre-trained models.Source: §A.3.1 · p. 18

A.3.2 IMAGE RESTORATION CAPABILITIES

The supplementary results report image restoration from low-resolution references, show use with personalized models and extensions, and demonstrate generation beyond six viewpoints. They also identify dependence on the base image model and report user-study preferences for image quality.

A.3.2 IMAGE RESTORATION CAPABILITIES

  • Training with probabilistically downscaled reference images lets MV-Adapter generate detailed high-resolution multi-view images from low-resolution inputs.The authors describe this training strategy as giving the model image-restoration capabilities.Source: §A.3.2 · p. 18

A.3.3 APPLICABILITY OF MV-ADAPTER

  • MV-Adapter is presented as applicable to personalized T2I models, with potential uses including multi-view portraits and stylized illustrations that retain identity or artistic coherence across angles.Source: §A.3.3 · pp. 18–19

A.3.4 EXTENDING MV-ADAPTER FOR ARBITRARY VIEW SYNTHESIS

  • The arbitrary-view extension generates eight anchor views, then conditions target views on four nearest known views and uses full self-attention for flexible viewpoint distributions.The reported visualizations show consistent, high-quality outputs beyond the six views designed for specific applications.Source: §A.3.4 · p. 21

A.4 LIMITATIONS AND FUTURE WORKS

  • MV-Adapter remains heavily dependent on its base T2I model and is unlikely to compensate when that model produces low-quality images or misses the prompt.The authors report better synthetic results with superior image backbones, particularly for text-conditioned generation.Source: §A.4 · p. 22, Figure 17 · p. 22
  • Proposed future directions include scene-level generation, dynamic multi-view video, and applying decoupled geometric learning to zoom, lighting, and other viewpoint-dependent properties.Source: §A.4 · p. 22

A.5.1 IMAGE-TO-MULTI-VIEW GENERATION

  • 36.80% of user-study preferences favored MV-Adapter for image quality; for multi-view consistency, MV-Adapter received 25.07% and Era3D 22.33%.The study compared SD2.1-based MV-Adapter with baselines using 30 samples and 50 participants.Source: §A.5.1 · pp. 22–23

A.6 MORE VISUAL RESULTS

  • The supplementary gallery includes camera-guided text generation with ControlNet and IP-Adapter, image-to-multiview, 3D generation, and geometry-guided texturing results.Generated-image backgrounds were removed in these visual results.Source: §A.6 · p. 23
Loading 2412.03632v1…