PaperlayerView exact revision on arXiv

Source-linked AI summary

MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editing

Mingdeng Cao, Xintao Wang, Zhongang Qi, Ying Shan, Xiaohu Qie, Yinqiang Zheng

arXiv:2304.08465v1cs.CV

TL;DR

Existing methods struggle with consistent multi-image generation and complex non-rigid editing while preserving source appearance. MasaCtrl addresses both tasks through tuning-free mutual self-attention and reports faithful synthesis and editing results.

  • Problem

    Existing methods struggle to generate multiple images with consistent objects or characters across views and poses, and to perform complex non-rigid editing while preserving textures and identity.

    Source: §1 · p. 2, §4 · p. 5

  • Method

    MasaCtrl converts diffusion-model self-attention into mutual self-attention, using source-image contents and textures, with masks extracted from cross-attention maps to guide foreground querying.

    Source: §1 · p. 2, §4.1 · p. 5, §4.2 · p. 6

  • Results

    MasaCtrl simultaneously achieves consistent image synthesis and complex non-rigid editing, combining target layouts with source-image contents for more faithful and fine-grained results.

    Source: §1 · p. 2, §5.2 · p. 10

  • Takeaways & Limitations

    The method provides a tuning-free approach for preserving source-image contents while changing layouts or poses in text-to-image diffusion pipelines.

    Source: §1 · p. 2, §5.2 · p. 10, §7 · p. 11

    MasaCtrl can fail when Stable Diffusion cannot produce the desired layout or when target content is unseen or its layout changes drastically.

    Source: §6 · p. 11

Abstract

from arXiv · show

Despite the success in large-scale text-to-image generation and text-conditioned image editing, existing methods still struggle to produce consistent generation and editing results. For example, generation approaches usually fail to synthesize multiple images of the same objects/characters but with different views or poses. Meanwhile, existing editing methods either fail to achieve effective complex non-rigid editing while maintaining the overall textures and identity, or require time-consuming fine-tuning to capture the image-specific appearance. In this paper, we develop MasaCtrl, a tuning-free method to achieve consistent image generation and complex non-rigid image editing simultaneously. Specifically, MasaCtrl converts existing self-attention in diffusion models into mutual self-attention, so that it can query correlated local contents and textures from source images for consistency. To further alleviate the query confusion between foreground and background, we propose a mask-guided mutual self-attention strategy, where the mask can be easily extracted from the cross-attention maps. Extensive experiments show that the proposed MasaCtrl can produce impressive results in both consistent image generation and complex non-rigid real image editing.

1. Introduction

Existing text-to-image and editing methods struggle with consistency across changed poses or views while preserving identity and texture. MasaCtrl addresses this with tuning-free mutual self-attention and mask guidance.

Motivation

  • Fixed noise and similar prompts can still produce images with different structures and identities, limiting consistent multi-image generation.This capability is needed for applications such as comic books and short videos.Source: §1 · p. 2
  • Existing editing methods preserve structure and layout but often fail at complex non-rigid changes while maintaining textures and identity.Imagic supports such edits but requires fine-tuning the diffusion model and optimizing textual embeddings for each image.Source: §1 · p. 2

MasaCtrl

  • MasaCtrl converts self-attention into mutual self-attention so new diffusion processes query correlated local structures and textures from source-image processes.Query features from the new process access key and value features from the source process.Source: §1 · p. 2
  • The method controls denoising steps and U-Net layers to preserve source textures and identity while target text controls structure, pose, and non-rigid variation.Directly applying mutual self-attention would otherwise produce images nearly identical to the source and ignore the target prompt.Source: §1 · p. 2
  • Mask-guided mutual self-attention uses masks extracted from cross-attention maps to reduce confusion between foreground and background regions.The strategy is intended for cases where foreground and background share similar patterns or colors.Source: §1 · pp. 2–3

Contribution

  • MasaCtrl is presented as a tuning-free method for consistent image synthesis and complex non-rigid real-image editing.The authors report effectiveness in both consistent generation and complex non-rigid editing.Source: §1 · pp. 2–3

2. Related Work

Related work spans GAN, autoregressive, and diffusion approaches to text-guided synthesis and editing. Diffusion models are efficient but existing methods remain limited in local, global, and non-rigid editing.

Text-to-Image Generation

  • Early text-conditioned generation methods relied heavily on GANs and targeted high-fidelity synthesis on domain-specific datasets.These models aligned text descriptions with image contents through multimodal vision-language learning.Source: §2.1 · p. 3
  • Autoregressive and diffusion models later achieved impressive diversity in text-to-image generation.Source: §2.1 · p. 3

Text-guided Image Editing

  • GAN-based text-guided editing achieved success on domain-specific datasets but had limited applicability and generality.Source: §2.2 · p. 3
  • VQGAN-CLIP produced high-quality, precise, diverse edits but suffered from slow generation and high computational cost.Source: §2.2 · p. 3

Diffusion-based Editing

  • Existing diffusion editing methods require extra masks, cannot always modify local details, or preserve the source layout and fail on non-rigid transformations.Source: §2.2 · p. 3

3. Preliminaries

Diffusion models generate data by progressively adding noise and then reversing that process, while Stable Diffusion performs denoising in latent space. Its U-Net uses self-attention for spatial features and cross-attention for textual conditioning, motivating attention-based control for image structure and content.

Diffusion models

  • Diffusion models synthesize data by iteratively removing noise from Gaussian noise through a learned reverse process.The forward process transforms x0 into noisy xt and ultimately Gaussian noise; the reverse process samples cleaner states iteratively.Source: §3.1 · p. 3
  • The reverse-process mean is obtained with a neural network that predicts the noise added at each timestep.The noise-estimation network is trained with a simplified mean-squared-error objective.Source: §3.1 · p. 3

Latent diffusion models

  • Stable Diffusion encodes images into latent representations and performs the denoising process in latent space before decoding the final latent representation.A pretrained autoencoder supplies the latent representation, and the denoising U-Net operates on it.Source: §3.1 · p. 4

Attention mechanism

  • Each Stable Diffusion U-Net block combines residual, self-attention, and cross-attention modules during denoising.Self-attention reorganizes spatial features, while cross-attention incorporates information from the text prompt.Source: §3.2 · p. 4
  • Attention computes a map from projected queries, keys, and values, then uses that map to aggregate value features.Keys and values come from spatial features in self-attention or textual embeddings in cross-attention.Source: §3.2 · p. 4
  • Attention layers contain information about synthesized-image structure, layout, and content formation, making them useful for image editing and faithful synthesis.Cross-attention binds spatial pixels to text embeddings, while self-attention features can be injected as plug-and-play features.Source: §3.2 · p. 4

4. Tuning-Free Mutual Self-Attention Control

MasaCtrl performs tuning-free consistent synthesis and non-rigid editing by querying source-image contents through mutual self-attention while retaining target-prompt control over layout. Mask-guided attention further restricts foreground and background queries to corresponding regions.

Mutual Self-Attention

  • MasaCtrl combines target-prompt layout with source-image contents to preserve textures and identity during synthesis and editing.The source image may be generated or obtained by DDIM inversion of a real image.Source: §4 · p. 5
  • Mutual self-attention keeps the current Query features while obtaining Key and Value features from the source-image reconstruction process.This converts self-attention into a cross-process operation that aggregates correlated source-image content during denoising.Source: §4 · p. 5, §4.1 · p. 5
  • Attention control begins after a selected denoising step and in selected U-Net decoder layers, allowing the target prompt to form layout before source content is injected.The control uses starting step S and layer index L; it is avoided in earlier steps because they can disrupt target-layout formation.Source: §4.1 · pp. 5–6

Mask-Guided Mutual Self-Attention

  • Mask-guided mutual self-attention addresses foreground-background confusion by extracting semantic masks from cross-attention maps and restricting queries to corresponding source regions.Cross-attention maps are averaged across heads and suitable layers at 16 × 16 resolution before foreground-token maps produce masks for source and target images.Source: §4.2 · p. 6, Figure 5

Integration to Controllable Diffusion Models

  • MasaCtrl can integrate with controllable synthesis pipelines such as T2I-Adapter and ControlNet without fine-tuning the Stable Diffusion model or optimizing textual embeddings.This integration combines controllable layouts or poses with contents queried from a reference image.Source: §4 · p. 5, §4.3 · p. 7

5. Experiments

Experiments evaluate MasaCtrl on synthetic and real-image editing, controllable diffusion pipelines, anime-style models, video synthesis, and attention-control settings. Results show improved consistency and prompt compliance, while ablations identify when mutual self-attention preserves source contents without overriding target layouts.

Experimental Setup

  • Stable Diffusion experiments use publicly available v1.4 checkpoints, while Anything-V4 evaluates anime-style synthesis and real-image editing includes DDIM inversion.Sampling uses 50 DDIM denoising steps and classifier-free guidance of 7.5; default attention control starts at S = 4 and L = 10.Source: §5 · p. 8

Comparisons with Previous Works

  • MasaCtrl synthesizes images that preserve source foreground and background contents while complying with the target prompt, unlike compared methods.The comparison evaluates tuning-free diffusion editing methods including SDEdit, P2P, and PnP.Source: §5.1 · p. 9
  • MasaCtrl edits real images while retaining input contents and offers non-rigid editing without the fine-tuning, embedding optimization, and interpolation required by Imagic.The paper describes non-rigid editing of real images as particularly challenging.Source: §5.1 · p. 9

Results with T2I-Adapter

  • Integrating MasaCtrl with T2I-Adapter combines externally controlled target layouts with source-image contents for more faithful and fine-grained synthesis and editing.With strong extra guidance, the experiment starts attention control earlier, using S = 2 and L = 8.Source: §5.2 · pp. 9–10

Robustness to Other Models: Anything-V4

  • On Anything-V4, MasaCtrl preserves anime-image identity and background while allowing pose, action, and expression changes through prompt modification.The results are presented as evidence of consistent synthesis and generalizability to a domain-specific model.Source: §5.3 · p. 10

Extension to Video Synthesis

  • With coherent dense guidance, MasaCtrl synthesizes video frames with highly similar content, but primarily animates foreground objects rather than dynamic backgrounds.The extension uses T2I-Adapter or ControlNet with guidance such as pose or edges.Source: §5.4 · p. 10

Ablation Study

  • Ablations show that attention control after several denoising steps and in decoder layers preserves target layouts while retaining source contents.Early or all-layer control can reproduce the source too closely, while unsuitable low-resolution layers lose source contents or target layouts.Source: Figure 12 · p. 10, §5.5 · p. 11

6. Limitations and Discussion

MasaCtrl inherits Stable Diffusion’s dependence on generating suitable target layouts and has additional failures when target structures or contents differ substantially from the source.

Limitations and Discussion

  • MasaCtrl may fail when Stable Diffusion cannot generate the desired target layout or shape.Controllable strategies can alleviate this limitation but may still fail.Source: §6 · p. 11
  • When target layouts change drastically or contain unseen content, MasaCtrl can produce artifacts because the source image lacks queryable content for the desired result.The paper illustrates this with a target palm absent from the source image.Source: §6 · p. 11

7. Conclusion

The conclusion presents MasaCtrl as a tuning-free mutual self-attention control method for non-rigid consistent image synthesis and editing.

Conclusion

  • MasaCtrl applies mutual self-attention control to text-to-image diffusion models for non-rigid consistent image synthesis and editing.Source: §7 · p. 11

Related papers

Loading 2304.08465v1…