PaperlayerView exact revision on arXiv

Source-linked AI summary

Stable Flow: Vital Layers for Training-Free Image Editing

Omri Avrahami, Or Patashnik, Ohad Fried, Egor Nemchinov, Kfir Aberman, Dani Lischinski, Daniel Cohen-Or

arXiv:2411.14430v2cs.CVcs.GRcs.LG

TL;DR

DiT’s layer roles are unclear for image editing, and flow-based models have lower diversity than earlier diffusion models. Stable Flow detects vital layers and selectively injects attention features, adding latent nudging for real-image inversion. The paper demonstrates stable edits across multiple task types and reports that nudging reduces undesired changes during FLUX forward passes.

  • Problem

    DiT lacks the UNet’s intuitive layer structure, making it unclear which layers to modify for image editing.

    Source: §3.1 · p. 3

  • Method

    Stable Flow detects vital DiT layers by measuring the image impact of bypassing them, then selectively injects source-image attention features and uses latent nudging for real-image inversion.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    The same vital-layer set supports non-rigid deformations, object addition and replacement, and global modifications, while latent nudging makes FLUX less prone to undesired forward-pass changes.

    Source: §1 · p. 2, §3.2 · p. 4

  • Takeaways & Limitations

    Stable Flow is a training-free image-editing method that uses attention injection into the same vital-layer group across varied editing tasks.

    Source: §5 · p. 8

    The method struggles with style changes, object dragging, and complete background replacement without leakage.

    Source: §5 · p. 8

Abstract

from arXiv · show

Diffusion models have revolutionized the field of content synthesis and editing. Recent models have replaced the traditional UNet architecture with the Diffusion Transformer (DiT), and employed flow-matching for improved training and sampling. However, they exhibit limited generation diversity. In this work, we leverage this limitation to perform consistent image edits via selective injection of attention features. The main challenge is that, unlike the UNet-based models, DiT lacks a coarse-to-fine synthesis structure, making it unclear in which layers to perform the injection. Therefore, we propose an automatic method to identify "vital layers" within DiT, crucial for image formation, and demonstrate how these layers facilitate a range of controlled stable edits, from non-rigid modifications to object addition, using the same mechanism. Next, to enable real-image editing, we introduce an improved image inversion method for flow models. Finally, we evaluate our approach through qualitative and quantitative comparisons, along with a user study, and demonstrate its effectiveness across multiple applications. The project page is available at https://omriavrahami.com/stable-flow

1. Introduction

Flow-based models’ reduced diversity, usually undesirable, is leveraged for training-free image editing, but DiT’s architecture makes it unclear which layers to use. Stable Flow identifies vital layers for consistent prompt-directed edits and adds latent nudging for real-image inversion.

1. Introduction

  • Flow-based models have lower diversity than earlier diffusion models, which the paper proposes to leverage for training-free image editing.The paper attributes this reduced diversity to straight-line rather than curved trajectories.Source: §1 · p. 1
  • DiT lacks the UNet’s fine-coarse-fine structure, leaving it unclear which layers to modify for editing.Prior parallel-generation approaches inject source-image features into the edited-image trajectory, but layer roles are less understood in DiT.Source: §1 · pp. 1–2
  • Stable Flow identifies vital DiT layers by measuring image-content deviation when each layer is bypassed; these layers are spread across the transformer.The paper reports no simple relationship between layer vitality and architectural position.Source: §1 · pp. 1–2
  • Injecting source-image features only into vital layers tends to preserve unrelated image content while making prompt-specified changes.The paper demonstrates non-rigid edits, object addition, and scene changes using the same feature-injection mechanism.Source: §1 · p. 2
  • For real-image editing with FLUX, OOD nudging perturbs the clean latent before inverse ODE solving and makes forward passes less prone to undesired changes.The paper says the standard inverse Euler ODE solver does not reconstruct the input satisfactorily without the nudging technique.Source: §1 · p. 2
  • The authors compare their method with baselines qualitatively and quantitatively, reaffirm results with a user study, and demonstrate applications.Source: §1 · p. 2

2. Related Work

Prior text-driven editing work spans translation, localized editing, feature injection, per-image fine-tuning, and instruction-trained models. The paper identifies a gap in training-free support for multiple editing types and notes that standard inverse ODE solving is insufficient for flow-model inversion.

Text-Driven Image Editing

  • Prior text-driven editing methods include prompt-based translation, localized edits using masks, parallel-generation feature injection, per-image fine-tuning, and models trained on synthetic instruction-edit datasets.Some methods infer masks from the image or use an additional click; the cited approaches vary in whether they fine-tune models.Source: §2 · p. 2
  • The authors identify no prior training-free method supporting non-rigid editing, object adding or replacement, and scene editing altogether.Source: §2 · p. 2

Image Inversion

  • Image inversion seeks a generator latent that faithfully reconstructs an image; for flow models, the paper finds the standard inverse ODE solver insufficient and proposes latent nudging.The passage situates inversion work across GAN and diffusion-based models before describing the paper’s flow-model approach.Source: §2 · pp. 2–3

3. Method

Stable Flow identifies DiT layers that strongly affect image formation and selectively injects source-image attention features into those vital layers to support stable edits. For real images, latent nudging improves inversion reconstruction and helps confine edits to intended regions.

3.1. Measuring the Importance of DiT Layers

  • Vitality is measured by bypassing each layer and comparing its output with the full model using DINOv2 perceptual similarity; influential layers are distributed across DiT.The vital-layer set comprises layers whose measured vitality meets a threshold.Source: §3.1 · pp. 3–4
  • Removing vital layers can produce noise (G0), global structure and identity changes (G18), or texture and fine-detail changes (G56), unlike minor non-vital-layer effects.Source: §3.1 · p. 4, Figure 5 · p. 4

3.2. Image Editing using Vital Layers

  • Parallel source and edited trajectories replace the edited image's visual embeddings with source embeddings only in vital layers, supporting diverse edits with the same mechanism.The demonstrated edits include non-rigid deformations, object addition and replacement, and global modifications.Source: §3.2 · p. 4
  • Vital-layer attention favors visual features for regions to preserve and relevant text tokens for targeted edits, unlike non-vital-layer attention.Source: §3.2 · p. 4, Figure 6 · p. 5

3.3. Latent Nudging for Real Image Editing

  • For real-image editing, inverse Euler ODE inversion yields corrupted reconstructions and unintended changes; multiplying the initial latent by λ = 1.15 reduces reconstruction errors and constrains edits.The scalar perturbation is described as visually imperceptible.Source: §3.3 · p. 5, Figure 7 · p. 5

4. Experiments

Stable Flow compares favorably with baselines on image similarity and image-text direction similarity, and users prefer it across the evaluated criteria. Ablations, applications, and stated limitations further characterize its behavior.

4.1. Qualitative and Quantitative Comparison

  • In real-image examples, Stable Flow adheres to editing prompts while preserving identities, whereas baselines show difficulties with identity preservation, object addition, or non-rigid edits.Source: §4.1 · p. 6, Figure 8 · p. 7
  • The evaluation dataset contains 3,200 COCO-based samples spanning non-rigid editing, object addition, object replacement, and scene editing, assessed with CLIPimg, CLIPtxt, and CLIPdir.Source: §4.1 · p. 6
  • Stable Flow achieves better image and image-text direction similarity; SDEdit and MagicBrush follow text prompts more but struggle on those metrics.Source: §4.1 · p. 6, Table 1 · p. 6

4.2. User Study

  • Users prefer Stable Flow over the baselines for prompt adherence, image preservation, realism, and overall edit quality, with win rates above 50% in every category.The study uses a two-alternative forced-choice format.Source: §4.2 · p. 6, Table 3 · p. 6

4.3. Ablation Study

  • Injecting attention in all layers or extending attention in all layers harms text similarity, while non-vital-layer injection or removing latent nudging reduces image similarity.Source: Table 2 · p. 6, §4.3 · p. 8

4.4. Applications

  • Applications include iterative scene editing, generating new images in a given style, and text edits such as color, case, or wording changes.Source: §4.4 · p. 8
  • The method struggles to transform photorealistic images into animation, sketches, or oil paintings, drag a cat to another location, or fully replace a background.Source: Figure 10 · p. 8

5. Limitations and Conclusions

Stable Flow is a training-free method for multiple image-editing tasks using attention injection into the same vital-layer group. Its limitations include style changes, object dragging, and complete background replacement.

5. Limitations and Conclusions

  • The method struggles to change an input image from one style to another, such as photorealistic to oil painting.Source: §5 · p. 8
  • The method is unable to drag an object to a different image location, a task linked to text-to-image models’ difficulty with spatial prompt adherence.Source: §5 · p. 8
  • The method struggles to replace an image background entirely without leakage.Source: §5 · p. 8
  • Stable Flow performs multiple image-editing tasks through attention injection into the same vital-layer group.The authors suggest automatic vital-layer detection may also benefit generative-model pruning and distillation.Source: §5 · p. 8

Supplementary Material

The supplementary material documents the layer-identification setup and implementation, including the selected vital layers and their use for attention injection.

A.1. Method Implementation Details

  • The layer study used 64 ChatGPT prompts, corresponding generated images, and same-seed images with each layer bypassed.Images were generated with Euler sampling for 15 steps at guidance scale 3.5.Source: §A.1 · p. 14
  • Layers with the lowest DINOv2 perceptual similarity after bypass were designated vital; the listed sets are [0, 1, 2, 17, 18, 25, 28, 53, 54, 56] for FLUX.1-dev and [0, 7, 8, 9] for SD3.The authors empirically removed layer 2 from the FLUX.1-dev set.Source: §A.1 · p. 14
  • Reference and edited images are generated in parallel, with image embeddings selectively replaced only within the vital layers.Source: §A.1 · p. 14

A.2. Baselines Implementation Details

The supplementary material records baseline implementations and prompt formats, evaluation metrics and dataset construction, user-study procedures, and additional experiments.

A.2. Baselines Implementation Details

  • Comparisons used SDEdit, P2P+NTI, Instruct-P2P, MagicBrush, and MasaCTRL; SDEdit was reimplemented with FLUX.1-dev and the other baselines used official implementations.Prompts were scene descriptions for SDEdit, P2P+NTI, and MasaCTRL, and instructional formats for Instruct-P2P and MagicBrush.Source: §A.2 · p. 14

A.3. Automatic Metrics Implementation Details

  • The COCO validation evaluation set contains 3,200 samples across non-rigid editing, object addition, object replacement, and scene editing, with prominent humans or animals occupying 5%–33% of each image.CLIPimg measures input-image similarity, CLIPtxt target-prompt similarity, and CLIPdir alignment between prompt-change and image-change directions.Source: §A.3 · p. 15

A.4. User Study Details

  • In two-alternative forced-choice trials, users compared our method with a baseline on prompt adherence, input preservation, realism, and overall quality.Five ratings per sample yielded 320 ratings per baseline and 1,920 responses; a binomial test suggested statistical significance (p-value < 5%).Source: §A.4 · p. 15

B. Additional Experiments

  • Additional experiments cover method comparisons, perceptual metrics, vital-layer-set sizes, latent nudging, layer-bypass visualizations, and evaluation on SD3.Source: B. Additional Experiments · p. 16

B.1. Additional Comparisons and Results

Additional comparisons report Stable Flow’s qualitative editing behavior, ablation outcomes, and further results using the same vital-layer set.

B.1. Additional Comparisons and Results

  • Stable Flow adheres to editing prompts while preserving object identities, whereas baselines show difficulties with identity preservation, adding objects, or non-rigid edits.The comparisons use real COCO images.Source: §B.1 · p. 16, Figure 14 · p. 17
  • Injecting or extending attention in all layers encourages input-image copying over the target prompt, while non-vital-layer extension or removing latent nudging reduces input-image similarity.Source: §B.1 · pp. 16, 22, Figure 15 · p. 18
  • Additional examples use the same vital-layer set for varied edits, including changing text in an image while preserving its background and style.Source: Figure 16 · p. 19, Figure 17 · p. 20, Figure 18 · p. 21
  • Layer removal has varying effects: some layers substantially alter generated images, influential layers are spread across the transformer, and the first vital layers are omitted from the figure because similarity approaches zero.Source: Figure 19 · p. 22

B.2. Different Perceptual Metrics

The appendix tests whether the detected vital-layer set depends on the perceptual metric and whether metric-driven set differences matter in practice. DINOv1 and DINOv2 agree on the set, while editing differences across the alternative sets are negligible.

B.2. Different Perceptual Metrics

  • DINOv1 and LPIPS removal comparisons show that influential layers are distributed across the transformer, not concentrated in specific regions.Lower perceptual similarity after removing a layer indicates a more significant change in generated images.Source: Figure 20 · p. 22
  • Vital-layer sets from DINOv1 and DINOv2 are equivalent, and edits using the slightly different metric-derived sets show negligible differences.CLIP and LPIPS replace two layers in the set, while retaining most of the DINO-selected layers.Source: §B.2 · p. 22

B.3. VLM-Based Quantiative Metric

The appendix evaluates editing with a VLM-based metric and tests how many detected vital layers can be omitted. The VLM-based metric follows the CLIP-based trend, while removing more than 20% of vital layers produces unintended changes.

B.3. VLM-Based Quantiative Metric

  • The VLM-based metric follows the CLIP-based trend: P2P+NTI, Instruct-P2P, and MasaCTRL have low text-prompt similarity, while SDEdit and MagicBrush struggle to avoid unintended changes.The metric counts “yes” answers for prompt adherence and whether only prompt-specified parts changed.Source: §B.3 · pp. 22–23

B.4. Number of Vital Layers

  • Removing 20% of the vital-layer set has negligible effects, but removing more causes unintended identity or background changes.The results align with the finding that the least-vital layers are less important for editing.Source: §B.4 · p. 23

B.5. Latent Nudging Experiments

The appendix examines latent nudging for real-image inversion and visualizes layer bypassing in FLUX. Nudging at λ = 1.15 supports reconstruction, while the bypassing study finds influential layers throughout the transformer.

B.5. Latent Nudging Experiments

  • With 1,000 inversion and denoising steps, λ = 1.0 still reconstructs poorly, λ = 1.15 is the smallest value enabling full reconstruction, and λ = 3.0 yields saturation.Values below 1.0 severely damage reconstruction quality.Source: §B.5 · p. 23
  • Latent caching alone achieves perfect inversion but can alter images or fail to preserve identity during editing; caching with nudging permits inversion in 50 steps.The authors use latent caching in addition to nudging in practice.Source: §B.5 · p. 23

B.6. Layer Bypassing Visualization

  • Metric-derived vital-layer sets differ slightly, but the editing differences between them are negligible.DINOv1 and DINOv2 produce the same set; CLIP and LPIPS replace two layers.Source: Figure 22 · p. 24
  • Removing 20% of the vital-layer set causes negligible changes, whereas removing more leads to unintended identity and background changes.The examples include identity changes in man and woman images and background changes in cat and blackboard images.Source: Figure 23 · p. 25
  • Across the FLUX bypassing visualizations, removing some layers substantially affects generated images, while influential layers are spread across the transformer.The study bypasses each layer using its residual connection and generates images with the same prompts and seeds as the reference set.Source: §B.6 · pp. 23, 25

B.7. Stable Diffusion 3 Results

The appendix tests Stable Flow on Stable Diffusion 3, a different DiT flow model from the FLUX.1-dev backbone used in the main experiments. Bypassing vital layers causes substantial image disruption, and the same reference-feature injection mechanism supports editing on this model.

B.7. Stable Diffusion 3 Results

  • Removing certain Stable Diffusion 3 layers substantially changes generated images, while removing others has minimal impact.The appendix measures perceptual similarity between generated images with and without each layer; lower similarity indicates larger changes.Source: §B.7 · p. 25, Figure 34 · p. 35
  • On Stable Diffusion 3, bypassing non-vital layers causes modest alterations, while bypassing vital layers produces noise or severe distortions.The examples include complete noise for G0 and severe distortions for G7, G8, and G9.Source: §B.7 · pp. 25–26, Figure 35 · p. 36
  • The authors demonstrate multiple Stable Flow editing operations on Stable Diffusion 3 by injecting reference-image information into vital layers.The main experiments used FLUX.1-dev; this appendix also tests the Diffusers medium implementation of Stable Diffusion 3.Source: §B.7 · pp. 25–26, Figure 36 · p. 37

Related papers

Loading 2411.14430v2…