Source-linked AI summary
FlowEdit: Inversion-Free Text-Based Editing Using Pre-Trained Flow Models
Vladimir Kulikov, Matan Kleiner, Inbar Huberman-Spiegelglas, Tomer Michaeli
TL;DR
Inversion-based text editing can lose source-image fidelity and often requires architecture-specific interventions. FlowEdit instead directly transports between source and target distributions with an inversion-free, optimization-free, model-agnostic ODE, achieving lower transport costs and state-of-the-art results on FLUX and Stable Diffusion 3.
Problem
Inversion-based editing often inadequately preserves source-image structure, while intervention-based fixes are not easily transferable across model architectures and sampling schemes.
Method
FlowEdit constructs a direct ODE path between source and target distributions without inversion, Gaussian traversal, optimization, or intervention in model internals.
Results
FlowEdit achieves lower transport costs on synthetic datasets and state-of-the-art performance across editing tasks evaluated with FLUX and Stable Diffusion 3.
Takeaways & Limitations
The method offers transferable text-based editing with strong structure preservation for precise editing tasks.
Takeaways & Limitations
The optimization viewpoint is unsupported because DDS iterations can increase their loss, and continued iterations can deteriorate edited-image quality.
Abstract
from arXiv · showhide
Editing real images using a pre-trained text-to-image (T2I) diffusion/flow model often involves inverting the image into its corresponding noise map. However, inversion by itself is typically insufficient for obtaining satisfactory results, and therefore many methods additionally intervene in the sampling process. Such methods achieve improved results but are not seamlessly transferable between model architectures. Here, we introduce FlowEdit, a text-based editing method for pre-trained T2I flow models, which is inversion-free, optimization-free and model agnostic. Our method constructs an ODE that directly maps between the source and target distributions (corresponding to the source and target text prompts) and achieves a lower transport cost than the inversion approach. This leads to state-of-the-art results, as we illustrate with Stable Diffusion 3 and FLUX. Code and examples are available on the project's webpage.
1. Introduction
Pre-trained diffusion/flow models are widely used for real-signal editing, but inversion often fails to preserve source fidelity and additional interventions limit transferability. FlowEdit instead directly connects source and target distributions with an inversion-free path designed for lower transport cost.
- Motivation: Inversion extracts a noise vector before generating the edited signal, but this common optimization-free workflow often provides insufficient source fidelity.The problem persists even for generated images with precisely known initial noise.
- Motivation: Feature-map injection can improve fidelity, but these interventions are not easily transferable across model architectures and sampling schemes.
- FlowEdit: FlowEdit constructs a direct path between source and target distributions instead of mapping the image to noise and back.The source and target distributions correspond to the source and target text prompts.
- FlowEdit: The proposed path is shorter than inversion’s path, which the paper states helps maintain better fidelity to the source image.
2. Related work
Prior real-image editing methods either optimize the image at test time or invert it into noise before editing. FlowEdit avoids both inversion and optimization while remaining adaptable to new models.
- Optimization-based methods: Optimization-based editing uses a pretrained text-to-image model as a loss to push an image toward a user-provided prompt.These methods are resource intensive at test time.
- Inversion-based methods: Most optimization-free editing methods begin with image-to-noise inversion, but naive inversion generally produces noise maps unsuitable for effective editing.
- Inversion-based methods: Improving inversion accuracy does not fully resolve editing quality, since exact inversion on synthetic data can still produce unsatisfactory edits.
- FlowEdit: FlowEdit maps source to target distributions without traversing Gaussian noise, avoids optimization and model-internal intervention, and adapts to new models.
3. Preliminaries
Flow models generate samples by solving time-dependent ODEs from noise, while text-conditioned velocity fields control the target distribution. The editing setup applies source- and target-conditioned trajectories to transform an image.
- Flow models: Generative flow models transport between distributions of random vectors through an ODE defined over time t ∈ [0, 1].
- Flow models: A time-dependent velocity field maps a boundary sample at t = 1 to samples from the data distribution at t = 0, commonly starting from Gaussian noise.
- Rectified Flow models: Rectified flows train marginals as linear interpolations between X0 and X1, producing relatively straight sampling paths.The straight paths permit solving the ODE with a small number of discretization steps.
- Image editing using ODE inversion: In inversion-based editing, the source image is mapped to noise using the source-conditioned forward ODE, then mapped back using the target-conditioned reverse ODE.
- FlowEdit: FlowEdit instead follows a noise-free direct path, adding random noise to source estimates at each timestep while averaging velocity predictions.
4. Reinterpretation of editing by inversion
Editing by inversion can be rewritten as a direct ODE, but its induced source-target pairings may be inefficient because the path inherits the detour through Gaussian noise. The reinterpretation motivates a lower-cost direct alternative.
- Direct-path reinterpretation: Inversion-based editing transports between source and target distributions through Gaussian noise, yet can be expressed as a direct path between those distributions.
- Transport cost: A Gaussian-mixture example shows that inversion can intermingle source modes rather than pairing each source mode with its closest target mode.The resulting pairings imply higher transport cost.
- Direct-path reinterpretation: The direct path is defined by an ODE obtained by differentiating the inversion coupling and substituting the source- and target-conditioned dynamics.
- Direct-path reinterpretation: The inversion-derived path is noise-free because source- and target-conditioned velocities remove roughly the same noise component.Their difference therefore captures differences between clean image predictions.
- Path behavior: Along the path, coarse structures change first and fine textures are updated later as the noise level decreases.This produces an autoregressive coarse-to-fine evolution.
5. FlowEdit
FlowEdit replaces inversion with a direct, lower-cost path between source and target distributions, using averaged velocity fields from random pairings. Its practical procedure uses discrete ODE timesteps and supports controllable edit strength, while experiments show improved distribution alignment and qualitative editing results.
- Method: FlowEdit maps source to target distributions through a direct path rather than traversing Gaussian noise, targeting lower transport cost than inversion.The method is motivated by alternative pairings that map source modes toward nearby target modes.
- Method: Multiple random pairings with the same marginals are used to compute averaged velocity directions for the editing ODE.The practical algorithm approximates the expectation with navg model predictions at each timestep.
- Practical considerations: FlowEdit initializes at a selected timestep, with larger nmax traversing the full path for stronger edits and smaller nmax shortening it.The process is driven by a discrete set of timesteps, and skipped initial steps produce weaker edits.
- Qualitative results: FlowEdit preserves original structure while following target prompts across diverse edits and qualitative comparisons with SD3 and FLUX.The reported examples include localized edits, multiple-object modifications, and pose changes.
- Relation to optimization-based methods: FlowEdit differs from optimization-based methods and is not naturally explained as optimization because its iterations can worsen the associated loss or image quality.The comparison discusses DDS and PDS, whose updates are based on forward-pass approximations to loss gradients.
- Experiments: 51.14 vs. 55.88 FID and 0.017 vs. 0.023 KID favor FlowEdit over inversion on the cat-to-dog experiment.Both metrics compare edited dog images with 1000 generated dog images.
6. Experiments
FlowEdit is evaluated on diverse real-image edits with SD3 and FLUX, showing strong simultaneous text adherence and structure preservation. Quantitative and style-editing experiments examine this balance and its trade-offs.
- Experimental setup: The experiments use official SD3 medium and FLUX.1 dev weights, with 50 and 28 sampling steps, respectively.SD3 uses nmax = 33, while FLUX uses nmax = 24.
- Experimental setup: The dataset contains over 70 diverse real images and more than 250 text-image pairs with handcrafted target prompts.Images come from DIV2K and royalty-free online sources.
- Qualitative evaluation: FlowEdit consistently adheres to target prompts while preserving source structure across qualitative edits with both SD3 and FLUX.Examples include localized edits, multiple-object modifications, pose changes, and changing sign text while preserving cars and backgrounds.
- Quantitative evaluation: LPIPS measures semantic structure preservation and CLIP measures text adherence; FlowEdit achieves a favorable balance across varied hyperparameters.Other methods tend either to preserve structure with weak edits or modify images while disregarding original semantics.
- Text-based style editing: FlowEdit enables style changes by allowing slight deviation from the original structure, using modified final sampling steps for greater stylistic flexibility.The style-editing results use both SD3 and FLUX.
7. Conclusion and limitations
The paper concludes that FlowEdit provides inversion-free, optimization-free, model-agnostic editing through a direct source-to-target ODE. It reports lower transport costs and state-of-the-art results, while identifying limits in balancing CLIP and LPIPS and in large-region edits.
- Conclusion: FlowEdit directly maps source to target distributions without traversing the standard Gaussian distribution used by inversion-based editing.The method is inversion-free, optimization-free, and model agnostic.
- Conclusion: Synthetic evaluations show lower transport costs, which correspond to stronger structure preservation in image editing.The paper illustrates state-of-the-art performance with FLUX and SD3.
- Limitations: FlowEdit often struggles with substantial modifications to large image regions, including pose and background editing.Increasing nmin can help, but is often insufficient for these edits.
- Limitations and comparisons: On FLUX, ODE Inversion can obtain high CLIP at high LPIPS, while lower LPIPS settings reduce text adherence; RF-Inversion with η = 1.0 also struggles with text adherence.Additional comparisons report that FlowEdit achieves the best balance.
B.3. Metrics comparisons
The appendix details evaluation metrics and controls for editing strength and structure deviation. It shows how nmax and nmin trade off source preservation against target adherence and stylistic change.
- Metrics: DreamSim and LPIPS use lower scores for better structure preservation, whereas CLIP-image and DINO cosine similarities use higher scores.CLIP-T measures text adherence in the metric tables.
- Metrics comparisons: FlowEdit is reported as the only method that both adheres to the text prompt and preserves the source structure across the evaluated metrics.The appendix reports these results for SD3 and FLUX.
- Edit-strength control: nmax controls edit strength: nmax = T traverses the full edit path, while nmax < T skips initial steps and shortens it.Larger nmax produces stronger edits, whereas smaller values retain more of the original image.
- Style-editing control: For style editing, increasing nmin permits stronger high-frequency modifications but increases deviation from the original structure.Small nmin values preserve structure and can weaken alignment with the target style.
G. Limitations
FlowEdit’s strong structure preservation limits substantial modifications to large image regions. The limitation is illustrated for pose and background editing, where target-prompt changes remain incomplete.
- Limitations: FlowEdit often cannot substantially modify large image regions such as pose and background.These cases are illustrated in Figure S6.
- Limitations: Increasing nmin can create greater deviation from the source, helping style edits but often remaining insufficient for background and pose changes.In these cases, the method may fail to fully modify the image according to the target prompt and preserve source identity.
H. Effect of random noise on the results
FlowEdit’s added noise can produce diverse edits for the same text-image pair, but different noise realizations can also cause artifacts and failures.
- Different random seeds produce diverse edits, such as varied bonsai-tree transformations and tent locations.With few averaging samples, FlowEdit is effectively stochastic.
- White-horse-to-brown-horse edits sometimes produce horses with more than four legs or other artifacts.These failures are attributed to changes caused by different added-noise realizations.
- Source-prompt variations and omission have little effect on FlowEdit results.The experiments compare generated source-prompt variations and an empty source prompt; the table caption identifies CLIP-T and structure-preservation scores.
J. Illustration of the noise-free path between the source and target distributions
The noise-free FlowEdit path directly evolves source images toward target images in a coarse-to-fine manner, while experiments compare its behavior with inversion and optimization-based alternatives.
- J. Illustration of the noise-free path between the source and target distributions: The noise-free path directly connects source and target states and supports iterative editing without traversing Gaussian noise.The path is described as autoregressive and coarse-to-fine, ending at the target state.
- J. Illustration of the noise-free path between the source and target distributions: At large timesteps, the update captures mainly low frequencies; as t decreases, higher-frequency details emerge.This establishes an autoregressive coarse-to-fine evolution from source to target.
- J. Illustration of the noise-free path between the source and target distributions: FlowEdit preserves structure and fine details through repeated edits, unlike the illustrated inversion path.The cat-to-lion examples specifically retain grass around the cat in FlowEdit results.
- J. Illustration of the noise-free path between the source and target distributions: Increasing navg substantially improves LPIPS and CLIP when T = 10 and nmax = 7, but has little effect when T = 50 and nmax = 33.For T = 10 and nmax = 7, LPIPS decreases by ∼0.3 and CLIP increases by 0.035; the default configuration uses navg = 1.
- J. Illustration of the noise-free path between the source and target distributions: Sequential timesteps keep model inputs within the trained distribution, unlike random timestep sampling that can create out-of-distribution inputs.The text gives a slightly noisy cat under the “lion” condition as an example.
- J. Illustration of the noise-free path between the source and target distributions: DDS iterations tend to increase, rather than decrease, their denoising loss, and continued optimization deteriorates editing results.This challenges the optimization interpretation of DDS.
- J. Illustration of the noise-free path between the source and target distributions: FlowEdit performance deteriorates when its step size differs from dt = ti−1 −ti, with only c = 1 balancing CLIP and LPIPS favorably.Even slight deviations from c = 1 cause a significant performance drop.
- J. Illustration of the noise-free path between the source and target distributions: The mathematical differences between FlowEdit, DDS, and PDS become clearer when their update steps are expressed in diffusion terms.The paper also argues that the optimization viewpoint for DDS and PDS is not fully justified.
M. Additional details about the Cats-Dogs experiment
The Cats-Dogs experiment compares FlowEdit with editing by inversion on synthetic cat-to-dog edits and evaluates alignment with generated dog images.
- M. Additional details about the Cats-Dogs experiment: The synthetic dataset contains 1000 cat images generated from 1000 Llama3-produced variations of “a photo of cat”.Corresponding dog prompts replace “cat” with “dog”.
- M. Additional details about the Cats-Dogs experiment: FlowEdit achieves lower transport cost than editing by inversion on both reported metrics.The comparison uses cat images edited into dogs with matched source and target prompt templates.
- M. Additional details about the Cats-Dogs experiment: FlowEdit achieves lower FID and lower KID against generated dog images, indicating alignment with the target distribution.The generated dog set uses SD3 and the same prompts with “cat” replaced by “dogs”.