Source-linked AI summary
Semantic Image Inversion and Editing using Rectified Stochastic Differential Equations
Litu Rout, Yujia Chen, Nataniel Ruiz, Constantine Caramanis, Sanjay Shakkottai, Wen-Sheng Chu
TL;DR
Real-image inversion must preserve faithfulness while enabling prompt-based editing, but diffusion inversion can be costly and rectified-flow inversion remains underexplored. The paper introduces an LQR-based controlled RF inversion method, proves its stochastic interpretation, and reports state-of-the-art zero-shot inversion and editing results, while noting limited comparison with expensive diffusion editors.
Problem
Real-image inversion must recover structured noise that preserves faithfulness and editability, while existing diffusion approaches face costly discretization or optimization and RF inversion is underexplored.
Method
The paper constructs controlled rectified-flow inversion using an LQR-derived optimal controller, proves an equivalent rectified SDE formulation, and extends the framework to Flux.
Results
89% more realism than NTI+P2P is reported for LSUN-bedroom stroke-to-image generation, while the method remains faithful to the input stroke and achieves state-of-the-art zero-shot performance.
Takeaways & Limitations
The framework provides an efficient, training-free route to inversion and editing for rectified-flow models including Flux.
Takeaways & Limitations
Comparison with expensive diffusion-based editing solutions is limited because they are unavailable for Flux or not directly applicable to its multimodal architecture.
Abstract
from arXiv · showhide
Generative models transform random noise into images; their inversion aims to transform images back to structured noise for recovery and editing. This paper addresses two key tasks: (i) inversion and (ii) editing of a real image using stochastic equivalents of rectified flow models (such as Flux). Although Diffusion Models (DMs) have recently dominated the field of generative modeling for images, their inversion presents faithfulness and editability challenges due to nonlinearities in drift and diffusion. Existing state-of-the-art DM inversion approaches rely on training of additional parameters or test-time optimization of latent variables; both are expensive in practice. Rectified Flows (RFs) offer a promising alternative to diffusion models, yet their inversion has been underexplored. We propose RF inversion using dynamic optimal control derived via a linear quadratic regulator. We prove that the resulting vector field is equivalent to a rectified stochastic differential equation. Additionally, we extend our framework to design a stochastic sampler for Flux. Our inversion method allows for state-of-the-art performance in zero-shot inversion and editing, outperforming prior works in stroke-to-image synthesis and semantic image editing, with large-scale human evaluations confirming user preference.
1 INTRODUCTION
The paper targets faithful, editable inversion of real images and introduces a zero-shot Rectified Flow approach using controlled dynamics and optimal control. It reports improved inversion and editing performance without added training or optimization.
- Image inversion must recover structured noise that reconstructs the reference faithfully and remains editable through new prompts.
- Diffusion-model inversion is challenged by expensive fine discretization and trajectory nonlinearities that reduce reconstruction faithfulness.
- The method constructs a controlled forward ODE from the input image and guides the reverse ODE with an LQR-derived optimal controller.
- The resulting vector field has a stochastic interpretation, and the framework extends to a stochastic sampler for Flux.
- The method improves photo realism in stroke-to-image generation by 89% over Mokady et al. while maintaining faithfulness to the input stroke.
- The approach requires no additional training, latent optimization, prompt tuning, or complex attention processors.
2 RELATED WORKS
Prior inversion and editing methods were designed mainly for diffusion models, while rectified-flow inversion remains underexplored and does not directly transfer to Flux. The paper situates its method among these limitations and connects RF sampling with optimal control and SDEs.
- DM Inversion: Optimization-free methods such as SDEdit avoid extra training but can be less faithful because they add noise to the image in one step.
- DM Inversion: DDIM inversion recursively predicts noise but can deviate from the original image because of nonlinear drift and diffusion and inexact score estimates.
- DM Inversion: Prompt-embedding and latent-optimization approaches reduce inversion deviation but have high time complexity.
- Challenges in RF Inversion: Existing diffusion-oriented inversion and editing methods do not directly apply to Flux because MM-DiT entangles text and image information unlike UNet cross-attention.
- Challenges in RF Inversion: Flux also uses T5, which lacks the image-aligned latent space available from CLIP encoders, requiring investigation of modern T2I inversion.
- Filtering, Control and SDEs: The paper studies rectified flows for conditional sampling and shows that their resulting drift field has an optimal-control interpretation.
3 METHOD
The method uses rectified-flow inversion with optimal control to balance reconstruction fidelity against typicality and editability. Its controlled ODE is connected to rectified SDEs, yielding exact recovery in limiting cases and tunable image editing.
- Rectified flows: Rectified flows generate images by integrating a learned vector field from Gaussian noise through an ODE.The reverse vector field can recover the original noise when the generative field is precisely known.
- Exact inversion: Exact RF inversion recovers a given image by simulating the reversed vector field to obtain structured noise, then applying the generative ODE.This guarantee assumes the vector field is precisely known.
- Controlled inversion: Corrupted or atypical images produce atypical noise, motivating a controller that converts any image into noise typical of the source distribution.The resulting typical noise produces a clean but potentially unrelated image, so inversion must balance competing objectives.
- Optimal control: The LQR solution equals the rectified linear-path conditional vector field, providing a minimum-energy controller toward a target noise sample.Its quadratic cost combines control energy with terminal deviation weighted by λ.
- Guidance trade-off: The controlled ODE interpolates between image consistency and clean-distribution consistency through guidance γ, while retaining comparable inversion accuracy.At γ=1, the controller drives toward typical noise and improves generation quality; suitable γ preserves faithfulness while applying edits.
4 ALGORITHM: INVERSION AND EDITING VIA CONTROLLED ODES
The algorithm performs inversion followed by text-guided editing using controlled ODEs and a pretrained Flux model. Controller guidance balances preserving reference content with applying the requested semantic change.
- Pipeline: The pipeline has two stages: compute structured noise by inversion, then edit the reference content with a desired text prompt.Both stages use the paper’s controlled ODE formulation.
- Problem setup: For corrupted guides, the objective is to produce a realistic image under p0 while remaining faithful to the original guide.The guides contain color strokes conveying high-level details and are typically not realistic samples.
- Problem setup: For clean guides, text-guided edits such as changing age or gender should preserve the reference image’s content.The input consists of a real image and an accompanying editing prompt.
- Inversion: During inversion, controlled ODE (8) starts from the reference image and uses Flux with a null prompt to compute structured noise.The conditional vector field is obtained analytically from the controller.
- Editing: During editing, controlled ODE (15) uses the desired prompt, with higher η favoring faithfulness and lower η permitting larger edits.η provides a smooth interpolation between faithfulness and editability.
5 EXPERIMENTAL EVALUATION
The evaluation compares RF inversion with diffusion-based baselines on stroke-to-image generation and semantic editing, showing gains in realism, faithfulness, efficiency, and user preference.
- The method is evaluated on LSUN-church, LSUN-bedroom, and SFHQ across Stroke2Image generation and semantic image editing.
- Stroke2Image generation: Controlled ODE inversion produces structured noise consistent with corrupted inputs and the invariant terminal distribution, yielding more realistic Stroke2Image outputs.
- Stroke2Image generation: On LSUN bedroom, the method is 4.7% more faithful and 13.79% more realistic than SDEdit-SD1.5, and 89% more realistic than NTI+P2P.
- User evaluations report at least 59.67% higher overall satisfaction than every baseline across 9,000 comparisons from 126 participants.
- Semantic image editing: The approach performs prompt-based stylization, age and gender editing, and sequential object insertion without additional optimization or complex attention processors.
- Semantic image editing: For face editing, the method exceeds optimization-free methods by at least 29% in face reconstruction, 6.6% in DINO patch-wise similarity, and 26.4% in CLIP-Image similarity.
6 CONCLUSION
The paper presents an efficient rectified-flow inversion and editing method based on interpolating unconditional and controlled vector fields, with theoretical and empirical validation.
- The method targets rectified-flow models such as Flux and requires no additional training, latent optimization, prompt tuning, or complex attention processors.
- Its vector field balances consistency with a possibly corrupted input against consistency with the distribution of clean images.
- The authors prove equivalence between this vector field and a rectified stochastic differential equation, and report state-of-the-art zero-shot inversion and editing.
- The paper demonstrates stroke-to-image synthesis, face editing, object insertion, and stylization, with large-scale human evaluation confirming user preference.
- A stated limitation is the lack of comparison with expensive diffusion-based editing solutions because they are unavailable for or not directly applicable to Flux.
BROADER IMPACT STATEMENT
The paper identifies both beneficial and harmful social impacts of semantic image inversion and editing, and mentions filters and watermarking as mitigation measures.
- The technology can generate photo-realistic images from high-level descriptions and modify attributes such as age, gender, or facial accessories.
- The same capabilities can be misused to manipulate photographs of individuals with inappropriate or offensive edits.
- The authors enable NSFW filters and suggest watermarking generated images to reduce misuse in inversion and editing applications.
A ADDITIONAL THEORETICAL RESULTS
The appendix establishes theoretical links between rectified-flow ODEs and equivalent SDEs, including a reverse SDE that enables stochastic sampling for models such as Flux.
- Rectified-flow theory: Lemma A.1 formalizes the standard rectified-flow ODE using its optimal vector field.The derivation starts from the linear interpolation between coupled endpoint samples.
- ODE–SDE equivalence: The rectified-flow ODE has the same marginal density evolution as an SDE with suitable drift and diffusion terms.This equivalence follows from the relationship between continuity and Fokker–Planck equations.
- Distributional result: The stationary distribution of the equivalent SDE converges to N(0, I) as t →1.This establishes the intended Gaussian noise endpoint for the forward process.
- Reverse sampling: Time reversal yields a rectified SDE that transforms noise into images and provides a stochastic sampler for rectified-flow models.The construction applies over t ∈[δ, 1] for any small δ ∈(0, 1).
- Reverse sampling: The sampler is practically implementable for Flux because the rectified-flow vector field gives an explicit formula for computing the score function.That score determines the drift and diffusion coefficients of the reverse SDE.
B.3 PROOF OF THEOREM 3.4
The proofs derive controlled vector fields and show their associated density evolutions match the relevant rectified SDEs after applying continuity, Fokker–Planck, and time-reversal arguments. The section also records implementation details and baseline requirements for experiments.
- Forward control proof: The controlled forward ODE is rewritten using a conditional vector field and a score-function term before its density evolution is analyzed.The proof then uses the continuity equation to connect the controlled ODE to an SDE formulation.
- Forward control proof: The conditional vector field is derived from the flow-matching objective through a minimum mean squared estimator and Tweedie’s formula.Linearity of expectation yields the vector-field relation used in the density calculation.
- Reverse control proof: For the reverse process, the controlled vector field is obtained from a modified LQR problem and combined with the score term to form the reverse dynamics.The resulting continuity equation is matched to the corresponding stochastic formulation.
- Reverse control proof: The reverse SDE proof shows that its Fokker–Planck equation agrees with the forward stochastic equivalent after time reversal.The argument tracks the marginal density q_t and uses the relationship between Y_t and X_t.
- Experimental setup: The experiments compare the proposed method with DDIM, NTI, P2P, SDEdit, InstructPix2Pix, and LEDIT++, using reconstruction, realism, and editing metrics.The method requires no additional training, prompt-embedding optimization, or attention manipulation scheme.
- Implementation: The implementation uses controlled forward and reverse ODEs with tunable guidance parameters γ and ηt, producing structured noise and edited images.The schedules are specified by starting time, stopping time, and guidance strength.
C.2 ABLATION STUDY
The ablations show that controller guidance improves faithfulness and smoothly interpolates between unconditional and conditional generation, while RF inversion reconstructs samples more accurately than DDIM in the synthetic study.
- Controller guidance: Increasing the starting time s improves faithfulness when controller guidance is absent.The controlled reverse ODE is initialized from a state obtained by integrating the controlled forward ODE.
- Controller guidance: Increasing the stopping time τ strengthens guidance toward the original image, with a phase transition around τ = 0.14 = 4/28.Beyond this transition, the conditional vector field dominates and drives the reverse flow toward the terminal image.
- Controller guidance: Increasing η improves faithfulness while producing a smooth interpolation between unconditional and conditional vector fields.The visual transition runs from “A young man” at η = 0 to the original image at the strongest guidance.
- Synthetic reconstruction: RF inversion achieves lower L2 and L1 reconstruction error than DDIM inversion in the synthetic experiment.The minimum error occurs for γ = η = 0, corresponding to reversing the standard rectified flows.
- Synthetic reconstruction: The stochastic equivalents of rectified flows recover the original samples more accurately than diffusion models, while controller guidance further reduces RF reconstruction error.The comparison uses the simulated ODE and SDE samplers described for the synthetic Gaussian setting.
C.4 ADDITIONAL RESULTS ON STROKE2IMAGE GENERATION
Additional results show that the method produces realistic Stroke2Image outputs and preserves editability under corrupted initialization, while controlled paths guide samples toward reference images.
- Stroke2Image generation: The method generates more realistic Stroke2Image results than leading training-free editing approaches and is competitive with InstructPix2Pix.The comparison includes optimization-based NTI and attention-based NTI+P2P.
- Sample paths: RF sample paths are straighter than diffusion-model paths, supporting coarse discretization and faithful reconstruction.The figure compares forward and reverse processes across the two model families.
- Controlled paths: The controlled ODE and SDE paths steer noise toward terminal states during inversion and guide generated states toward the reference image.The controlled paths significantly reduce reconstruction error in the illustrated examples.
- Robustness: Under corrupted initialization, the method reconstructs a similar stroke input while retaining editability with a new prompt.The comparison highlights that other methods can preserve appearance but do not produce similarly editable results.
C.5 ADDITIONAL RESULTS ON SEMANTIC IMAGE EDITING
The method supports controlled semantic editing, object insertion, and stylization while preserving relevant reference content and style. Qualitative results and human evaluation assess these capabilities across multiple settings.
- Gender editing: The method smoothly interpolates between “A man” and “A woman” while gradually morphing facial expression and hairstyle.
- Age editing: Reducing controller strength ηt regulates age editing, making a young woman appear older and an old man appear younger.
- Object insertion: Sequential text-guided insertion adds pepperoni, mushrooms, and green leaves without deleting previously inserted objects, followed by lego-style presentation.
- Reference stylization: Prompt-based facial-expression edits preserve the identity of a stylized girl while generating expressions such as surprise.
- Reference stylization: A single reference style image transfers melting elements, golden color, and 3d rendering across 12 text prompts spanning living and non-living objects.
- Reference stylization: Stylization experiments use text prompts to specify both generated content and the style represented by different reference images.