Source-linked AI summary
Taming Rectified Flow for Inversion and Editing
Jiangshan Wang, Junfu Pu, Zhongang Qi, Jiayi Guo, Yue Ma, Nisha Huang, Yuxin Chen, Xiu Li, Ying Shan
TL;DR
Rectified-flow diffusion transformers generate high-quality images and videos but remain inaccurate for inversion, limiting downstream editing. The paper proposes RF-Solver, a training-free higher-order ODE sampler, and RF-Edit, a feature-sharing editing framework; experiments show versatility across generation, inversion, reconstruction, and editing.
Problem
Rectified-flow models have strong generation performance, but inversion inaccuracies limit their use in reconstruction and downstream image and video editing.
Method
RF-Solver uses an exact rectified-flow ODE formulation with higher-order Taylor expansion, while RF-Edit shares inversion self-attention value features during denoising.
Results
Experiments demonstrate improved inversion, reconstruction, sampling, and editing performance across image and video tasks, with RF-Edit achieving high-quality edits while preserving source structure.
Takeaways & Limitations
The training-free approach is compatible with rectified-flow models including FLUX and OpenSora without additional training or optimization.
Abstract
from arXiv · showhide
Rectified-flow-based diffusion transformers like FLUX and OpenSora have demonstrated outstanding performance in the field of image and video generation. Despite their robust generative capabilities, these models often struggle with inversion inaccuracies, which could further limit their effectiveness in downstream tasks such as image and video editing. To address this issue, we propose RF-Solver, a novel training-free sampler that effectively enhances inversion precision by mitigating the errors in the ODE-solving process of rectified flow. Specifically, we derive the exact formulation of the rectified flow ODE and apply the high-order Taylor expansion to estimate its nonlinear components, significantly enhancing the precision of ODE solutions at each timestep. Building upon RF-Solver, we further propose RF-Edit, a general feature-sharing-based framework for image and video editing. By incorporating self-attention features from the inversion process into the editing process, RF-Edit effectively preserves the structural information of the source image or video while achieving high-quality editing results. Our approach is compatible with any pre-trained rectified-flow-based models for image and video tasks, requiring no additional training or optimization. Extensive experiments across generation, inversion, and editing tasks in both image and video modalities demonstrate the superiority and versatility of our method. The source code is available at https://github.com/wangjiangshan0725/RF-Solver-Edit.
1. Introduction
Rectified-flow models perform strongly in image and video generation but suffer from inversion errors that degrade reconstruction and downstream editing. The paper introduces RF-Solver to improve ODE solving and RF-Edit to preserve source structure during editing.
- Rectified-flow models show strong image and video generation but have been less explored for inversion-reconstruction and editing.
- Vanilla rectified-flow inversion accumulates timestep errors, causing reconstructed images and videos to diverge from their sources.Image reconstructions can shift object positions and individual appearances, while video inversion produces noticeable distortion.
- RF-Solver derives the exact rectified-flow ODE solution and uses higher-order Taylor expansion to estimate its nonlinear neural-network component.The sampler targets reduced ODE-solving error without additional training or optimization and integrates with rectified-flow models.
- RF-Solver improves inversion and reconstruction accuracy while also improving fundamental tasks such as text-to-image generation.The green curve in Figure 2 represents RF-Solver inversion, compared with vanilla rectified flow in red.
- RF-Edit extends RF-Solver with self-attention value-feature sharing from inversion to denoising for image and video editing.The design uses DiT structures from FLUX and OpenSora to preserve source information during edits.
2. Related Work
Prior work established inversion and training-free editing methods mainly for diffusion models, while inversion and editing in rectified-flow diffusion transformers remained comparatively under-explored.
- 2.1. Inversion: Inversion maps real images and videos into noise-space representations as the reverse of generation.
- 2.1. Inversion: DDIM inversion recursively adds predicted noise, motivating methods that mitigate its discretization error.
- 2.1. Inversion: Inversion research for rectified-flow models such as FLUX and OpenSora remains limited, with existing approaches requiring optimization or adding a source-conditioned vector field.The paper instead targets error from the original rectified-flow vector field directly.
- 2.2. Editing: Training-free editing methods use strategies including prompt refinement, attention sharing, mask guidance, noise initialization, motion guidance, latent manipulation, and canonical representations.
- 2.2. Editing: Editing performance for rectified-flow-based diffusion transformers has remained largely under-explored.
3. Method
The method combines RF-Solver, a higher-order ODE sampler for more accurate inversion and reconstruction, with RF-Edit, which shares inversion self-attention features to preserve source structure during image and video editing.
- RF-Solver: RF-Solver derives an exact rectified-flow ODE formulation and uses Taylor expansion to approximate its nonlinear neural-network component.Higher-order expansion separates network prediction derivatives from the integral and reduces per-timestep ODE-solving error.
- RF-Solver: The n-th-order RF-Solver solution reduces to vanilla rectified flow when n = 1, while experiments use n = 2 to mitigate errors.The second-order setting is reported as effectively reducing accumulated inversion and reconstruction errors.
- RF-Solver: RF-Solver estimates the first-order derivative of the network prediction by taking an auxiliary step of size ∆t and comparing two network evaluations.The experimental auxiliary timestep is ∆t = 0.01; the derivative cannot be obtained analytically because of the neural network’s complex architecture.
- RF-Solver: The derived inversion process reverses the ODE sampling process in the limit of small steps, mapping image or video data back into noise-space latents.Latents are tracked during inversion and denoising so corresponding intermediate states can be compared.
- RF-Edit: RF-Edit stores self-attention Value features from selected late inversion timesteps and transformer blocks, then substitutes them during editing.The framework is applied to FLUX image editing and OpenSora video editing, with architecture-specific feature sharing.
- RF-Edit: RF-Edit shares features in FLUX single blocks and adapts self-attention features for editing to preserve source-image or source-video structure.The framework is intended to enable high-quality editing while retaining structural information from the source.
4. Experiment
Experiments evaluate RF-Solver and RF-Edit across image and video generation, reconstruction, and editing. The results show improved reconstruction accuracy, editing quality, structural or temporal consistency, and a trade-off in feature-sharing and Taylor-expansion choices.
- Text-to-image generation: RF-Solver demonstrates superior performance in text-to-image generation compared with vanilla RF, Heun, and DPM-Solver baselines.The reported qualitative and quantitative results indicate higher-quality images that align more closely with human cognition.
- Inversion and reconstruction: RF-Solver significantly improves reconstruction accuracy for both images and videos over vanilla RF and Heun sampling.Table 2 reports quantitative improvements, while qualitative results show less image drift and video distortion.
- Image editing: RF-Edit achieves the highest reported image-editing CLIP score while balancing prompt fidelity with preservation of the source image.The method supports adding, replacing, and stylization, whereas baselines may change backgrounds or fail at desired edits.
- Video editing: RF-Edit achieves higher VBench metrics for video editing and maintains temporal consistency while delivering superior quality.It also handles complicated multi-subject edits that other baseline methods fail to perform.
- Taylor expansion ablation: Second-order Taylor expansion improves performance over first-order expansion, while higher orders provide no further enhancement under the same NFE.Higher-order variants require more inference steps and derivatives, increasing complexity; RF-Solver-2 is therefore used downstream.
- Feature-sharing ablation: Five feature-sharing steps effectively improve editing, but the optimal setting varies by image difficulty and excessive sharing can undermine the edit.Too many sharing steps can make outputs overly similar to the source image or cause editing failure.
5. Conclusion
The paper proposes RF-Solver to reduce rectified-flow ODE-solving error and RF-Edit to preserve source structure during image and video editing. Experiments demonstrate their versatility and effectiveness across generation, reconstruction, and editing tasks.
- 5. Conclusion: RF-Solver is a versatile rectified-flow sampler that solves the RF ODE with reduced error across image and video tasks.The paper applies it to sampling and reconstruction while enhancing generation quality.
- 5. Conclusion: RF-Edit extends RF-Solver with feature sharing to achieve high-quality editing while preserving structural information in source images and videos.The conclusion presents RF-Edit as the editing framework built upon RF-Solver.
- 5. Conclusion: Extensive experiments demonstrate the versatility and effectiveness of the proposed method.The reported scope includes generation, reconstruction, and editing tasks in image and video modalities.
B.1. Baselines and Implementation Details
The experiments use pretrained FLUX and OpenSora models with task-specific datasets, prompts, baselines, and feature-sharing configurations. Image and video editing evaluations cover multiple editing requirements and established comparison methods.
- Text-to-image generation: Text-to-image experiments use 10,000 MS-COCO validation images at 1024×1024 resolution and set total NFE to 10 for RF-Solver and baselines.Compared methods include FLUX with vanilla sampling, Heun Solver, and DPM-Solver applied to Stable Diffusion.
- Inversion: Image inversion uses MS-COCO images and captions, while video inversion uses about 40 videos from TikTok and other public sources.GPT-4o generates detailed prompts because prompt quality substantially influences inversion quality.
- Editing: Editing experiments share features from the last 19 FLUX single blocks for images and the last 14 OpenSora blocks for videos.The feature-sharing-step hyperparameter is adjusted separately for image and video editing.
- Editing: Image-editing evaluation uses over 300 real and generated images, with 2–3 target prompts per source covering adding, replacing, and stylization.Source prompts are generated with GPT-4o and manually refined before comparison with RF-inversion and diffusion-based baselines.
- Baselines: Image baselines include P2P, DiffEdit, SDEdit, PnP, Pix2pix, and RF-Inversion, while video baselines include FateZero, FLATTEN, COVE, RAVE, and TokenFlow.These methods provide comparisons across image and video editing settings.
B.2. Evaluation Metrics
The evaluation uses task-specific metrics for text-to-image sampling and inversion, alongside VBench criteria for video editing.
- Text-to-image sampling: Text-to-image sampling is evaluated with FID and CLIP Score.FID assesses generated-image distribution quality, while CLIP Score measures alignment with textual descriptions.
- Inversion: Inversion is evaluated using MSE, LPIPS, SSIM, and PSNR.These metrics quantify pixel error, perceptual similarity, structural similarity, and signal-to-noise quality.
- Video editing: Video editing is evaluated with VBench metrics for subject consistency, motion smoothness, aesthetic quality, and imaging quality.The criteria assess subject preservation, temporal smoothness, visual appeal, and frame quality.
- Qualitative results: Stylized generation is presented among the qualitative evaluation results.The paper labels these results as stylized generation.
C. Qualitative Results for Text-to-image Sampling
RF-Solver produces text-to-image samples that are qualitatively higher quality and better aligned with human perception than vanilla Rectified Flow.
- Text-to-image sampling: RF-Solver generates higher-quality text-to-image images than vanilla Rectified Flow.The qualitative comparison is shown in Figure 10.
D. More Qualitative Results for Image Editing
Additional qualitative results demonstrate RF-Edit's application to image editing and stylization.
- Image editing: The paper provides more qualitative results for image editing and stylization.These examples are presented in Figure 8.
E. More Qualitative Results for Video Editing
Additional results cover video editing with RF-Solver and RF-Edit, including an implementation on HunyuanVideo and broader potential applications across rectified-flow tasks.
- Video editing: RF-Solver and RF-Edit are implemented on HunyuanVideo, with RF-Edit sharing features in its single-stream block.The results are shown in Figure 9.
- Potential applications: RF-Solver is described as a universal rectified-flow sampler applicable to generation, editing, and other diffusion-based tasks.The feature-sharing method in RF-Edit is also described as applicable to other image and video editing methods.
- Video editing: Additional video editing results are presented for the proposed approach.The paper identifies these examples as Figure 9.
- Text-to-image generation: Text-to-image generation results are also included among the paper's qualitative examples.Figure 10 presents these results.