PaperlayerView exact revision on arXiv

Source-linked AI summary

DragFlow: Unleashing DiT Priors with Region Based Supervision for Drag Editing

Zihan Zhou, Shilin Lu, Shuli Leng, Shaocong Zhang, Zhuming Lian, Xinlei Yu, Adams Wai-Kin Kong

arXiv:2510.02253v3cs.CVcs.AIcs.LG

TL;DR

Drag-based editing remains prone to distortions because earlier base-model priors and point supervision do not adequately support natural, localized edits. DragFlow uses region-based affine supervision with adapter-enhanced inversion and hard background constraints to harness FLUX, and it surpasses existing baselines on DragBench-DR and ReD Bench.

  • Problem

    Drag-based editing needs stronger generative priors and supervision suited to DiT features because earlier methods often distort target regions and point-based guidance transfers poorly to DiTs.

  • Method

    DragFlow is a DiT-oriented region-based framework combining affine region supervision, gradient mask-based hard background constraints, adapter-enhanced inversion, and MLLM-assisted task interpretation.

  • Results

    DragFlow consistently outperforms state-of-the-art baselines on DragBench-DR and ReD Bench, while achieving the lowest MD across both benchmarks.

  • Takeaways & Limitations

    Region-level supervision and the framework’s inversion and background-handling design make stronger DiT priors usable for controllable drag editing.

  • Takeaways & Limitations

    CFG-distilled FLUX has larger inversion drift, and highly intricate structures can still show reconstruction detail loss and degraded visual quality.

Abstract

from arXiv · show

Drag-based image editing has long suffered from distortions in the target region, largely because the priors of earlier base models, Stable Diffusion, are insufficient to project optimized latents back onto the natural image manifold. With the shift from UNet-based DDPMs to more scalable DiT with flow matching (e.g., SD3.5, FLUX), generative priors have become significantly stronger, enabling advances across diverse editing tasks. However, drag-based editing has yet to benefit from these stronger priors. This work proposes the first framework to effectively harness FLUX's rich prior for drag-based editing, dubbed DragFlow, achieving substantial gains over baselines. We first show that directly applying point-based drag editing to DiTs performs poorly: unlike the highly compressed features of UNets, DiT features are insufficiently structured to provide reliable guidance for point-wise motion supervision. To overcome this limitation, DragFlow introduces a region-based editing paradigm, where affine transformations enable richer and more consistent feature supervision. Additionally, we integrate pretrained open-domain personalization adapters (e.g., IP-Adapter) to enhance subject consistency, while preserving background fidelity through gradient mask-based hard constraints. Multimodal large language models (MLLMs) are further employed to resolve task ambiguities. For evaluation, we curate a novel Region-based Dragging benchmark (ReD Bench) featuring region-level dragging instructions. Extensive experiments on DragBench-DR and ReD Bench show that DragFlow surpasses both point-based and region-based baselines, setting a new state-of-the-art in drag-based image editing. Code and dataset are available at https://github.com/Edennnnnnnnnn/DragFlow.

1 INTRODUCTION

Drag-based editing offers precise spatial control but still suffers from distortions caused by weak generative priors and limitations in adapting DiT features. DragFlow addresses these issues with region-based supervision, adapter-enhanced inversion, and hard background constraints, and is evaluated on two benchmarks.

  • Drag-based editing provides finer spatial control than text-driven editing but often produces unnatural deformations in intricate or structurally complex images.Text instructions can underspecify geometry and locality, whereas drag instructions specify localized motions more precisely.
  • Stronger DiT-based priors offer an opportunity for drag editing, but directly applying earlier point-based methods to DiTs yields suboptimal results because their features do not match point supervision.The supplied introduction identifies this mismatch as a core obstacle when transferring prior drag-editing methods to DiTs.
  • DragFlow combines region-level affine motion supervision, hard constraints for editable-region background handling, and adapter-enhanced inversion to improve controllability, subject fidelity, and realism.The framework is designed to exploit DiT priors while reducing deformation artifacts and preserving identity and background fidelity.
  • ReD Bench adds point-to-region alignment, explicit relocation, deformation, and rotation tags, and contextual descriptions, while evaluation covers ReD Bench and DragBench-DR.The benchmark is intended to clarify region-level dragging instructions and user intent.

2 RELATED WORK

Prior drag-editing methods are organized into optimization-based, finetuning-based, and other broad categories, with optimization methods predominating. DragFlow remains optimization-based but replaces point supervision with region-level supervision and requires less manual region specification than RegionDrag.

  • Existing drag-editing methods are broadly grouped into three categories, including optimization-based, finetuning-based, and other approaches.The supplied related-work passage introduces this taxonomy but does not enumerate the third category in full.
  • Optimization-based methods are the most prevalent category and iteratively refine inverted noisy latents during inference, often using point-wise motion supervision and tracking.These methods can produce unnatural deformations when optimized latents move away from the natural image manifold.
  • DragFlow stays within the optimization-based paradigm but replaces point-based motion supervision with region-level supervision, requiring only a target-point center instead of a manually predefined target-region mask.This distinguishes it from RegionDrag in non-rigid scenarios where manual mask definition is difficult.

3 METHODOLOGY

DragFlow adapts drag editing to DiT feature geometry through region-level supervision, while combining constrained background preservation, adapter-enhanced subject consistency, and MLLM-assisted intent interpretation.

  • 3.1 WHY POINT-BASED DRAG FAILS ON DIT: DiT’s finer-grained features weaken point-based motion supervision because each feature location captures less semantic context than a UNet feature point.Directly adapting point-based drag frameworks to FLUX yields only limited improvement over their Stable Diffusion counterparts.
  • 3.2 REGION-LEVEL AFFINE SUPERVISION: DragFlow optimizes inverted noisy latents with DiT feature matching between user-provided source masks and affine-transformed target regions.The framework encodes the image, inverts it to a noisy latent, and iteratively updates that latent using the drag objective.
  • 3.2 REGION-LEVEL AFFINE SUPERVISION: Region-level affine supervision replaces point matching with source-to-target region feature alignment, providing richer semantic context and avoiding brittle handle-point tracking.The target mask is progressively transformed from the source configuration toward the target using translation, deformation, or rotation-specific affine parameters.
  • 3.3 BACKGROUND PRESERVATION: Background preservation hard-constrains only the editable region’s update, avoiding competition with a consistency loss that can be unreliable under inversion drift.A separate reconstruction path supplies the original latent for the preserved background and yields better background preservation in challenging FLUX settings.
  • 3.4 SUBJECT CONSISTENCY ENHANCEMENT: Adapter-enhanced inversion uses pretrained personalization adapters to reinforce subject representations during DiT-based editing.The method uses adapters as auxiliary subject representations integrated into the text-to-image generation process.

4 EXPERIMENTS

DragFlow is evaluated on DragBench-DR and ReD Bench using quantitative, qualitative, and ablation studies. It combines region-aware supervision, hard background constraints, and adapter-enhanced inversion to improve alignment, fidelity, and subject consistency.

  • Experimental Setup: ReD Bench provides point-to-region alignment, explicit relocation, deformation, and rotation tags, and contextual intent descriptions for evaluation.These annotations address limitations of point-based and coarse-region datasets.
  • Quantitative Analysis: DragFlow achieves the lowest MD across both benchmarks and outperforms most baselines on background fidelity while ranking second on IFbg.The authors attribute the remaining IFbg gap largely to inversion limitations in the CFG-distilled model.
  • Qualitative Analysis: DragFlow consistently follows specified dragging operations while preserving global scene coherence across challenging scenarios.RegionDrag and InstantDrag introduce structural distortions, FreeDrag and FastDrag struggle with rotations, and CLIPDrag and DragLoRA often misinterpret relocation as deformation.
  • Ablation Study: Region-level manipulation reduces MD1 by 19.95 and increases IFs2t by 0.027 compared with the baseline.The ablation attributes these gains to richer semantic guidance than sparse point-level supervision.

5 CONCLUSION

The paper concludes that DragFlow adapts DiT priors to controllable drag editing through region-level supervision, background hard constraints, and adapter-enhanced inversion. It surpasses existing methods on ReD Bench and DragBench-DR, while intricate structures remain vulnerable to inversion-related detail loss.

  • Conclusion: DragFlow combines region-level motion supervision, background hard constraints, and adapter-enhanced inversion to improve faithfulness, controllability, and output quality.The framework is designed specifically for DiT-based drag editing.
  • Conclusion: DragFlow consistently surpasses existing state-of-the-art methods on ReD Bench and DragBench-DR.The reported gains are described in terms of stronger faithfulness, better controllability, and higher-quality outputs.
  • Limitations and Future Work: Images with highly intricate structures can still exhibit detail loss and degraded visual quality because FLUX inversion drift remains larger in its CFG-distilled variant.Adapter-enhanced inversion mitigates but does not eliminate these artifacts.

B.1 DIFFUSION METHODS

DragFlow builds on rectified-flow DiT generation and replaces sparse point supervision with region-level affine guidance. Its workflow combines latent inversion, motion supervision, point tracking, and region-preserving constraints for localized editing.

  • Stable Diffusion with DDIM Inversion: Classic drag editing starts from a VAE latent, iteratively applies motion supervision and point tracking, and updates the latent until handles reach their targets.Inputs include an image, source and target handle points, and a mask protecting regions that should remain unchanged.
  • Rectified Flow with ODE Solver: DiT-based rectified flow uses straight latent trajectories and backward ODE integration, enabling deterministic sampling with fewer steps than DDIM.The forward process interpolates between clean latent and noise, while the backward process solves the velocity-field ODE from noise toward data.
  • Point-Based Drag Editing: Motion supervision combines feature alignment, smoothness, and mask losses to preserve source features while moving selected points toward targets.Alignment compares local feature patches around original and predicted handle locations; mask loss penalizes changes outside user-defined regions.
  • Point-Based Drag Editing: Point tracking introduces uncertainty and narrow guidance because nearest-neighbor matching and individual control points affect only limited feature neighborhoods.Region-based dragging addresses this limitation by operating over semantically coherent masks for more stable and meaningful guidance.
  • Region-Based Dragging: DragFlow applies progressive affine transformations to region masks, providing richer feature supervision than point-wise objectives during localized editing.The motion schedule is computed initially and then linearly interpolated across subsequent iterations, supporting smooth region transitions.

C.2 RELOCATION TASKS

DragFlow represents relocation, deformation, and rotation as progressive affine transformations over region masks. Relocation preserves region geometry, deformation modifies selected edges, and rotation reorients a region around an explicit anchor.

  • Relocation: Relocation shifts an entire source region toward its target while preserving the region’s geometry and scale.An affine translation adds displacement components to every point in the operation patch.
  • Deformation: Deformation selectively displaces object edges, enabling localized elongation or shortening without moving the whole region rigidly.It uses the same affine transformation framework as relocation but applies it only to edge areas selected for shape adjustment.
  • Rotation: Rotation reorients a region around a user-specified anchor using an angle derived from the source centroid, anchor, and target centroid.The transformation translates the anchor to the origin, applies rotation, and translates the result back to the original coordinate frame.

C.5 DETAILS ABOUT EXPERIMENTAL SETTINGS

DragFlow selects FLUX’s D17 and D18 double-stream blocks for feature manipulation and refines region motion through 70 optimization steps. Layer ablations and mask visualizations support these design choices.

  • Layers for Feature Manipulation: D17 and D18 provide the preferred FLUX features because they retain stable semantic, spatial, and identity-related information for editing.Other layers are either overly clean for meaningful edits or contain too little semantic information to preserve identity and control dragging precisely.
  • Layer-based Performance Comparison: Across all 57 evaluated FLUX layers, double-stream layers outperform single-stream layers, with D17 and D18 achieving the strongest MD1, IFs2s, and IFs2t performance.The metrics assess displacement precision, preservation of post-dragging local features, and suppression of original-location features.
  • Affine Transformation Steps: The main drag motion runs for 50 affine-transformation steps, followed by 20 additional steps that repeat the final transformation to refine post-dragging features.The progressive operation masks evolve smoothly across the full 70-step process.

C.6 QUANTITATIVE ANALYSIS FOR DIT SUPERVISION GRANULARITY

DragFlow evaluates region-level supervision against point-based DiT supervision and adds adaptive mechanisms for multi-region weighting, background protection, and user-aligned input processing. The supplied passages state that region supervision bridges DiT’s fine-grained feature limitations, but provide no quantitative table values.

  • Quantitative Supervision Comparison: Region-based supervision is compared with point-based supervision on DragBench-DR using the same DiT backbone and auxiliary modules.This isolates the supervision mechanism rather than conflating it with differences in background preservation or inversion components.
  • Quantitative Supervision Comparison: The comparison concludes that sparse point tracking is insufficient for DiT’s high-frequency features, whereas region-level affine supervision better exploits the model’s generative potential.The passage describes this as empirical corroboration, but does not provide the table’s numerical values.
  • Adaptive Input Processing: An MLLM-assisted interface converts rough scribbles and target clicks into operational masks, task labels, prompts, and tags with a confirmation step.The workflow supports relocation, deformation, and rotation while reducing user effort and resolving ambiguous intent.
  • Adaptive Multi-Region Weighting: Adaptive weights prevent large manipulated regions from dominating optimization by assigning higher relative influence to smaller regions.Weights are computed from source-mask sizes and normalized across multiple operations.
  • Background Protection: Gradient masking restricts optimization to transformed editable areas, directly blocking updates in uneditable regions rather than merely penalizing them after changes occur.An adaptive bounding mask provides a static envelope covering the full dragging process.

D.4 EFFECTIVENESS OF MLLM-DRIVEN INTENT PARSING

MLLM-generated intent prompts improve DragFlow’s performance, while the framework remains effective without text guidance and degrades under incorrect prompts.

  • The evaluation compares Null Prompt, Incorrect Prompt, and Matched Prompt settings, including prompts generated by GPT-5 and QWen-VL.
  • DragFlow remains robust without textual guidance, demonstrating that region-level affine supervision provides the core editing signal.
  • Matched prompts improve performance, whereas intentionally incorrect prompts degrade results through conflicting guidance.The workflow mitigates misinterpretation by requiring users to select the optimal intent from generated candidates.
  • The framework evaluates visual fidelity, background integrity, and spatial accuracy using Image Fidelity and Mean Distance criteria.
  • ReD Bench provides point- and region-level drag annotations, intention labels, and complementary region masks for evaluating regional editing.

G.2 ADOPTION OF THE DragBench-DR BENCHMARK

DragBench-DR extends drag evaluation from sparse points to regions, but its original point-based protocol can mismatch region-level edits, motivating centroid-based evaluation.

  • The benchmark includes image samples, source images, operation masks, drag points, centroids, background prompts, and editing-intention prompts.
  • DragBench-DR adapts the classic DragBench benchmark to region-based operations, better representing user intentions in interactive editing.
  • Its original MD2 protocol still compares pre- and post-drag regions using sparse annotated points, which can misrepresent region-level edits.
  • The updated evaluation replaces point annotations with an automatic centroid-based feature comparison for region methods.

H.1 ADDITIONAL QUALITATIVE SAMPLES

Additional experiments examine DragFlow under complex instructions, intricate structures, input errors, and resource constraints, while testing transfer across DiT backbones.

  • Composite movements and non-affine operations are approximated by composing elementary operations in parallel or sequentially.
  • DragFlow maintains editing effectiveness and structural fidelity on complex textures such as hair and cloth, supporting its affine assumption empirically.
  • Incomplete or undersized masks compromise editing quality, whereas slightly oversized masks are tolerated when the target semantic object remains covered.
  • DragFlow requires more resources than baselines because it operates on the larger, more complex FLUX.1 DiT architecture.
  • On SD3.5, DragFlow achieves MD1 35.21 and IFs2t 0.945, compared with 31.59 and 0.948 for FLUX, respectively.

J.2 GENERALIZABILITY OF ID-PRESERVATION DESIGN

DragFlow’s identity-preservation design supports both generic IP-Adapters and subject-specific LoRA, with a practical trade-off between performance and per-subject training.

  • The comparison evaluates DragFlow with IP-Adapter versus LoRA on averaged DragBench-DR performance.
  • The LoRA-based variant performs slightly better than the generic IP-Adapter because it is optimized for the specific subject.
  • IP-Adapters are more practical and broadly applicable because they avoid training a new LoRA for each subject.
Loading 2510.02253v3…