PaperlayerView exact revision on arXiv

Source-linked AI summary

Drag as Evidence: Motion-Grounded Latent Recomposition for Drag-Based Editing

Xinyu Pu, Hongsong Wang, Jie Gui, Pan Zhou

arXiv:2609.36755v2cs.CV

TL;DR

Text-based image editors remain limited in precise spatial control, while drag-based methods struggle to balance accuracy and natural synthesis. MoRe-Drag injects warped-image motion evidence through region-aware recomposition and stage-adaptive conditioning, with drag-aware instruction inference. It improves drag accuracy over base editors and achieves the best drag accuracy on both benchmarks among the compared methods, while maintaining strong visual and semantic consistency.

  • Problem

    Text-based editors have limited precise spatial control, while existing drag-based methods struggle to balance drag accuracy with natural, plausible, intent-aligned generation.

    Source: §1 · p. 2

  • Method

    MoRe-Drag treats pixel-space warping as coarse motion evidence and injects it into sampling through region-aware latent recomposition and stage-adaptive conditioning.

    Source: §1 · p. 2, §3.1 · p. 5

  • Results

    MoRe-Drag achieves the best drag accuracy on both benchmarks with strong CP and PF scores; on DRAGBENCH-DR, its Mean Distance (MD) improves from 46.73 to 20.03 over LongCat-Image-Edit alone.

    Source: §1 · p. 3, §4.2 · p. 9

  • Takeaways & Limitations

    Pixel-space warping can serve as coarse motion evidence for advanced editors, with region roles and sampling stage governing its use alongside natural appearance generation and completion.

    Source: §3.4 · p. 7, §5 · p. 10

    The method is primarily developed for editing backbones with an MLLM text encoder and an MMDiT-style generative decoder.

    Source: §5 · p. 10

Abstract

from arXiv · show

Modern image editors excel at semantic manipulation and visual synthesis, yet remain limited in precise spatial control, motivating the development of drag-based editing. However, existing drag-based methods often struggle to balance drag accuracy with natural, plausible, and intent-aligned generation. We propose MoRe-Drag, a motion-grounded drag-based editing method. Our key insight is to treat pixel-space warping as coarse motion evidence, and to inject this evidence into the generative sampling trajectory. Specifically, MoRe-Drag performs region-aware latent recomposition over refinement, inpainting, and preservation regions, coupled with stage-adaptive conditioning that progressively shifts from motion-grounded structure formation to semantic refinement. We further support a no-manual-instruction interface by adapting the MLLM-based text encoder for drag-aware instruction inference. Experiments on DragBench-SR and DragBench-DR show that MoRe-Drag substantially improves drag precision over strong base editors and achieves superior drag accuracy among SOTA drag-based methods, while delivering strong semantic consistency and visually realistic results. Project page: https://xinyu-pu.github.io/projects/more-drag/.

1 Introduction

Text-based editing offers semantic control but limited spatial precision, while drag-based methods face a trade-off between accurate motion and natural synthesis. MoRe-Drag treats pixel-space warping as motion evidence and combines it with region-aware recomposition, stage-adaptive conditioning, and drag-aware instruction inference.

1 Introduction

  • Text instructions specify what to change but often lack precise spatial control, while strong motion constraints and generative regularization pull drag editing toward accuracy and naturalness, respectively.This tension motivates coupling modern editors’ generative priors with explicit motion evidence.Source: §1 · p. 2
  • MoRe-Drag treats the warped image as coarse motion evidence rather than an intermediate edit, complementing the generative prior with explicit drag control.Its components are region-aware recomposition, generation-stage adaptive conditioning, and instruction interaction.Source: §1 · p. 2
  • MoRe-Drag partitions the warped observation into motion-evidence, disoccluded, and preservation regions, maintaining separate latent trajectories that are recomposed according to those roles.The regions respectively provide coarse motion cues, semantic completion, and unchanged-content preservation.Source: §1 · p. 2
  • MoRe-Drag emphasizes the warped observation early, then shifts toward the source image and instruction for semantic refinement, while a lightweight adapter infers drag-aware instructions.The adapter is enabled for instruction inference and disabled during generation; unlike Inpaint4Drag [29], the warp is used as motion evidence, not an intermediate edit.Source: §1 · p. 3
  • On DRAGBENCH-DR, MoRe-Drag reduces LongCat-Image-Edit’s Mean Distance (MD) from 46.73 to 20.03, and on DRAGBENCH-SR from 53.62 to 18.54.The paper reports natural and accurate edits alongside these improvements.Source: §1 · p. 3

2 Related Work

Text-guided editors use pretrained generative models and multimodal conditioning, but text remains limited for precise spatial motion control. Drag-based methods add explicit spatial control through approaches that include point tracking, latent or pixel-space deformation, and model-based motion injection.

Text-guided image editing

  • MLLM-powered editors jointly interpret the reference image and instruction, improving instruction following and appearance preservation, but text alone remains limited for precise spatial motion control.The cited examples include FLUX.2, Qwen-Image-Edit, LongCat-Image-Edit [16], Step1X-Edit [17], and FireRed-Image-Edit [35].Source: §2 · p. 3

Drag-based image editing

  • DragGAN [18] introduced point tracking and motion supervision, and DragDiffusion [19] extended drag-based editing to diffusion models; later methods pursue robustness, accuracy, and efficiency.Subsequent approaches include pixel-space warp-then-inpaint in Inpaint4Drag [29] and methods for MMDiT-based models [39] [40].Source: §2 · p. 3
  • Unlike Inpaint4Drag [29], MoRe-Drag uses the warped image only as motion evidence, and unlike ContextDrag, it injects motion externally without changing the MMDiT editor.ContextDrag relies on attention-level reference injection.Source: §2 · p. 3

3 Methodology

MoRe-Drag treats a warped image as coarse motion evidence and injects it into generative sampling to support natural, precise drag editing. It combines region-aware latent recomposition, stage-adaptive conditioning, and automatic drag-aware instruction inference.

3 Methodology

  • MoRe-Drag combines coarse motion evidence from a warped image with generative priors to produce natural, precise drag edits.The warp guides the requested motion rather than serving as the intermediate edit or final result.Source: §3 · p. 3, §3.1 · p. 4

3.1 Region-Aware Recomposition

  • Region-aware recomposition assigns refinement, inpainting, and preservation roles to separate masks and recomposes latent trajectories accordingly.The refinement region blends evidence and generation branches; inpainting uses the conditioned generation branch, while preservation uses the evidence branch.Source: §3.1 · pp. 4–5

3.2 Generation-Stage Adaptive Conditioning

  • Adaptive conditioning uses the warped image early for motion-grounded structure, then the original image and editing instruction for semantic refinement.The switch timestep is set independently of the recomposition interval.Source: Figure 3 · p. 4, §3.2 · p. 6

3.3 Instruction Interaction

  • A no-manual-instruction interface reuses the editor’s MLLM text encoder to infer drag-aware instructions from drag annotations.The encoder is fine-tuned with LoRA for inference, and its adapter is disabled during generation.Source: §3.3 · p. 7

3.4 Discussion

  • The evidence branch carries motion, recomposition determines where it is reliable, conditioning determines when it dominates, and generation restores appearance and missing content.Instruction interaction provides high-level guidance on what content should be restored.Source: §3.4 · p. 7

4 Experiments

MoRe-Drag is evaluated on two drag-editing benchmarks against base editors and prior methods, with quantitative, qualitative, human-preference, and ablation analyses. The results favor MoRe-Drag in drag accuracy, while ablations examine its recomposition, conditioning, and instruction-interaction components.

4.1 Implementation Details and Experimental Setup

  • The evaluation uses LongCat-Image-Edit and Qwen-Image-Edit, with MoRe-Drag and Inpaint4Drag taking warped images while other compared drag methods take points and masks.The benchmarks are DRAGBENCH-SR and DRAGBENCH-DR; reference instructions were generated and human-corrected because the benchmarks provide none.Source: §4.1 · pp. 7–8

4.2 Comparison with SOTAs

  • MoRe-Drag with LongCat-Image-Edit achieves the best drag accuracy on both benchmarks while maintaining strong CP and PF scores; Table 1 reports DRAGBENCH-SR/DR results.The comparison also finds that Inpaint4Drag with the stronger FLUX.1-Fill-dev backbone falls short in precise and natural editing.Source: Table 1 · p. 8, §4.2 · p. 9
  • MoRe-Drag produces more accurate, natural edits with better texture, shape, local-structure, boundary, and fine-detail preservation than representative methods.The reported examples include coherent lion-mouth details and coordinated sunset-sky and reflection edits.Source: Figure 6 · p. 8, §4.2 · p. 9
  • MoRe-Drag receives the best average rank for naturalness, drag faithfulness, and preservation in a 42-participant human preference study.Participants ranked anonymized results from FastDrag, Inpaint4Drag, and MoRe-Drag; rank 1 was best.Source: §4.2 · pp. 9–10, Table 4 · p. 10

4.3 Ablation Studies

  • Removing RAR weakens local deformation control, while removing GAC yields less plausible structures; single-stage conditioning preserves motion cues or semantic editing but can leave distortions or competition.Different recomposition-weight schedules perform similarly, so the constant schedule is used by default.Source: §4.3 · p. 10
  • LoRA adaptation of instruction inference better captures drag intent than misaligned instructions, improving editing faithfulness and reducing artifacts or unsupported content.Figure 9 illustrates the drag-aware instruction comparisons.Source: Figure 9 · p. 9, §4.3 · p. 10

5 Conclusion

MoRe-Drag grounds modern editors with motion evidence, combining region-aware recomposition and stage-adaptive conditioning to improve spatial control while preserving natural, coherent generation. The paper identifies backbone scope and dependence on null-conditioned reconstruction as limitations, and notes potential misuse risks from more controllable editing.

5 Conclusion

  • MoRe-Drag combines motion evidence, region-aware recomposition, and stage-adaptive conditioning to support spatially accurate, natural, and visually coherent editing.The conclusion also describes an instruction-free mode for drag-aware intent inference.Source: §5 · p. 10
  • The method is primarily developed for editors with MLLM text encoders and MMDiT-style decoders, and its consistency depends on null-conditioned reconstruction ability.The authors propose improved reconstruction priors and editor-agnostic calibration as future work.Source: §5 · p. 10

A Broader Impacts

  • More controllable editing may reduce the effort needed to create convincing deceptive media, motivating consent, provenance, watermarking, and authentication safeguards.The stated risks include altering visual evidence and non-consensual edits of personal images.Source: A Broader Impacts · p. 14

B Drag-Instruction Dataset Construction

The dataset pipeline filters ByteMorph for local human or object motion, mines drag correspondences, and scores annotations for quality. It retains approximately 220K motion-related samples before score-based filtering yields approximately 68K drag-instruction samples.

B Drag-Instruction Dataset Construction

  • The pipeline filters ByteMorph for local motion edits, retaining human or object motion while discarding global camera changes, and keeps approximately 220K samples.Qwen3.5-9B parses instructions and identifies edited entities for later mask extraction.Source: B Drag-Instruction Dataset Construction · p. 14
  • For retained samples, entity masks and bidirectional optical flow identify valid local motion candidates for drag annotations.Grounding DINO detects entity boxes, SAM converts them to masks, and RAFT computes flow; both forward and corresponding backward flow magnitudes must exceed 1.5 pixels.Source: B Drag-Instruction Dataset Construction · p. 15
  • The candidate motion mask intersects entity and valid-motion masks; regions are filtered, clustered by flow direction, and sampled for handle points whose targets follow forward flow.The pipeline removes a 4-pixel border and components smaller than 64 pixels, retains at most 8 motion regions, and caps handle points at 8 per region and 64 per sample.Source: B Drag-Instruction Dataset Construction · p. 15
  • A VLM scores camera motion, instruction alignment, and target alignment, and filtering below Savg 0.6 plus human spot-checking yields approximately 68K high-quality samples.Each dimension is scored from 0 to 3, with higher values indicating better quality.Source: B Drag-Instruction Dataset Construction · p. 15

C.1 Implementation Details

MoRe-Drag is implemented on LongCat-Image-Edit and Qwen-Image-Edit with 30 sampling steps by default. Recomposition and adaptive-conditioning switch points, guidance scales, and recomposition weights vary by benchmark.

C.1 Implementation Details

  • Both editors use Qwen2.5-VL-7B text encoding and MMDiT generators, with 6B and 20B parameters for LongCat-Image-Edit and Qwen-Image-Edit, respectively.Experiments use NVIDIA RTX 4090 GPUs and 30 sampling steps unless specified otherwise.Source: §C.1 · p. 15
  • For DRAGBENCH-SR, recomposition runs from steps 1–16 and adaptive conditioning switches at step 12; DRAGBENCH-DR uses steps 1–14 and switches at step 11.CFG scales are 3 for DRAGBENCH-SR and 2 for DRAGBENCH-DR; recomposition weights are λmax = 0.5 and λmin = 0.3.Source: §C.1 · p. 15

C.2 Training Details

The appendix details MoRe-Drag’s training and sampling procedure, including how it constructs motion evidence, adapts instructions, and evaluates edits. Sampling recomposes separate latent trajectories using scheduled weights and timestep-dependent conditioning.

C.2 Training Details

  • The drag-aware instruction encoder is LoRA-fine-tuned on 68K filtered drag instructions for 3 epochs, with the vision tower, LLM backbone, and multimodal merger frozen.Training uses two NVIDIA RTX PRO 6000 GPUs for 15 GPU-hours.Source: §C.2 · p. 16

C.3 Formulation of Weight Schedule

  • The constant recomposition-weight schedule is the default; the appendix also gives linear and inverse-square schedules.The supplied formulas express the linear schedule with ρ_i and the inverse-square schedule with ρ^2.Source: §C.3 · p. 16

C.4 Details of Bidirectional Warping

  • Bidirectional warping turns handle–target displacements into a coarse warped observation, then separates preservation, refinement, and inpainting regions for latent recomposition.The warped image serves as motion evidence rather than an intermediate edited image; forward warping may create holes or overlaps, motivating a backward mapping.Source: §C.4 · pp. 16–17

C.5 Benchmark Instruction Generation

  • Benchmark instructions are generated from source images and drag annotations by Qwen3.5-9B, then manually checked and corrected before being added to sample metadata.The original drag points and masks remain the basis for drag-based editing and metric computation.Source: §C.5 · p. 17

C.6 Evaluation Metrics

  • Mean Distance measures average handle-to-target distance, while Concept Preservation and Prompt Following assess source-concept preservation and instruction alignment with content preservation.Image Fidelity is omitted because valid geometric transformations can increase LPIPS despite preserving intended identity.Source: §C.6 · pp. 17–18

D Algorithmic Details of MoRe-Drag

  • Inference maintains an instruction-conditioned generation trajectory and a null-condition evidence trajectory, injecting warped-observation evidence only during the recomposition interval.The generation condition switches from the warped image and restoration prompt to the original image and editing instruction across the sampling stages.Source: D Algorithmic Details of MoRe-Drag · p. 18, §C.6 · p. 18

E.1 Quantitative Results

The quantitative comparison reports that MoRe-Drag outperforms RegionDrag and DragFlow across all metrics, with its Mean Distance below the value reported in the original DragFlow paper. The supplied table captions identify additional hyperparameter and design-alternative analyses.

E.1 Quantitative Results

  • MoRe-Drag outperforms RegionDrag and DragFlow on all metrics, and its MD is lower than DragFlow’s original reported result.Table 6 compares the methods on DRAGBENCH-SR/DR; CP and PF are scaled by 100, and MoRe-Drag uses LongCat-Image-Edit.Source: Table 6 · p. 18, §E.1 · p. 18
  • 31.59 is the original DragFlow paper’s reported MD on DRAGBENCH-DR.The comparison passage states that MoRe-Drag’s MD remains below that reported value.Source: §E.1 · p. 18
  • The appendix identifies Table 7 as a DRAGBENCH-SR hyperparameter analysis and Table 8 as a comparison with design alternatives.The supplied captions establish their scope but do not provide cell values or outcomes.Source: Table 7 · p. 19, Table 8 · p. 19

E.2 Qualitative Results

Qualitative examples contrast MoRe-Drag’s accurate, natural edits with failure patterns in other drag-based methods. Design alternatives further show that role-specific recomposition and an evolving evidence branch are important to the reported balance of outcomes.

E.2 Qualitative Results

  • Across translation, pose adjustment, and motion scenarios, MoRe-Drag produces accurate and natural edits, including realistic crocodile-mouth content and a more plausible Iron Man arm than Inpaint4Drag.The cited comparisons describe distortions from warping-based editing and detail loss or inaccuracy from FastDrag.Source: §E.2 · p. 19

E.3 Comparison with Design Alternatives

  • Global blending performs worse than removing RAR, supporting separate handling of motion-evidence, disoccluded, and preservation regions.The global-blending alternative retains two branches and the blending weight but mixes the whole latent without role masks.Source: §E.3 · p. 19
  • Re-noising the warped latent degrades all metrics, while direct restoration retains drag accuracy but changes source identity and appearance; the model-evolved evidence branch balances both.The alternatives contrast timestep matching without editor updates against removing the evidence branch.Source: §E.3 · p. 19

E.4 Robustness to Input Masks

The robustness analyses examine sensitivity to input masks, disocclusion severity, and automatic instruction inference. Their reported results distinguish which aspects of editing are affected by mask errors and show improved drag accuracy with LoRA-adapted instructions.

E.4 Robustness to Input Masks

  • Mild mask perturbations cause small drops; erosion and missing regions mainly hurt MD, while strong dilation preserves MD but lowers CP and PF.SAM masks often cover only a coarse or local part of the object and cause a clear drop, so coarse user masks remain the default and automatic masks optional.Source: §E.4 · pp. 19–20, Table 9 · p. 20

E.5 Effect of Disocclusion Severity

  • The high-disocclusion DRAGBENCH-SR group has worse MD while CP and PF remain stable; low and medium groups are not strictly ordered.The text notes that disocclusion ratio only partly reflects difficulty and edit type also matters.Source: §E.5 · p. 20, Table 10 · p. 20

E.6 Effect of Instruction Interaction

  • On DRAGBENCH-SR, LoRA-adapted instruction inference reduces MD from 19.94 to 18.54 and increases CP and PF compared with the vanilla MLLM.The adapted model more accurately identifies the edited subject and translates drag geometry into editing intent.Source: §E.6 · p. 20

E.7 Hyperparameter Analysis

The hyperparameter analysis finds a trade-off between stronger motion grounding and semantic refinement, selecting (16, 12) as the default. Failure cases identify ambiguous warps and imperfect base-editor reconstruction as boundaries on performance.

E.7 Hyperparameter Analysis

  • Increasing RAR steps and delaying the GAC switch consistently lowers MD, but gains become marginal beyond (16, 12) as CP and PF decline slightly.Both settings prolong motion-evidence influence during sampling; the authors therefore use (16, 12) by default.Source: §E.7 · pp. 20–21
  • With RAR fixed at 16, switching too early degrades drag accuracy, while switching too late limits instruction-guided refinement and naturalness.Table 12 reports the effect of switching timestep tc on DRAGBENCH-SR.Source: §E.7 · p. 21, Table 12 · p. 21

E.8 Failure Cases

  • Ambiguous or misleading 2D warps can leave edits only partially aligned, especially with large disocclusions or strong 3D motions.The examples include insufficient constraints for rotation and unclear geometry for closing a lid.Source: §E.8 · p. 21
  • Closely spaced drag annotations can obscure local image content, causing subtle shape changes to be misread as object motion or orientation changes.In the tram example, injected motion evidence partly compensates for an imperfect inferred instruction.Source: §E.8 · p. 21

E.9 Comparison of Base Editor Reconstruction Ability

  • LongCat-Image-Edit reconstructs more stably than Qwen-Image-Edit, and semantic consistency and preservation depend on reconstruction ability rather than model scale alone.MoRe-Drag improves drag accuracy with both tested editors.Source: §E.9 · p. 22

E.10 Runtime and Memory Analysis

This section analyzes MoRe-Drag’s runtime and peak GPU memory usage, including average editing time, under its default sampling setup. It also describes compute-worker details and accompanying experimental materials, including prompting, evaluation, reproducibility, societal-impact, annotation-dataset, and vision-language-model documentation.

E.10 Runtime and Memory Analysis

  • Generation time rises from 30.74s to 41.31s, while peak GPU memory rises from 16.05 GB to 17.18 GB.The default applies region-aware latent recomposition for 16 of 30 sampling steps; the added evidence-branch forward passes account for the runtime overhead. Measurements average 10 samples on an NVIDIA RTX 4090 with CPU offloading enabled.Source: §E.10 · p. 22

F Declaration of LLM Usage

  • Qwen3.5-9B supports dataset construction, while GPT-4o computes PF and CP scores under DreamBench++ [49] evaluation.Qwen3.5-9B filters camera-centric samples, parses motion-centric instructions, and generates natural-language instructions for DRAGBENCH-SR and DRAGBENCH-DR.Source: F Declaration of LLM Usage · p. 22

G Prompts Used in Experiments

  • The prompts infer concise edit instructions from source images and drag annotations, grounding the edit subject and type in visible content and motion geometry.They guide inference of the intended final state from target position and drag distance, with unrelated content preserved; a separate VLM prompt scores camera motion, instruction alignment, and target alignment conservatively.Source: G Prompts Used in Experiments · pp. 23–24

NeurIPS Paper Checklist

  • The checklist reports that implementation details are in Appendix C and that data are public and code is included in the supplementary materials.The authors also report that limitations are discussed in Section 5.Source: NeurIPS Paper Checklist · pp. 30–32
  • The paper reports average editing time and peak GPU memory usage in Table 13, and describes the compute workers used for training and inference in Appendix C.Results cover the full benchmark sets; repeated-run experiments report mean and standard deviation.Source: NeurIPS Paper Checklist · pp. 32–33

10. Broader impacts

  • The paper states that it discusses potential positive and negative societal impacts in Appendix A.It also states that it reviewed and follows the NeurIPS Code of Ethics.Source: §10 · p. 33, NeurIPS Paper Checklist · p. 33
  • The release introduces no new pretrained model or scraped image dataset; it uses derived annotations and processing metadata from previously released datasets and pretrained models.The drag-instruction annotation dataset is based on previously released data, and its construction, filtering, quality scoring, and statistics are documented in Appendix B.Source: §10 · pp. 34–35

16. Declaration of LLM usage

  • The paper declares using large vision-language models for instruction generation and editing-result scoring, with details in Appendix F.Source: §16 · p. 36
Loading 2609.36755v2…