PaperlayerView exact revision on arXiv

Source-linked AI summary

ThinkDrag: Semantic Drag-Based Image Editing with Visual Reasoning

Armando Fortes, Tianyi Wei, Chindanai Trakantannarong, Shangchen Zhou, Xingang Pan

arXiv:2610.13316v1cs.CV

TL;DR

Drag-based editing gives users spatial control, but many methods treat drags as geometric displacements rather than interpreting the intended semantic transformation. ThinkDrag combines semantic drag supervision, optional visual reasoning, and correspondence markers; it achieves state-of-the-art generation quality with competitive point-following precision. Its reasoning mode adds interpretability, particularly for ambiguous multi-drag edits, but adds inference latency.

  • Problem

    Many drag-editing methods treat point instructions as geometric displacements, although the intended edit can depend on image content, object structure, and user intent.

    Source: §1 · p. 1

  • Method

    THINKDRAG trains a unified multimodal model on semantic drag transformations and reasoning traces, using Drag Endpoint Markers to ground handle–target correspondence in image space.

    Source: §1 · p. 2

  • Results

    THINKDRAG achieves state-of-the-art generation quality while maintaining competitive point-following precision, and its reasoning mode adds interpretability for ambiguous multi-drag edits.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    The reasoning mode provides additional interpretability, particularly when edits involve ambiguous multi-drag instructions.

    Source: §5 · p. 9

    The reasoning mode adds inference latency, and its benefits are concentrated in ambiguous multi-drag edits.

    Source: D LIMITATIONS AND FUTURE WORK · p. 19

Abstract

from arXiv · show

Drag-based image editing provides an intuitive interface for spatially controlled image manipulation by allowing users to specify handle and target points. Existing methods have made substantial progress through optimization-based, one-step, and feed-forward formulations, but they often interpret drag instructions as geometric displacements. This limits their effectiveness when the desired edit depends on interpreting image content and inferring a plausible semantic transformation from sparse point constraints. We introduce ThinkDrag, a unified multimodal framework for drag-based image editing with visual reasoning. ThinkDrag is trained to associate point constraints with meaningful object- and scene-level transformations, and can optionally produce an explicit reasoning trace that interprets the intended edit before image generation. This reasoning trace improves interpretability, particularly for ambiguous drags. Drag Endpoint Markers, motivated by an analysis of cross-image attention, provide an explicit correspondence signal that grounds each drag in image space. To support this framework, we construct a supervised dataset of semantic drag transformations paired with reasoning traces and introduce DragBench++, a benchmark targeting challenging drag-based editing scenarios with reference edit solutions. Experiments show that ThinkDrag achieves state-of-the-art performance, improving generation quality and plausibility while maintaining competitive point-following precision.

1 INTRODUCTION

Drag-based editing offers spatial control, but existing methods often treat point instructions as displacements even when object-aware interpretation is needed. ThinkDrag combines semantic reasoning and correspondence grounding, and its evaluation reports strong benchmark performance and user preference.

1 INTRODUCTION

  • Existing methods often treat drags as spatial displacements, although the same handle–target movement can require different edits depending on the object and user intent.For example, moving a point on an articulated object may call for rotating a part rather than translating a local patch.Source: §1 · p. 1
  • THINKDRAG learns semantic interpretations of sparse point constraints from supervised drag transformations paired with reasoning traces, and uses Drag Endpoint Markers to encode handle–target correspondence.The model can edit directly or reason first when an edit is ambiguous or needs higher-level scene understanding.Source: §1 · p. 2
  • 80% of overall-preference votes on DragBench++ and 61% on DragBench favored THINKDRAG; it achieves state-of-the-art generation quality with competitive point-following precision.The reasoning mode makes the model’s interpretation explicit and is particularly useful for ambiguous multi-drag edits.Source: §1 · p. 2
  • DragBench++ evaluates challenging edits requiring higher-level scene understanding and semantic interpretation, with reference edit solutions.The paper also reports state-of-the-art results on drag-editing benchmarks and joint text-drag conditioning, where text disambiguates intent and drags provide spatial control.Source: §1 · p. 2

2 RELATED WORK

Unified multimodal models combine visual-language understanding with image generation, while multimodal reasoning and synthesis remain underexplored for spatially grounded manipulation. Prior drag-editing approaches span optimization, one-step, and feed-forward methods; ThinkDrag instead interprets instructions semantically and uses explicit image-space correspondence.

2.1 UNIFIED MULTIMODAL UNDERSTANDING AND GENERATION

  • Unified multimodal models combine image and text modeling or autoregressive language modeling with diffusion generation; multimodal reasoning and synthesis for spatially grounded manipulation remain underexplored.Earlier paradigms separated perception and reasoning from high-fidelity image synthesis.Source: §2.1 · pp. 2–3

2.2 DRAG-BASED IMAGE EDITING

  • Drag-editing methods include test-time optimization, one-step alternatives, and feed-forward generation, while THINKDRAG interprets instructions within a unified multimodal model and grounds edits with explicit image-space correspondence.Prior feed-forward methods commonly treat drags as geometric constraints; Cui et al. (2024) infer intended edits with an LLM at test time.Source: §2.2 · p. 3

3 THINKDRAG

ThinkDrag builds semantic drag supervision and trains a unified multimodal editor to interpret sparse point constraints. Its conditioning combines textual coordinates with spatial endpoint markers, and it supports both direct and reasoning-conditioned editing.

3.1 DATASET CONSTRUCTION

  • Unlike video-derived correspondences that can be geometric, unreliable under occlusion, or corrupted by camera motion, the dataset targets spatial edits rather than semantic discontinuities.It focuses on transformations such as translation, pose change, deformation, and local warping, excluding insertion, removal, and global style transfer.Source: §3.1 · p. 3
  • An MLLM-assisted pipeline generates source images and compatible edit prompts, synthesizes target images, then filters invalid or low-quality pairs.Source prompts come from subjects and environments, and target images are synthesized with FLUX.2.Source: §3.1 · p. 3
  • Gemini identifies semantic keypoint correspondences, which are checked with DINOv3 features and manual inspection; reasoning chains are generated for both edit directions.The chains interpret each drag and specify which regions should remain unchanged.Source: §3.1 · pp. 3–4
  • Verification yields approximately 11K image pairs from 7K source images, or approximately 22K bidirectional editing samples, with 2.15 drags per sample on average.74% of samples contain multiple drags, and reasoning traces average 123 words.Source: §3.1 · p. 4

3.2 DRAG CONDITIONING IN A UNIFIED MULTIMODAL MODEL

  • Text tokens expose normalized handle and target coordinates, while shared endpoint markers anchor each drag at corresponding VAE token locations and encode drag identity.The same marker is added at both endpoints, giving the handle and target a shared signature that biases the target to attend to the handle.Source: §3.2 · pp. 4–5

3.3 JOINT GENERATION MODELING

  • Direct editing generates from the multimodal prompt, while reasoning-based editing first produces a chain that conditions subsequent image generation.The chain describes the scene, interprets each drag semantically, and summarizes the intended transformation.Source: §3.3 · p. 5
  • Training combines autoregressive reasoning-chain prediction with BAGEL’s rectified-flow image-generation objective, weighted together in the final loss.The reasoning loss weight controls the contribution of reasoning supervision.Source: §3.3 · pp. 5–6

4 EXPERIMENTS

Experiments evaluate ThinkDrag on DragBench and the more semantically demanding DragBench++, comparing editing quality, point-following, and user preference. They also examine joint text-drag conditioning and the contributions of reasoning and Drag Endpoint Markers.

4.1 EXPERIMENTAL SETUP

  • DragBench++ contains 145 samples emphasizing higher-level scene understanding and semantic interpretation, with more drags per sample and longer displacements than DragBench.Its reference edits provide at least one plausible solution per example and support reference-based evaluation; they are used only for evaluation.Source: §4.1 · p. 6
  • Editing accuracy is measured with MD and MDMLLM, while generation quality is assessed with IF and FID or reference-based IF* on DragBench++.MDMLLM uses GPT-5.5 for point tracking because DIFT-based correspondence can be unreliable under semantic ambiguity or large structural changes.Source: §4.1 · p. 6

4.2 COMPARISON WITH BASELINES

  • ThinkDrag more consistently produces spatially grounded, semantically plausible edits, while baselines may under-follow drags, introduce artifacts, or use implausible changes.On DragBench, ThinkDrag also better preserves identity and local appearance during edits.Source: §4.2 · p. 6
  • ThinkDrag achieves the best generation quality across both benchmarks and the best MD and MDMLLM on DragBench++, while remaining competitive in point-following on DragBench.On DragBench, test-time optimization and one-step methods achieve the strongest MD scores, consistent with their explicit inference-time enforcement of point displacement.Source: §4.2 · p. 6, Table 1 · p. 7
  • Participants gave ThinkDrag 80% of overall-preference votes on DragBench++ and 61% on DragBench.On DragBench, ThinkDrag also received 62% of edit-accuracy votes despite a higher DIFT-based MD than three baselines.Source: Table 3 · p. 7, §4.2 · p. 7

4.3 JOINT TEXT-DRAG CONDITIONING

  • Joint text-drag conditioning lets different text prompts resolve the same drag into different valid edits, while the drag points spatially anchor each transformation.Text conditioning is used during training for 10% of samples, but is not mandatory at inference.Source: §4.3 · p. 7, Figure 7 · p. 9

4.4 ABLATION STUDIES

  • Reasoning improves reference-based fidelity over direct generation, and yields lower MDMLLM more often for multi-drag edits than single-drag edits: 56% versus 36%.The text prompt changes MDMLLM only slightly, while reasoning helps interpret the intended transformation.Source: §4.4 · pp. 7–9, Table 4 · p. 9
  • On DragBench++, target-point attention peaks at the handle for 51% of drags with DEM, compared with 6% without it.Adding DEM further reduces MD after training on the dataset without DEM, with similar IF and FID; the attention analysis links endpoint markers to improved spatial precision.Source: §4.4 · p. 9, Figure 8 · p. 9, Table 5 · p. 9

5 CONCLUSION

ThinkDrag interprets sparse drag constraints through scene semantics and object structure, using semantic drag transformations, reasoning traces, and Drag Endpoint Markers. Experiments report strong generation quality and competitive point-following, with reasoning especially useful for ambiguous multi-drag edits.

5 CONCLUSION

  • ThinkDrag learns to interpret sparse point constraints in relation to scene content and object structure rather than only as geometric displacements.It is trained on semantic drag transformations and reasoning traces, and grounds drags with Drag Endpoint Markers.Source: §5 · p. 9
  • DragBench++ adds challenging drag-editing scenarios with reference edits.Source: §5 · p. 9
  • ThinkDrag achieves state-of-the-art generation quality while maintaining competitive point-following precision, and reasoning adds interpretability particularly for ambiguous multi-drag edits.Source: §5 · p. 9

A.1 ADDITIONAL VISUAL RESULTS

Additional qualitative examples illustrate ThinkDrag's spatial accuracy and its preservation of image identity and visual quality.

A.1 ADDITIONAL VISUAL RESULTS

  • Target-point overlays on generated images make editing accuracy easier to inspect in the additional comparisons.Source: §A.1 · p. 13
  • The examples illustrate spatially accurate edits that preserve image identity.Source: §A.1 · p. 13
  • The additional examples also illustrate preservation of visual quality.Source: §A.1 · p. 13

A.2 REASONING ANALYSIS

The reasoning analysis compares explicit reasoning with direct generation and assesses whether ThinkDrag grounds textual coordinates in image content. Its traces generally align with annotated edit prompts, while coordinate-grounding examples show ThinkDrag identifying the queried object and part.

A.2 REASONING ANALYSIS

  • Reasoning wins 75 of 145 DragBench++ samples (51.7%); its win rate is 56% for multi-drag edits versus 36% for single-drag edits.Traces match annotated edit prompts for 78.6% of samples and deliberately incorrect prompts for 2.8%; the authors call the first figure a conservative estimate because ambiguous drags can have multiple valid interpretations.Source: §A.2 · p. 13, Table 6 · p. 13
  • Table 7 breaks down reasoning-versus-direct win rates by edit type, grouping verbs with fewer than five samples under Other.Edit type is determined by the first verb of the edit instruction.Source: Table 7 · p. 13

A.3 COORDINATE GROUNDING

  • Coordinate grounding is evaluated on 250 annotated handle points, with examples showing ThinkDrag identifying the queried object and part while BAGEL describes unrelated objects.The evaluation supplies each model an image and normalized coordinate, then compares its object-and-part identification with labels.Source: §A.3 · p. 13, Table 8 · p. 13

A.4 ATTENTION ANALYSIS: TARGET POINT TO HANDLE CORRESPONDENCE

The attention analysis uses cross-image attention peaks to assess whether target points draw on their corresponding source handles. DEM increases handle-peaking hits on both benchmarks; the appendix also reports runtime, memory, and implementation settings.

A.4 ATTENTION ANALYSIS: TARGET POINT TO HANDLE CORRESPONDENCE

  • Figure 9 overlays target points on generated images for DragBench and DragBench++ comparisons; given an image and normalized coordinate, Figure 10 shows THINKDRAG identifies the queried object and part, whereas BAGEL often describes an unrelated object.Source: Figure 9 · p. 14, Figure 10 · p. 15
  • DEM substantially increases the fraction of drags whose target-point attention peaks at the corresponding handle on both benchmarks.A hit is a peak in the handle’s latent cell or one of its eight neighbors; drags shorter than three cells are excluded because handle and target patches overlap.Source: §A.4 · p. 15, Table 9 · p. 15

A.5 RUNTIME AND MEMORY MEASUREMENT

  • Reasoning mode increases runtime because it generates an autoregressive reasoning chain before image synthesis, while peak GPU memory remains nearly unchanged.Runtime and peak memory are measured on one NVIDIA H200 GPU; Table 10 reports the measurements.Source: §A.5 · p. 15, Table 10 · p. 15

B.1 IMPLEMENTATION DETAILS

  • ThinkDrag uses rank-256 LoRA adapters trained for 100K steps, and inference defaults to 50 sampling steps with guidance scales of 3.0 for text and 1.5 for image conditioning.Training takes around 60 hours on four NVIDIA H200 GPUs; up to 10 drag pairs per image are supported during training.Source: §B.1 · p. 16

B.2 MLLM-BASED EVALUATION

The MLLM-based evaluation uses a vision-language model to track source handles in generated images, alongside DIFT tracking. The appendix reports that MLLM tracking is more robust in challenging cases and presents user-study preferences favoring ThinkDrag.

B.2 MLLM-BASED EVALUATION

  • MLLM-based tracking is more robust than DIFT-based tracking in challenging cases involving semantic ambiguity, large structural changes, or visually similar distractors.The protocol asks GPT-5.5 to return corresponding physical or semantic handle locations in the generated image; the authors note that MLLM tracking is also imperfect.Source: §B.2 · p. 16, Figure 11 · p. 17
  • On reference edits from DragBench++, DIFT reports a mean distance of 43.42, while GPT-5.5 reports 27.38.Because the reference edits satisfy their drags by construction, the authors attribute most DIFT distance on this benchmark to tracking failures; they regard DIFT and MLLM tracking as complementary because DIFT remains accurate on more local DragBench edits.Source: §B.2 · p. 16

B.3 USER STUDY

  • In the user study, ThinkDrag receives about 70% of votes under each criterion and the most votes on 18 of 20 questions.Thirty participants each answered 20 questions sampled evenly from DragBench and DragBench++; Table 11 reports vote shares for editing accuracy, generation quality, and overall preference.Source: §B.3 · p. 17, Table 11 · p. 17

C DATASET DETAILS

The dataset covers several semantic edit types, led by rotations and raising or lowering. DragBench++ uses held-out source images with manually annotated drags, and a mask check indicates that the masks cover edited regions.

C DATASET DETAILS

  • Rotations account for 28.9% of the final dataset, raising and lowering for 24.3%, and translations for 16.6%.Source: C DATASET DETAILS · p. 18
  • Opening and closing make up 10.8% of edits, deformations 5.5%, scaling 3.8%, with the remainder listed as other edits.Source: C DATASET DETAILS · p. 18
  • DragBench++ pairs use source images absent from training and manually annotated drags; LPIPS is 0.118 inside the provided masks and 0.018 outside, supporting that masks cover edited regions.Table 12 summarizes the dataset-construction pipeline stages.Source: C DATASET DETAILS · p. 19, Table 12 · p. 19

D LIMITATIONS AND FUTURE WORK

ThinkDrag’s reasoning mode adds latency and is most beneficial for ambiguous multi-drag edits; its language-based traces may not capture fine-grained visual transformations. DragBench++ and automatic metrics also leave evaluation challenges, motivating more robust protocols.

D LIMITATIONS AND FUTURE WORK

  • Reasoning generation adds inference latency, and its benefit is concentrated in ambiguous multi-drag edits.The authors suggest distillation or adaptive reasoning that invokes it only when an edit is ambiguous.Source: D LIMITATIONS AND FUTURE WORK · p. 19
  • Language-space reasoning improves interpretability but may not fully represent fine-grained visual transformations.The authors suggest exploring reasoning in latent or pixel space.Source: D LIMITATIONS AND FUTURE WORK · p. 19
  • DragBench++ expands evaluation toward semantic and ambiguous edits, but automatic metrics only partially reflect human preference.The authors call for more robust protocols when multiple valid edits can satisfy the same instruction.Source: D LIMITATIONS AND FUTURE WORK · p. 19
  • Drag-based editing remains under-constrained, especially when multiple valid edits satisfy the same instruction.The authors identify this as a setting for future benchmark development.Source: D LIMITATIONS AND FUTURE WORK · p. 19
Loading 2610.13316v1…