PaperlayerView exact revision on arXiv

Source-linked AI summary

DragLoRA: Online Optimization of LoRA Adapters for Drag-based Image Editing in Diffusion Model

Siwei Xia, Li Sun, Tiantian Sun, Qingli Li

arXiv:2505.12427v2cs.CV

TL;DR

Drag-based editing needs more expressive motion supervision and more efficient point tracking than direct optimization of DDIM-inverted features provides. DragLoRA performs online LoRA adaptation with DDS regularization, cyclic feature adaptation, and adaptive optimization, and experiments report improved precision and runtime. The method is intended for point-based interactive editing, but quantitative metrics are not direct and accurate, and challenging cases can still produce low-fidelity edits.

  • Problem

    Direct optimization of DDIM-inverted features can limit motion-supervision accuracy, while large search regions make iterative point tracking computationally inefficient.

  • Method

    DragLoRA replaces direct latent-feature optimization with online LoRA adaptation, combining drag and DDS losses with cyclic feature adaptation and adaptive optimization.

  • Results

    Experiments report that DragLoRA outperforms existing methods in both precision and runtime for drag-based image editing.

  • Takeaways & Limitations

    DragLoRA provides a point-based interactive editing framework that improves control precision and computational efficiency while preserving semantic fidelity.

  • Takeaways & Limitations

    The paper notes that its quantitative metrics are not direct and accurate, and that challenging cases can still produce low-fidelity edited images.

Abstract

from arXiv · show

Drag-based editing within pretrained diffusion model provides a precise and flexible way to manipulate foreground objects. Traditional methods optimize the input feature obtained from DDIM inversion directly, adjusting them iteratively to guide handle points towards target locations. However, these approaches often suffer from limited accuracy due to the low representation ability of the feature in motion supervision, as well as inefficiencies caused by the large search space required for point tracking. To address these limitations, we present DragLoRA, a novel framework that integrates LoRA (Low-Rank Adaptation) adapters into the drag-based editing pipeline. To enhance the training of LoRA adapters, we introduce an additional denoising score distillation loss which regularizes the online model by aligning its output with that of the original model. Additionally, we improve the consistency of motion supervision by adapting the input features using the updated LoRA, giving a more stable and accurate input feature for subsequent operations. Building on this, we design an adaptive optimization scheme that dynamically toggles between two modes, prioritizing efficiency without compromising precision. Extensive experiments demonstrate that DragLoRA significantly enhances the control precision and computational efficiency for drag-based image editing. The Codes of DragLoRA are available at: https://github.com/Sylvie-X/DragLoRA.

1. Introduction

DragLoRA replaces direct latent-feature optimization with online LoRA adaptation to improve precision and efficiency in drag-based editing. Its dual losses, feature adaptation, and adaptive optimization preserve semantic fidelity while stabilizing motion supervision.

  • Motivation: Drag-based editing struggles with unstable motion supervision from limited feature expressiveness and costly point tracking over large spatial regions.Existing methods use motion supervision and point tracking, but direct optimization of DDIM-inverted features limits precision while iterative tracking increases computation.
  • DragLoRA: DragLoRA dynamically adapts LoRA parameters during user interactions, expanding deformation capacity beyond static latent representations while preserving pretrained diffusion priors.The adapter is integrated into all attention layers of the pretrained Unet and decouples deformation control from fixed latent features.
  • Dual-objective optimization: The dual-objective framework combines drag loss with DDS loss comparing original and LoRA-enhanced Unet noise predictions on perturbed features.DDS regularizes online optimization so flexible deformation remains aligned with the original model’s semantic behavior.
  • Feature adaptation: Cyclic denoise-renoise feature adaptation propagates handle adjustments into latent space, stabilizing motion supervision through accumulated deformation trajectories.The process improves semantic fidelity and provides a more consistent input feature for subsequent operations.
  • Adaptive optimization: Adaptive optimization prioritizes efficient input-feature updates when tracking succeeds and switches to motion supervision for LoRA refinement when deviations occur.By toggling between two modes according to tracking quality, the strategy minimizes redundant optimization steps.

2. Related Works

Diffusion-based image editing spans text, optimization, point-dragging, region-based, reference-image, and general-purpose approaches. DragLoRA targets point-based drag editing and reports state-of-the-art key metrics with lower time costs than comparable methods, without heavy offline training.

  • Diffusion image editing: Text-based methods such as P2P, PnP, and MasaCtrl modify attention mechanisms, whereas optimization-based methods train on one image or small image sets.These approaches represent alternative ways to control diffusion-generated image editing [Hertz et al., 2022; Tumanyan et al., 2023; Cao et al., 2023; Kawar et al., 2023; Valevski et al., 2023; Gal et al., 2022].
  • Dragged-based image editing: Drag-based editing lets users deform images by specifying handle and target points, following motion supervision and point tracking mechanisms introduced by DragGAN.DragDiffusion adapts point-based dragging to Stable Diffusion, while later methods formulate, accelerate, or stabilize diffusion-based dragging [Pan et al., 2023; Shi et al., 2024b; Nie et al., 2024; Liu et al., 2024; Ling et al., 2024].
  • Extensions: Region-based, reference-image, and general-purpose methods extend drag editing beyond point-based interaction or enable rapid adaptation across tasks and datasets.DragonDiffusion, DiffEditor, and RegionDrag address region or reference-image editing, while InstantDrag and LightningDrag train general-purpose models [Mou et al., 2023; Mou et al., 2024; Lu et al., 2024; Shin et al., 2024; Shi et al., 2024a].
  • Point-based drag editing: DragLoRA is a point-based drag editing method that reports state-of-the-art key metrics while reducing time costs relative to methods in the same category.The paper positions it against other point-based methods on precision and efficiency.
  • Comparison: Unlike general models such as LightningDrag, DragLoRA avoids heavy offline training while delivering better results.Its online optimization design is presented as a distinction from offline general-purpose training.

3. Method

DragLoRA replaces direct latent-feature optimization with online LoRA adaptation, combining motion supervision, fidelity regularization, input-feature adaptation, and adaptive point tracking for efficient drag-based editing.

  • Motion Supervision and Point Tracking: Motion supervision aligns features around temporal target points with handle-point features, while mask regularization preserves non-target regions during optimization.The tracked point is reprocessed through the UNet and evaluated using matching distance and Euclidean displacement to guide subsequent supervision.
  • DragLoRA and Its Online Optimization: DragLoRA dynamically optimizes LoRA parameters instead of directly optimizing z35, increasing deformation capacity while preserving pretrained diffusion priors.The adapter is integrated into the pretrained UNet and initialized from reconstruction fine-tuning without adding extra LoRA modules.
  • DragLoRA and Its Online Optimization: The dual-objective loss combines drag supervision with DDS regularization, aligning handle points while constraining the LoRA-enhanced model toward the original model.DDS perturbs the predicted clean signal and minimizes the discrepancy between original and LoRA-enhanced UNet noise predictions.
  • Input Feature Adaptation: A cyclic denoise-renoise process adapts the foreground input feature, propagating handle adjustments into coherent LoRA updates while leaving the background untouched.The feature is denoised one step with the full LoRA-augmented model and re-noised with a DDPM schedule within the foreground mask.
  • Adaptive Optimization Scheme with Two Modes: Adaptive switching chooses ILFA-only when tracking is confident and DOO plus ILFA when occlusions or ambiguities degrade tracking quality.The minD-based decision balances the efficiency of gradient-free input adaptation with the robustness of gradient-based LoRA refinement.
  • Efficient Point Tracking: Point tracking narrows candidates to distance-closer or angle-closer regions, reducing unnecessary search while retaining a minD-based safeguard against unreliable updates.Distance-closer provides the best performance among the selected strategies, whereas angle-closer is the most efficient.

4. Experiments

Experiments on DragBench and qualitative comparisons evaluate DragLoRA's editing quality, fidelity, accuracy, efficiency, and module contributions.

  • DragLoRA outperforms existing approaches in perceptual quality and drag-editing accuracy in qualitative comparisons.The compared methods include DragDiffusion, DragNoise, and GoodDrag.
  • The evaluation uses 1-LPIPS for consistency and MD for editing accuracy, while noting that whole-image DIFT point searches can produce unreliable metrics.The drag-back evaluation supplements these metrics for substantial edits, where 1-LPIPS alone may not reflect consistency.
  • DragLoRA achieves state-of-the-art editing quality and the lowest time consumption among optimization-based approaches on DragBench.DragLoRA uses distance-closer EPT, while DragLoRA-Fast uses angle-closer EPT.
  • DragLoRA achieves better drag-back results visually and quantitatively, supporting preservation of original image information across two symmetric edits.Lower LPIPS and higher CLIP indicate stronger preservation and sufficient proximity to the target after the first edit.
  • Ablations show that DOO stabilizes drag updates, ILFA strengthens edit accuracy, and EPT and ASS improve efficiency and editability.The study incrementally adds these modules and evaluates 1-LPIPS and MD; the results are summarized in Table 3.

5. Conclusion

DragLoRA improves drag-based editing through online LoRA optimization, dual-objective regularization, feature adaptation, and adaptive optimization. Experiments report improved precision, runtime, and drag-back fidelity, while limitations remain in metric directness and challenging edits.

  • DragLoRA replaces latent-feature optimization with online LoRA adaptation to improve editing precision and efficiency while preserving semantic fidelity.Its dual objective combines drag loss with DDS loss to align online optimization with pretrained diffusion priors.
  • The framework further stabilizes motion supervision through cyclic input-feature adaptation and improves efficiency with adaptive optimization.The adaptive strategy supports efficient optimization while retaining precision.
  • Drag-back evaluation measures fidelity by applying a second, reversed drag edit and comparing the result with the input.The procedure validates image fidelity and edit accuracy across symmetric edits.
  • DragLoRA outperforms existing methods in precision and runtime, supporting interactive drag-based image editing.
  • The authors identify inaccurate quantitative metrics and challenging low-fidelity edits as limitations, including camera movements that reveal previously occluded faces.They propose future user studies and advanced generative models as possible directions.

A. Additional Results

Additional analyses evaluate point-tracking distance and optimization stability across benchmark images. DragLoRA reaches the lowest reported distances and drives handles toward targets more efficiently than existing methods.

  • DragLoRA reaches the lowest average minD through 80 optimization steps across 205 DragBench images.This result is reported as evidence of a stable and reliable optimization process.
  • DragLoRA drives handle points to target positions more efficiently than various existing methods according to dT curves.dT is the averaged Euclidean distance between handle and target points across points and images.

B. More discussion

The additional discussion analyzes DOO, ILFA, EPT, and ASS, including their contributions and qualitative ablation results.

  • The discussion examines DOO, ILFA, EPT, and ASS through module analyses and supplements quantitative ablations with qualitative results.

B.1. Dual-Object Optimization

DragLoRA adapts the DDS loss into a regularizer for drag-based editing rather than using it for semantic editing. This reversed application constrains the edited model to remain close to the original model while optimizing drag loss.

  • B.1. Dual-Object Optimization: The section explicitly analyzes how the loss introduced by Hertz et al. (2023) differs from DragLoRA’s LDDS.
  • B.1. Dual-Object Optimization: DragLoRA uses LDDS as a regularization term alongside Ldrag, reversing the original DDS application.The original DDS guides semantic editing, whereas DragLoRA uses the difference between original and edited noise predictions to regularize model adaptation.
  • B.1. Dual-Object Optimization: The reversed DDS gradient constrains the edited model to preserve the original model’s generative capability and prevents excessive updates.

B.2. Input Latent Feature Adaptation

Input Latent Feature Adaptation resolves layout conflicts caused by keeping the input feature fixed during LoRA optimization. It denoises and renoises the feature, producing more precise editing and extending to DragNoise.

  • B.2. Input Latent Feature Adaptation: ILFA denoises the input latent feature from z_t to z_t−1 and then renoises it to obtain an adapted z_t.The method combines the denoising and renoising operations into a single formulation.
  • B.2. Input Latent Feature Adaptation: ILFA mitigates conflicts between fixed input features and DragLoRA, producing more precise editing results.Fixed z35 preserves the initial layout, forcing LoRA to generate growing displacements and causing unclear object outlines during small-step supervision.
  • B.2. Input Latent Feature Adaptation: The SDS-form ILFA performs better in DragLoRA than the DDS-form according to the comparison in Table 4.ILFA uses the weighted difference between the DragLoRA prediction and random noise as the editing direction for the input feature.
  • B.2. Input Latent Feature Adaptation: ILFA improves editability and reduces layout-conflict ambiguity when applied to DragNoise, demonstrating portability beyond DragLoRA.

B.3. Efficient Point Tracking

Efficient point tracking is addressed by restricting the candidate region for tracked points. The study compares increasingly constrained strategies to reduce ambiguity and error-prone reverse-direction points.

  • B.3. Efficient Point Tracking: Large common neighborhoods can mislead tracked points with ambiguous candidates and cause dragging to become stuck.
  • B.3. Efficient Point Tracking: DragLoRA tests distance-closer, angle-closer, and linear candidate regions with progressively fewer candidate points.The tested strategies impose increasingly strict constraints on the points considered during tracking.
  • B.3. Efficient Point Tracking: The experiments include comparisons on DragNoise-based ILFA and strategies used in EPT.

B.4. Ablation Study

The ablation shows that unrestricted LoRA optimization can produce extreme edits, while DOO improves fidelity and subsequent ILFA, ASS, and EPT additions increase stable editing strength.

  • B.4. Ablation Study: Unrestricted LoRA optimization produces unexpected extreme edits, whereas DOO restricts parameter deviation and increases image fidelity.
  • B.4. Ablation Study: Adding ILFA, ASS, and EPT successively enhances the degree of editing while keeping the resulting edits stable and reliable.
  • B.4. Ablation Study: Figure 10 presents the visual ablation study across the input baseline, DOO, ILFA, and ASS & EPT configurations.
Loading 2505.12427v2…