Source-linked AI summary
Drag Your GAN: Interactive Point-based Manipulation on the Generative Image Manifold
Xingang Pan, Ayush Tewari, Thomas Leimkühler, Lingjie Liu, Abhimitra Meka, Christian Theobalt
TL;DR
Precise, flexible control over GAN-generated spatial attributes remains difficult with annotated-data, 3D-model, and text-guided approaches. DragGAN lets users drag multiple image points to targets using feature-based motion supervision and discriminative-feature tracking. It supports realistic, category-agnostic edits and real-image manipulation through GAN inversion, while its quality depends on training-data diversity and texture-rich tracking points.
Problem
Existing GAN controllability approaches often lack flexibility, precision, and generality for editing spatial attributes across object categories.
Method
DragGAN optimizes latent codes with feature-based motion supervision and tracks handle points using discriminative GAN features.
Results
DragGAN enables precise point-based manipulation across diverse categories, tends to preserve realistic object structure, and also supports real-image editing through GAN inversion.
Takeaways & Limitations
Multiple precisely controlled points enable diverse and accurate image manipulation without domain-specific modeling or auxiliary tracking networks.
Takeaways & Limitations
Editing quality is affected by training-data diversity, and textureless handle points can experience greater tracking drift.
Abstract
from arXiv · showhide
Synthesizing visual content that meets users' needs often requires flexible and precise controllability of the pose, shape, expression, and layout of the generated objects. Existing approaches gain controllability of generative adversarial networks (GANs) via manually annotated training data or a prior 3D model, which often lack flexibility, precision, and generality. In this work, we study a powerful yet much less explored way of controlling GANs, that is, to "drag" any points of the image to precisely reach target points in a user-interactive manner, as shown in Fig.1. To achieve this, we propose DragGAN, which consists of two main components: 1) a feature-based motion supervision that drives the handle point to move towards the target position, and 2) a new point tracking approach that leverages the discriminative generator features to keep localizing the position of the handle points. Through DragGAN, anyone can deform an image with precise control over where pixels go, thus manipulating the pose, shape, expression, and layout of diverse categories such as animals, cars, humans, landscapes, etc. As these manipulations are performed on the learned generative image manifold of a GAN, they tend to produce realistic outputs even for challenging scenarios such as hallucinating occluded content and deforming shapes that consistently follow the object's rigidity. Both qualitative and quantitative comparisons demonstrate the advantage of DragGAN over prior approaches in the tasks of image manipulation and point tracking. We also showcase the manipulation of real images through GAN inversion.
1 INTRODUCTION
DragGAN addresses limited controllability in GAN image synthesis with interactive point dragging that flexibly and precisely manipulates diverse spatial attributes. It combines motion supervision and point tracking to produce manifold-consistent edits across object categories.
- Existing controllable GAN methods often rely on annotated data or prior 3D models, limiting generalization, editable attributes, or editing precision.
- DragGAN lets users specify handle and target points, then drives each handle to its corresponding target for category-agnostic spatial control.
- The method jointly solves motion supervision and point tracking, using GAN feature information to optimize latent codes and localize handles during editing.
- Evaluations across animals, humans, cars, and landscapes show diverse manipulations whose deformations tend to follow learned object structures rather than simple image warping.
2 RELATED WORK
Related work spans controllable GANs, diffusion-based synthesis, conventional image deformation, and optical-flow or particle-video tracking. DragGAN instead uses discriminative GAN features for point tracking without additional tracking networks, supporting efficient interactive editing.
- Controllable GAN methods use conditional inputs, latent manipulation, annotations, 3D models, or architectural changes, but their controls can be category-, attribute-, or pose-limited.
- Text-conditioned diffusion models provide expressive synthesis, but natural-language guidance lacks fine-grained spatial control.
- Conventional mesh and feature-based deformation methods impose geometric constraints without fully modeling object structure or hallucinating occluded content.
- Optical-flow and particle-video approaches track points across images or frames, with methods such as PIPs using multiple frames for improved long-range tracking.
- DragGAN performs GAN-image point tracking through feature matching, avoiding additional neural networks and outperforming RAFT and PIPs in the reported experiments.
3 METHOD
DragGAN enables interactive point-based editing by iteratively supervising handle-point motion and tracking those points in discriminative StyleGAN2 features. It optimizes the latent representation to move user-selected points toward targets, supports masked regional editing and real-image manipulation through GAN inversion.
- Interactive Point-based Manipulation: DragGAN lets users specify handle and target points to control semantic positions in GAN-generated images, optionally restricting edits with a movable-region mask.The goal is to move image content so each handle point reaches its corresponding target.
- Interactive Point-based Manipulation: Each iteration alternates motion supervision, which optimizes the latent code, with point tracking, which updates handle positions after the image changes.The process typically continues for 30-200 iterations, and users may stop or continue editing interactively.
- Motion Supervision: Motion supervision uses a shifted feature-patch loss on StyleGAN2 generator features to move handle points incrementally toward their targets.Features after StyleGAN2’s sixth block provide a resolution-discriminativeness trade-off, while an optional reconstruction term preserves unmasked regions.
- Results and Real Image Editing: Across animals, humans, cars, and landscapes, DragGAN produces diverse, natural edits and more accurately reaches target points than UserControllableLT in qualitative comparisons.The paper also demonstrates pose, hair, shape, and expression edits on real images after GAN inversion.
- Latent Optimization: The latent code is optimized in W+ space while only the first six layers are updated, because these layers mainly affect spatial attributes and later layers mainly affect appearance.W+ is more expressive and enables better editability, though it can produce out-of-distribution manipulations.
- Point Tracking: Point tracking searches for each handle point’s original feature in a local patch of updated generator features using nearest-neighbor matching.The method uses features after StyleGAN2’s sixth block, with 256 × 256 feature maps bilinearly interpolated when needed.
4 EXPERIMENTS
DragGAN is evaluated across diverse object categories and tasks, with comparisons covering manipulation, tracking, real-image editing, masking, out-of-distribution behavior, and limitations. The results show precise point control, improved tracking and image quality over baselines, and realistic but bounded editing behavior.
- Experimental setup: Evaluations use StyleGAN2 models trained on diverse datasets spanning animals, humans, cars, landscapes, and other image domains.The experiments include FFHQ, AFHQCat, SHHQ, LSUN Car, LSUN Cat, Landscapes HQ, microscope, Lion, and Dog datasets.
- Qualitative evaluation: Compared with UserControllableLT, DragGAN more accurately reaches target points while producing natural edits and better preserving unmasked regions.The comparison covers animal pose, car shape, landscape layout, human clothing, car backgrounds, and masked cat images.
- Point tracking: DragGAN tracks handle points more reliably than RAFT and PIPs, whereas inaccurate or absent tracking can move the wrong image region and fail to stop.In the lion example, baseline tracking drifts from the nose, while no tracking eventually drives background content.
- Real image editing: GAN inversion enables DragGAN to edit real images, including face pose, hair, shape, and expression.The real-image example uses PTI inversion before applying a series of point-based manipulations.
- Quantitative evaluation: DragGAN achieves lower face-landmark matching error and better FID than UserControllableLT, while more accurate tracking improves manipulation relative to RAFT and PIPs.The quantitative evaluation considers one, five, and 68 landmarks; FID and running time are reported in the one-point setting.
- Discussions and limitations: Masks reduce ambiguity by restricting movement to selected regions, while out-of-distribution edits can extrapolate beyond training data but may produce artifacts.Masking keeps other regions almost fixed; the authors note that additional latent regularization could discourage out-of-distribution results, and training-data diversity limits editing quality.
5 CONCLUSION
DragGAN enables intuitive point-based image editing by optimizing latent codes and tracking handle points with GAN feature maps. It produces precise, realistic deformations and outperforms prior GAN-based manipulation methods.
- DragGAN uses a pre-trained GAN to produce images that follow user input while remaining on the manifold of realistic images.
- The framework avoids domain-specific modeling and auxiliary networks, providing general point-based editing across image content.
- Latent-code optimization incrementally moves multiple handle points toward target locations during editing.
- Point tracking uses intermediate GAN feature maps to faithfully trace handle trajectories and support pixel-precise deformations.
- DragGAN outperforms the state of the art in GAN-based manipulation and motivates extending point-based editing to 3D generative models.