PaperlayerView exact revision on arXiv

Source-linked AI summary

GeoDiffuser: Geometry-Based Image Editing with Diffusion Models

Rahul Sajnani, Jeroen Vanbaar, Jie Min, Kapil Katyal, Srinath Sridhar

arXiv:2404.14403v2cs.CV

TL;DR

Existing image-editing methods may be imprecise, require additional input, or be limited to 2D edits. GeoDiffuser embeds geometric transformations in diffusion-model attention in a zero-shot optimization method, and the authors report better quantitative results than existing methods. Large 3D rotations that create foreground disocclusions remain a limitation.

  • Problem

    Existing approaches can be imprecise for spatially controlled edits, require additional input, or support only limited geometric editing.

    Source: §1 · p. 1

  • Method

    GeoDiffuser uses zero-shot optimization to incorporate geometric transforms into shared diffusion-model attention layers for object editing.

    Source: §1 · p. 2, §4.1 · p. 5, §4.2 · p. 6

  • Results

    The authors report that GeoDiffuser outperforms existing methods quantitatively while supporting a variety of image edits.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    One approach handles common 2D and 3D object edits on images without additional training.

    Source: §1 · p. 2, §6 · p. 8

    GeoDiffuser cannot yet handle foreground-object disocclusions caused by large 3D rotations, which require accurate novel-view synthesis.

    Source: §6 · p. 8

Abstract

from arXiv · show

The success of image generative models has enabled us to build methods that can edit images based on text or other user input. However, these methods are bespoke, imprecise, require additional information, or are limited to only 2D image edits. We present GeoDiffuser, a zero-shot optimization-based method that unifies common 2D and 3D image-based object editing capabilities into a single method. Our key insight is to view image editing operations as geometric transformations. We show that these transformations can be directly incorporated into the attention layers in diffusion models to implicitly perform editing operations. Our training-free optimization method uses an objective function that seeks to preserve object style but generate plausible images, for instance with accurate lighting and shadows. It also inpaints disoccluded parts of the image where the object was originally located. Given a natural image and user input, we segment the foreground object using SAM and estimate a corresponding transform which is used by our optimization approach for editing. GeoDiffuser can perform common 2D and 3D edits like object translation, 3D rotation, and removal. We present quantitative results, including a perceptual study, that shows how our approach is better than existing methods. Visit https://ivl.cs.brown.edu/research/geodiffuser.html for more information.

1. Introduction

Existing image-editing methods can be imprecise for spatial control, difficult to extend to geometric edits, or require extra input. GeoDiffuser unifies 2D and 3D object edits by embedding geometric transformations in diffusion-model attention, and reports better quantitative results than existing methods.

1. Introduction

  • Text-based edits can lack spatial precision, object stitching and removal are difficult to extend to geometric edits, and point- or flow-based methods require additional input.Examples of additional input include a text prompt or optical flow.Source: §1 · p. 1
  • GeoDiffuser treats common image-editing operations as geometric transformations and unifies multiple object-editing capabilities in one method.The transformations are applied to parts of the image, such as through 2D translation or 3D rotation.Source: §1 · pp. 1–2
  • The zero-shot optimization method incorporates geometric transforms into diffusion-model shared attention layers to preserve object style while enabling 2D and 3D edits.It requires no additional training and can support diffusion models with attention layers.Source: §1 · p. 2
  • GeoDiffuser applies transforms through self- and cross-attention to produce edits with lighting, shadows, and reflections while inpainting disoccluded regions.The workflow segments the object and computes a transform from user input; it can optionally extract depth for 3D edits.Source: §1 · p. 2
  • The authors report that GeoDiffuser outperforms existing methods quantitatively and supports multiple 2D and 3D editing operations with one approach.Evaluation includes a perceptual study and metrics for preservation of foreground and background content.Source: §1 · p. 2

2. Related Work

Prior work spans text-guided, point- or stroke-based, and geometry-aware editing, but each approach has constraints on spatial control, edit types, or required inputs. GeoDiffuser is presented as a training-free approach to precise 2D and 3D transforms without additional user input.

2. Related Work

  • Text-guided methods support appearance or object changes, but specific 3D geometric transforms are difficult to direct with prompts, while point- and stroke-based edits do not support translations.Some point-based approaches preserve identity, but use motion supervision or latent optimization.Source: §2 · p. 2
  • ObjectStitch can translate objects using a masked target region and inpainting, but preserving object style is difficult; a lifting-based 3D approach requires an additional text prompt.The latter approach uses a depth-to-image generative model conditioned on input depth.Source: §2 · pp. 2–3
  • Some training-free methods allow translation and scaling but remain limited to 2D, while several 3D-aware approaches require difficult-to-obtain multiview images.Concurrent drag-based methods use extensive training on large-scale video data but do not embed object geometry in the architecture.Source: §2 · pp. 2–3
  • GeoDiffuser is presented as using precise geometric transformations for 2D and 3D edits while preserving identity and requiring no additional user input.The authors contrast this with flow-guided editing, for which optical flow can be harder to obtain than depth.Source: §2 · p. 3

3. Background

The background describes diffusion as iterative noise addition and removal, then introduces conditioning, attention, and an inversion-based editing framework. In that framework, a reference diffusion process guides a parallel edit process through shared attention.

3. Background

  • DDPM adds noise to an image over timesteps according to a schedule and Gaussian transitions, then trains a network to estimate the source noise for denoising.The noised image is expressed directly from the input image and sampled noise using the cumulative schedule.Source: §3 · p. 3
  • Diffusion can be conditioned on images or text; DDIM reduces denoising timesteps, and latent diffusion performs the process in a lower-dimensional feature space.The paper uses latent diffusion with classifier-free guidance for text conditioning.Source: §3 · p. 3
  • Attention computes a softmax attention map from queries and keys, then applies it to values; cross-attention can relate different input sources, such as words and image pixels.Source: §3 · p. 3
  • After inversion supplies a noise latent, parallel reference and edit diffusion processes regenerate the input and edited image, with shared attention guiding the edit and helping preserve unedited regions.Figure 2 depicts the inversion trajectory and the reference-guided edit process; GeoDiffuser modifies shared attention for geometric edits.Source: §3 · pp. 3–4, Figure 2 · p. 4

4. GeoDiffuser

GeoDiffuser unifies segmented-object editing by expressing user-specified 2D and 3D operations as geometric transformations, then using diffusion attention and optimization to produce edits and fill disoccluded regions.

4. GeoDiffuser

  • GeoDiffuser handles 2D translation and scaling, 3D rotation, translation, or scaling, and object removal, inpainting background disoccluded by the edit.The method targets segmented foreground objects and can use a depth map for 3D edits or a constant depth of 0.5 m.Source: §4 · p. 4

4.1. Edits via Shared Attention

  • The method represents edits as per-pixel geometric transformations, with depth supporting 3D edits and optical flow as an alternative when the edit field is known.The paper notes that optical flow is harder to obtain for a single image than depth maps.Source: §4.1 · p. 5
  • Transforming reference query embeddings guides edit attention, while reference values help change object geometry without changing its appearance.For self-attention, edit attention uses reference keys; for cross-attention, it uses edit keys to allow gradients to reach null-initialized text embeddings.Source: §4.1 · p. 5
  • The output combines transformed reference guidance in the edited foreground with edit guidance elsewhere to preserve object and background identity.The foreground mask is transformed along with the reference guidance.Source: §4.1 · p. 6

4.2. Optimization

  • Optimization combines background preservation, object placement, disocclusion inpainting, and smoothness losses while adjusting latents and null-initialized text embeddings.The inpainting objective favors background features over repeated foreground features in disoccluded regions.Source: §4.2 · p. 6
  • The removal-loss weight adapts to its value because the inpainting loss is difficult to optimize and varies across images.The paper also penalizes foreground depth-smearing artifacts with an amodal loss.Source: §4.2 · p. 6

5. Results & Experiments

GeoDiffuser is evaluated on real-image 2D and 3D edits using edit-adherence and scene-preservation metrics, alongside a perceptual study. The reported comparisons favor GeoDiffuser overall, with ablations examining geometric attention sharing and loss components.

5. Results & Experiments

  • The evaluation collected 70 real images and measured 102 edits using Mean Distance, Warp Error, and CLIP similarity because FID and Image Fidelity do not assess edit adherence or disocclusion inpainting.The edits comprised 36 2D and 66 3D edits.Source: §5 · p. 6, §5.1 · p. 7

5.1. Quantitative Evaluation

  • GeoDiffuser has MD(2D) 5.579 and MD(3D) 7.304 and the least Mean Distance and Warp Error among the compared methods.The comparison covers 2D and 3D edits against Dragon Diffusion, FreeDrag, Diffusion Self Guidance, and Diffusion Handles.Source: §5.1 · p. 7, Table 1 · p. 7
  • Participants preferred GeoDiffuser’s inpainting over LaMa 94.06% of the time and its edits over Zero123-XL 86.48% for realism and 88.48% for edit adherence.The forced-choice study included 53 participants.Source: §5.1 · p. 7
  • Most methods preserved scene context with CLIP similarity above 0.920, while Diffusion Handles and Diffusion Self Guidance scored 0.890 and 0.809 on 3D edits.FreeDrag had 3D CLIP similarity of 0.977 versus GeoDiffuser’s 0.967, but sometimes failed to remove the foreground appropriately.Source: §5.1 · p. 7
  • More geometric-attention-sharing steps improve style preservation, while ablations link placement, background-preservation, and removal losses to edit accuracy, scene consistency, and avoiding duplicates.The ablation caption also reports improvements in Mean Distance, Warp Error, and CLIP similarity from more sharing steps and adaptive optimization.Source: Table 2 · p. 7, §5.1 · p. 8, Figure 5 · p. 8
  • GeoDiffuser transforms or removes objects together with their reflections and shadows in qualitative 3D results.Figure 4 compares the intended edit with prior methods; its caption reports that GeoDiffuser best adheres to the edit and preserves the scene without prompts.Source: §5.1 · p. 8, Figure 4 · p. 8

6. Conclusion

The conclusion presents GeoDiffuser as one zero-shot diffusion-based method for common 2D and 3D object edits. It reports better results than prior work while identifying limits for large rotations and artifacts.

6. Conclusion

  • GeoDiffuser formulates image editing as geometric transformation and incorporates it into shared attention layers of a diffusion editing framework.The conclusion describes it as a unified method for common 2D and 3D object edits.Source: §6 · p. 8
  • The authors report that one approach handles a wide variety of image edits and produces better results than prior work.Source: §6 · p. 8
  • Large 3D rotations can create foreground disocclusions that the method cannot yet handle; it also occasionally produces artifacts and depends on the base diffusion model’s capabilities.Handling foreground gaps requires accurate novel-view synthesis of in-the-wild objects, which the authors describe as difficult.Source: §6 · p. 8

A. Qualitative Results

The section points to additional qualitative examples and comparisons of GeoDiffuser’s edits.

A. Qualitative Results

B. Implementation Details

The implementation optimizes latents and text embeddings during reverse diffusion and reports runtimes and baseline-specific settings. The comparisons describe distinct speed, edit-quality, and appearance trade-offs, while attention visualizations show the car and its shadow moving together.

B. Implementation Details

  • GeoDiffuser shares attention across UNet blocks through step 45 and optimizes every other step in the first 32 diffusion steps.On an Nvidia RTX3090 at 512 resolution, reported runtimes range from 30 seconds for removal to 45 seconds for 2D and 3D edits, including inversion and generation.Source: B. Implementation Details · p. 9

C. Evaluation and Baselines

  • The evaluation details separate baseline procedures and timing, with FreeDrag, Diffusion Handles, Dragon Diffusion, Zero123-XL, and DSG assessed against GeoDiffuser.Source: C. Evaluation and Baselines · p. 9

C.1. FreeDrag [31]

  • FreeDrag uses 40 sampled object-mask points, 200 LoRA steps, and 1000 drag-optimization steps, taking 116.26 and 165.24 seconds for those stages.The authors increased the optimization from 300 to 1000 steps because large edits did not converge correctly at 300; they report that FreeDrag averages nearby vectors and may stretch objects or miss edits.Source: §C.1 · p. 9

C.2. Diffusion Handles [45]

  • Diffusion Handles combines depth-based projection with inpainting, but its style preservation depends on depth maps matching its training distribution.The reported procedure takes 60 seconds for null-text optimization and 35 seconds for editing; merged depth can also produce improper removal or duplicate objects.Source: §C.2 · p. 9

C.3. Dragon Diffusion [40,41]

  • Dragon Diffusion edits in about 20 seconds but performs poorly on disocclusion inpainting and foreground preservation.The authors report that it uses a memory bank and avoids 3D geometry projection; incomplete removal can leave duplicates and inflate CLIP similarity.Source: §C.3 · p. 10

C.4. Zero123-XL + LaMa [33,59]

  • Zero123-XL plus LaMa takes about five seconds per edit, but the reported results do not preserve foreground appearance accurately or control per-pixel transforms well.The authors attribute these issues to real images being outside the model’s training distribution and to its inferred object geometry.Source: §C.4 · p. 10

C.5. Diffusion Self Guidance (DSG) [17]

  • DSG can lose appearance detail with strong shape guidance or fail to move objects with strong appearance guidance, reflecting entanglement of geometry and appearance.The described implementation uses 50 DDIM steps and takes 50 seconds per edit.Source: §C.5 · p. 11

D. Edit Attention Progression

E. Metrics

The section defines three measures for edit adherence and content preservation, then shows how geometric attention sharing and optimization choices affect visual results.

E. Metrics

  • Mean Distance (MD) measures target-location error at foreground interest points, Warp Error (WE) measures foreground preservation and edit adherence, and Clip Similarity (CS) measures source-to-edit content preservation.MD uses SiFT interest points and DiFT correspondences; WE compares the warped foreground with the edited image using L1 error, while CS compares image embeddings by cosine similarity.Source: E. Metrics · p. 11
  • A good edit should have low MD and WE and high CS.These directions respectively favor edit adherence, foreground preservation, and image-content preservation.Source: E. Metrics · p. 11
  • Without adaptive optimization, the same losses inpaint some images but fail on others; adaptive optimization makes the loss function work across images.Figure 7 illustrates this contrast.Source: Figure 7 · p. 11
  • Increasing geometric attention-sharing steps better preserves object style in translation edits.Figure 8 shows this relationship as the number of steps t increases.Source: Figure 8 · p. 11

F. Ablations

The visual ablations show that attention sharing supports style preservation and edit adherence, while adaptive optimization avoids image-specific loss-weight tuning.

F. Ablations

  • Skipping geometric attention sharing through step 45 degrades style preservation, while adaptive optimization avoids image-specific loss-weight tuning.The ablations identify both choices as important to the visual results and scalable use of the same losses.Source: F. Ablations · pp. 11–12
  • Across Stable Diffusion versions, inpainted backgrounds and line completeness vary, but geometric attention sharing keeps the foreground aligned with the edit.Figure 9 compares the same edit across models.Source: Figure 9 · p. 12, F. Ablations · p. 12

G. Perceptual Study

The perceptual study favors GeoDiffuser for removal realism, edit realism, and edit adherence. Further sections describe failure boundaries, optional editing behaviors, correction and loss mechanisms, and future applications.

G. Perceptual Study

  • Participants preferred GeoDiffuser for removal edit realism in 94.06% of 318 choices, for edit realism in 86.48% of 636 cases, and for edit adherence in 88.48% of 636 cases.The study used 70 images and 53 participants; edit adherence included both 2D and 3D edits.Source: G. Perceptual Study · p. 12

H. Failure Cases

  • Diffusion generation can yield sub-optimal foreground or background results, and the method cannot generate novel views with large rotations.The authors identify large rotations as a future direction.Source: H. Failure Cases · p. 12, Figure 11 · p. 12

I. Miscellaneous Edits

  • Turning off optimization or setting the removal loss to zero enables object duplication.Figure 12 illustrates this alternative behavior.Source: §I · p. 12, Figure 12 · p. 12

K. Object Removal

  • The object-removal loss compares edited and reference attention maps, using foreground-to-background and foreground-to-foreground correlations and background-coordinate distance.The algorithm uses these quantities to form Lremove.Source: §I · p. 13, K. Object Removal · p. 13

J. Diffusion Correction

  • When a transform is incorrect, Diffusion Correction lets the model partly disregard it and find correspondences for more realistic results.It is enabled for the last 15 reverse-diffusion steps and is useful for sharp structures such as buildings.Source: J. Diffusion Correction · p. 13, Figure 13 · p. 13

L. Amodal Loss

  • A small amodal-loss penalty targets foreground regions smeared by transformation, using an interpolated amodal mask and an L1 feature difference.The mask is obtained by interpolating features after reprojection.Source: L. Amodal Loss · p. 13

M. Future Work & Impact

  • The authors propose that geometry-based attention manipulation and optimization may inform diffusion-model analysis and video editing, while novel-view synthesis for real-world scenes remains future work.They also suggest these ideas might improve Score Distillation Sampling.Source: M. Future Work & Impact · p. 13

N. Discussion on Concurrent Works that Train on Video Data

The discussion contrasts GeoDiffuser with video-trained editing methods, describes interface choices, and reports results on complex shapes and human images alongside remaining boundaries.

N. Discussion on Concurrent Works that Train on Video Data

  • Video-trained drag editors can offer accurate in-distribution inpainting and faster inference, but require large-scale training, may move backgrounds, and can lose geometric consistency or identity beyond their flow range.The authors suggest GeoDiffuser's geometry attention sharing and loss functions could help address geometry consistency in future video models.Source: N. Discussion on Concurrent Works that… · pp. 13–14

O. Discussion on Slider based UI as opposed to Drag UI

  • The slider interface supports precise rotation control and geometry preservation more easily than drag-based input, though a drag-based interface remains possible.The authors note that controlling rotations is difficult with drag-based input.Source: O. Discussion on Slider based UI as… · p. 14

Q. Complex Shapes and Human Edits

P. User Interface

Loading 2404.14403v2…