Delta Denoising Score

1. Introduction

• SDS editing can blur image details beyond the elements targeted by the prompt.The introduction also describes SDS outputs as blurry and focused on prompt-mentioned elements.Source: §1 · p. 2
• DDS subtracts the score from a matched reference image-text query from the target-query score to guide image editing.The reference text matches the reference image content.Source: §1 · p. 2
• In its basic form, DDS keeps the reference image-text pair fixed while optimizing the target image to match a target prompt.The delta score is described as modifying edited regions while leaving other image regions unchanged.Source: §1 · p. 2
• DDS supports caption-only prompt-to-prompt editing without masks and zero-shot translation training without paired data.Training uses source-distribution images with simple captions describing source and target distributions, and can cover single or multiple tasks with synthetic or real images.Source: §1 · p. 2
• The authors report experiments comparing their approach with existing state-of-the-art text-driven editing techniques.Source: §1 · p. 2

2. Related Work

• Earlier editing approaches add noise and denoise from a chosen step, use DDIM inversion, or generate a mask to preserve backgrounds.The passage notes that noise-and-denoise methods struggle to preserve input details, while DiffEdit uses an automatically produced mask to avoid distortion.Source: §2 · p. 2
• Bar-Tal et al. [4] offer mask-free localized texture editing but not complex structural changes, while Prompt-to-prompt [12] manipulates local or global details in synthesized images.A separate inversion approach [23] applies prompt-to-prompt to real images.Source: §2 · p. 2
• Imagic [17] and UniTune [43] demonstrate text-driven editing but require costly model fine-tuning.InstructPix2Pix [5], plug-and-play [41], and [25] accept an instruction or target prompt to manipulate real images.Source: §2 · pp. 2–3
• Prior work applies SDS to 3D asset generation [27] and StyleGAN adaptation [38], while StyleGAN-NADA [10] uses CLIP guidance for text-only domain translation.This paper applies SDS to image editing and proposes cleaning its undesired gradients.Source: §2 · p. 3

3.1. SDS overview

• SDS uses gradients to update a generator toward greater prompt fidelity, but its tendency to converge to specific modes can produce blurry, non-diverse outputs.The paper uses Stable Diffusion for its experiments and describes the derivation as general to parametric image-rendering functions.Source: §3.1 · pp. 3–4, §3 · p. 3

3.2. Editing with SDS

• Initializing SDS editing from the source image can blur details unrelated to the target prompt, despite the goal of preserving source structure and details.The paper frames editing as incorporating the input image’s structure and details while conforming to the target prompt.Source: §3.2 · p. 4

3.3. Denoising the Editing Direction

• DDS subtracts the SDS score of a matched reference image-text pair from the target-prompt score to obtain a distilled editing direction.The reference branch estimates noisy directions, while the target branch supplies the requested edit.Source: §3.3 · p. 4
• DDS assumes related image-prompt pairs share similar noisy directions and that a matched pair should have near-zero average SDS gradients, although experiments find nonzero matched-pair scores.The authors attribute those nonzero gradients to the noisy component that the reference branch estimates.Source: §3.3 · p. 5
• Small classifier-free guidance values slow convergence and reduce text fidelity, while large values reduce fidelity to the input image.Figure 5 presents DDS results across guidance-scale settings.Source: §3.3 · p. 5, Figure 5 · p. 5

4. Image-to-Image Translation

• Direct pixel-space DDS optimization requires captions for the input and edit, performs worse on real images than synthetic ones, and takes about 20 seconds per edit.The authors introduce the translation-training pipeline to address these drawbacks of optimizing each edit separately.Source: §4 · p. 5
• The translator is trained without paired examples, using source images and captions for source and target distributions, and DDS to optimize the network.For multiple tasks, training samples a task embedding and corresponding target caption; an identity regularization term helps maintain input fidelity.Source: §4 · pp. 5–6
• Figure 7 contrasts the full method with training without CFG warmup and with SDS replacing DDS.The figure identifies the input and full-method output columns, the no-warmup row, and the SDS column.Source: Figure 7 · p. 6
• An identity-regularization weight decreases with a cosine schedule, injecting prior knowledge early in training and reducing it over time.The weight λID(t) is gradually reduced during training.Source: §4 · p. 6
• Gradually increasing classifier-free guidance during training addresses mode collapse, where outputs otherwise repeat a fixed lion pose and location regardless of the input cat.The authors report that high guidance pushes outputs aggressively toward the target, while gradual increases let the network learn larger changes without the reported local minima.Source: §4 · p. 6

SDS evaluation

• Matched image-text pairs retain non-negligible SDS gradients, and their gradient norms differ from those of unmatched pairs.The norm evaluation averages 200 measurements for each estimate across 100 matched and 100 caption-permuted pairs.Source: §5 · pp. 6–7

DDS evaluation

• SDS directions for matched similar image-text pairs are strongly correlated, supporting the use of a reference pair to estimate and remove a shared noise component.The evaluation compares average cosine similarity for matched and unmatched pairs from InstructPix2Pix.Source: §5 · p. 7

Comparison to zero-shot editing methods

• Direct DDS editing achieves higher average fidelity to both the text prompt and source image than SDEdit and Plug-and-Play; increasing CFG brings negligible text-fidelity gains alongside worse source fidelity.The comparison uses CLIP for text-image correspondence and LPIPS for similarity to the original image; Figure 9 reports results for different CFG values.Source: §5 · p. 7, Figure 9 · p. 7, Figure 10 · p. 7

Limitations

• DDS editing can also change unintended content: changing curtain color affects pillows, while replacing a dog reflection with a shadow alters lighting, weather, and background details.The paper attributes these imperfect edits to diffusion-model biases or limitations in language understanding.Source: Figure 12 · p. 8

Image-to-image translation comparison

• The DDS-trained translator achieves better fidelity to both the input and target domain and runs 50× faster than diffusion methods requiring iterative inference.Qualitative results also report better preservation of cat structure and background in animal translations.Source: Table 1 · p. 7, §5 · p. 8, Figure 11 · p. 8

Ablation Study

• Without CFG warmup, cat-to-lion training collapses to repeated lion outputs; replacing DDS with SDS worsens translation quality and preservation of high-frequency input detail.The SDS comparison retains the other components, including identity regularization and CFG warmup.Source: §5 · p. 8

Conclusions

• DDS uses the input image's SDS score to calculate cleaner gradients for text-prompt editing, and the authors demonstrate an unsupervised translator that needs no paired images.The model can be trained on real images.Source: §6 · p. 9

Limitations

• DDS edits remain imperfect when diffusion-model or language-model limitations misbind adjectives to nouns, causing changes beyond the intended object.Examples include changing pillows while editing curtain color and altering background details when replacing a dog's reflection with a shadow.Source: §6 · p. 9
• The authors say the multi-task model may be improved with multiple-expert training or semi-supervised training using some paired data.Source: §6 · p. 9

Future work

• The authors identify generalization across editing tasks as future work and suggest exploring DDS for 3D shape, video, and motion editing.Source: §6 · p. 9

Conclusions

• The paper frames extracting efficient, clean gradients from generative models as a step toward understanding how to use their learned knowledge.Source: §6 · p. 9

• The unsupervised image-editing framework could be exploited to produce fake content, a risk common to image-editing techniques.Source: A. Societal Impact · p. 11
• Large text-to-image diffusion models may contain undesired biases from their auto-filtered training data.Source: A. Societal Impact · p. 11
• Those biases could enter editing results or distilled networks through optimization, while DDS may help reveal them and undesirable editing directions.Source: A. Societal Impact · p. 11

• Zero-shot DDS optimization starts from the reference latent and runs 200 iterations, taking about 18 seconds on one A100 GPU.Source: B. Implementation Details · p. 11
• Zero-shot optimization uses SGD at learning rate 2, decayed by 0.9 every 20 iterations.Source: B. Implementation Details · p. 11
• Image-to-image networks train for 125,000 iterations with batch size 2 and Adam at 10^-5; CFG and identity regularization use cosine warmups or schedules.CFG warms from 1 to 25 over 20,000 iterations; λid cools from 3 to 0.1 over 20,000 iterations.Source: B. Implementation Details · p. 11

Number of optimization steps

• Most desired edits converge within 200 steps, but large structural or color changes may need more; replacing a flamingo with an elephant takes 400 steps.A flamingo-to-giraffe edit remains an example where 200 steps suffice because the overall structures are similar.Source: C. Additional Ablations · p. 11

Optimizer Selection

• Both DDS optimizers outperform SDS in quality, but vanilla SGD beats Adam, whose updates introduce more prompt-irrelevant changes and artifacts.Source: C. Additional Ablations · p. 11
• Adam’s adaptive normalization may magnify outlier gradients or downweight good ones, yielding more uniform pixel updates than SGD’s edit-focused updates.Source: C. Additional Ablations · p. 11

Regularized SDS optimization

• Increasing λid in regularized SDS harms prompt fidelity, while the SDS blurriness remains.Source: C. Additional Ablations · p. 11

Zero-shot image editing results on real images

• On real COCO and Unsplash images, DDS applies simple edit prompts; a reference text is used for edits to a specific object, otherwise the null-text embedding is used.Source: D. Additional Results · p. 11

Image-to-image networks results

• Synthetic-data networks translate sofa material, add flowers to potted plants, and change people into characters; the person networks train on FFHQ in-the-wild images.Source: D. Additional Results · p. 12

Zero-shot image editing results on real images

• In DDS Optimization, 200 steps are sufficient in most cases, though some challenging edits—such as changing a flamingo to an elephant—may take longer to converge than other edits.Source: Figure 13 · p. 13
• Real-image DDS examples include global lighting changes and local object substitutions, as well as structural and stylistic edits.The figure captions illustrate house-to-igloo, bicycle-to-Vespa, and coffee-to-matcha edits.Source: Figure 16 · p. 16, D. Additional Results · pp. 16–17, Figure 17 · p. 17

Image-to-image networks results

• The sofa and flower networks train on synthetic images and are evaluated on real images, including ILSVRC [33] and COCO examples.The figure captions specify synthetic living-room training for sofas and real living-room tests, and synthetic training with real flowerpot and potted-plant tests.Source: D. Additional Results · p. 12, Figure 18 · p. 18, Figure 19 · p. 19