Weeding Out Bad Seeds: Initial-Noise-Robust Unlearning for Text-to-Image Diffusion Models

1 INTRODUCTION

• Existing optimization-based methods suppress targets while regularizing general performance, but generally estimate gradients by uniformly sampling standard Gaussian noise.The paper identifies this shared sampling convention as a blind spot for robustness to random initial noise.Source: §1 · pp. 1–2
• With identical prompts, forbidden concepts can re-emerge under varied noise even when fixed benchmark seeds show none, reaching up to 55% of generations across evaluated methods.The paper calls apparent erasure on limited initializations despite persistence elsewhere in the noise space “probabilistic forgetting.”Source: §1 · p. 2
• Under limited update budgets, uniform Gaussian sampling wastes updates on irrelevant regions, leaving target concepts intact in unexplored noise regions.Simply increasing update steps risks degrading global image quality because unlearning gradients can perturb non-target knowledge.Source: §1 · pp. 2–3
• Adaptive, concept-conditioned sampling prioritizes noise vectors likely to trigger forbidden content, framing unlearning as risk minimization over a tilted noise distribution.The reweighting scheme is presented as a tractable importance-sampling estimator that can integrate with optimization-based unlearning frameworks.Source: §1 · pp. 2–3
• Across four main SD 1.4 baselines, the method reduces failure count by 61.4% while maintaining competitive general image quality and improving the unlearning–retention trade-off.General image quality is indicated by CLIP and FID scores.Source: §1 · p. 3
• The experiments span six unlearning methods and four diffusion backbones, covering nudity, artistic-style, and object removal plus black- and white-box adversarial attacks.The reported outcomes include stronger robustness, competitive generative utility, and a more favorable balance between target erasure and non-target retention.Source: §1 · p. 3

2.1 THE ROLE OF INITIAL NOISE IN TEXT-TO-IMAGE GENERATION

• Noise exploration finds semantic alignment and compositional success concentrated in sparse regions, while noise optimization refines initial latents at inference time; unlearning generally ignores this structure.The cited exploration work includes Karthik et al. (2023), Li et al. (2024), Samuel et al. (2024), Mao et al. (2024), and Liu et al. (2024); optimization work includes Guo et al. (2024), Eyring et al. (2024), and Kasaei et al. (2025).Source: §2.1 · p. 3

2.2 MACHINE UNLEARNING

• Unlearning methods range from optimization-based ESD Gandikota et al. (2023) and FMN Zhang et al. (2024a) to closed-form UCE Gandikota et al. (2024) and training-free SAFREE Yoon et al. (2025), yet optimization methods generally sample noise uniformly.The passage also describes attention, saliency, parameter-efficiency, and preservation approaches.Source: §2.2 · pp. 3–4

2.3 VULNERABILITIES OF UNLEARNED DIFFUSION MODELS

• Unlike prompt-, latent-, and trajectory-based attacks, this work tests re-emergence under naturally sampled initial noise without optimizing prompts or noise or intervening in denoising.The stated scenario is that a prompt appearing safe in an initial test may reproduce the erased concept under another standard random initialization.Source: §2.3 · p. 4

• Standard unlearning can leave residual high-activation noise regions, while adaptive sampling focuses updates on those concept-related regions.Source: Figure 3 · p. 4
• A diffusion model starts from Gaussian noise and iteratively denoises it into an image conditioned on a text prompt.Source: §3 · p. 4
• Unlearning targets forbidden concepts using prompts and a synthetic forget set generated from the model’s own prior knowledge.Source: §3 · p. 4
• Because the current model generates the forget set, its distribution changes with parameters, and existing methods use a greedy approximation to avoid backpropagating through the denoising chain.At each update, generated samples are treated as constants.Source: §3 · p. 5

4 METHOD: ADAPTIVE NOISE SAMPLING

• Uniform Gaussian sampling wastes limited unlearning updates on safe regions because forbidden concepts are localized in the noise space.The resulting gradients are sparse, leaving the optimization budget less focused on noise regions that trigger the target.Source: §4 · p. 5

4.1 CONCEPT-CONDITIONED NOISE DISTRIBUTION

• The concept-conditioned distribution tilts the Gaussian prior toward initial noise vectors likely to trigger the forbidden concept and down-weights safe regions.A frozen classifier scores generated images for the target concept; the formulation accepts any classifier returning a confidence score in [0, 1].Source: §4.1 · pp. 5–6
• Importance sampling rewrites the expectation under the evolving tilted distribution as an expectation under the fixed standard Gaussian prior.It targets initialization noise xT; the diffusion noise ϵ remains standard Gaussian, preserving standard local denoising dynamics.Source: §4.1 · p. 6
• The importance weight amplifies loss for initializations that strongly activate the target while suppressing already-safe vectors.The weights adapt as unlearning progresses, and the normalizer is estimated from the same batch of candidate noise vectors.Source: §4.1 · p. 6

4.2 OPTIMIZATION STABILITY AND REGULARIZATION

• Regularization with γ > 0 prevents numerical instability as both concept activation and its normalizer approach zero.When the normalizer is much smaller than γ, weights scale approximately as A_i/γ and decrease with residual activation.Source: §4.2 · p. 6
• Stop-gradient weights scale per-sample updates, while Top-M truncation backpropagates only through the highest-weight candidates.The default scheme samples N = 20 candidates and retains the Top-M = 4, avoiding backpropagation through negligible weights.Source: §4.2 · p. 7

5.1 CONCEPT UNLEARNING: NUDITY

• Nudity re-emergence falls from 4%–25% with four optimization baselines to 0.5%–10% with Adaptive Noise Sampling across alternative seeds.The evaluation tests 200 prompts under 20 random initializations after excluding prompts that fail the fixed-seed evaluation.Source: §5.1 · p. 8
• Multi-seed failures decrease by 44% for EAP, 53% for RECELER, 61% for ESD, and 88% for ACE.For ESD, failures drop from 510 to 201 among 4,000 images; for ACE, they drop from 491 to 60.Source: §5.1 · p. 8
• Adaptive Noise Sampling consistently reduces attack success rates under black-box Ring-A-Bell and white-box UnlearnDiffAtk evaluations.Human assessments on 2,000 images per method closely align with the automated concept-removal results.Source: §5.1 · p. 9
• Qualitative examples show residual nudity removed under identical prompts and seeds while visual structure is preserved.The experiments use Stable Diffusion v1.4 as the main model, with additional evaluations on Stable Diffusion 2.1, Stable Diffusion 3, and FLUX.1 [dev].Source: §5 · p. 7, §5.1 · p. 9

5.2 STYLE UNLEARNING

• Adaptive Noise Sampling improves or preserves mean Unlearning Accuracy, In-domain Retain Accuracy, and Cross-domain Retain Accuracy across ACE, EAP, ESD, and RECELER.The largest reported unlearning-accuracy gain is 4.10 points for EAP; retention metrics improve by up to 2.87 points.Source: §5.2 · p. 9, Table 2 · p. 9

5.3 OBJECT UNLEARNING

• The object evaluation averages erased-class and other-class ResNet-50 Top-1 accuracy across ten Imagenette target classes.Lower erased-class accuracy and higher accuracy on the other nine classes are the respective objectives; the supplied passages do not report comparative outcomes.Source: §5.3 · p. 9, Table 3 · p. 9

6.1 CONCLUSION

• Adaptive Noise Sampling reduces concept re-emergence across all four iterative Stable Diffusion 1.4 nudity baselines and lowers failure counts on three additional backbones.Style experiments improve or preserve reported removal and retention metrics; object experiments primarily improve non-target retention, with method-dependent changes in erased-class accuracy.Source: §6.1 · p. 10

6.2 DISCUSSION

• Results across safety, object, and artistic-style unlearning generalize across concept domains and guidance-model families, though improvement magnitude varies by guidance model.The discussion reports evaluations with five guidance architectures, independent evaluation classifiers, human nudity judgments, and corrupted guidance signals.Source: §6.2 · p. 10
• Finite-sample tests show consistent reductions in probabilistic forgetting, not certified worst-case concept removal.Rare concept-triggering initializations may remain unobserved despite the 20-seed protocol and concentration analysis.Source: §6.2 · p. 10

B EFFICIENT APPROXIMATION: THE TOP-M STRATEGY

• Top-M approximation explores N candidate noises but backpropagates only through the M highest-weight samples, reducing gradient-update steps to reported checkpoints.The approximation assumes discarded samples contribute negligibly to the weighted gradients and separates forward-pass exploration from backward-pass optimization.Source: B EFFICIENT APPROXIMATION: THE TOP-M… · p. 16

C HANDLING THE NORMALIZING CONSTANT

• The empirical batch mean approximates the intractable normalizer, and bounded classifier confidence supports stable estimation even with N = 20.For N = 20, the stated worst-case standard error is approximately 0.11; observed estimator variance is roughly 0.0067.Source: C HANDLING THE NORMALIZING CONSTANT · pp. 16–17

2. Preservation of Gradient Direction: Even if the empirical mean ˆZ(N)

• When normalizer estimation is imperfect, its error scales the batch gradient without changing its direction, while adaptive optimizers absorb uniform scalar fluctuations.The expected gradient energy also decreases as the target activation landscape flattens during unlearning, tightening the approximation over time.Source: §2 · p. 18

D DERIVATION OF THE IMPORTANCE-SAMPLED OBJECTIVE

• Importance sampling rewrites the tilted-noise expectation using standard Gaussian samples, yielding a tractable forget-loss objective.The derivation substitutes the simplified importance weight into the expectation over conditions, timesteps, and diffusion noise.Source: D DERIVATION OF THE IMPORTANCE-SAMPLED… · p. 18

E EMPIRICAL ANALYSIS OF THE NORMALIZER ESTIMATE

• During Stable Diffusion 1.4 nudity unlearning, the batch estimate of the normalizer trends downward across optimization steps.Figure 6 averages NudeNet confidence over 20 independently sampled initial-noise vectors; it shows the batch estimate, not the intractable population normalizer.Source: E EMPIRICAL ANALYSIS OF THE NORMALIZER… · p. 18, Figure 6 · p. 19

STYLE-UNLEARNING RESULTS

• Across styles, UA never decreases and IRA and CRA increase in every comparison; EAP gains about 4.1 UA points per target, while ACE and RECELER improve retention despite already having 100% UA.ESD improves all three metrics across every target style; mean gains are 0.60 UA, 2.51 IRA, and 2.75 CRA points.Source: STYLE-UNLEARNING RESULTS · p. 19

G OBJECT UNLEARNING

• On Imagenette, Adaptive Noise Sampling improves non-target preservation while maintaining robust erasure for RECELER, with method-dependent erasure changes for other methods.The supplied results note RECELER's baseline non-target accuracy of 72.5%, but do not provide the augmented comparison in these passages.Source: G OBJECT UNLEARNING · pp. 19–20

H GENERALIZATION TO ALTERNATIVE CLASSIFIER ARCHITECTURES FOR OBJECT UNLEARNING

• With EfficientNet guidance, retention improves for all four methods; with MobileNetV2 guidance, target-class error rises for all four, while retention varies by method.EfficientNet guidance improves target erasure for ESD, leaves ACE unchanged, and slightly weakens RECELER and EAP; alternative classifiers can be incorporated without changing the framework.Source: H GENERALIZATION TO ALTERNATIVE… · p. 21, Table 6 · p. 21, Table 7 · p. 22

I.1 CLIP CLASSIFIER ABLATION

• CLIP guidance retains Adaptive Noise Sampling’s improvement over uniform-sampling baselines across all four methods; EAP records 644 failures with CLIP versus 638 with NudeNet.The CLIP-guided result maintains comparable general image quality, supporting classifier substitution in the nudity setting.Source: §I.1 · p. 22, Table 8 · p. 22

I.2 CLASSIFIER ROBUSTNESS TO IMAGE DISTORTIONS

• NudeNet confidence remains 0.68 at σ = 0.05 versus 0.77 when clean, but falls to 0.51 and below at σ ≥ 0.10.This isolates the guidance classifier’s stability under image corruption.Source: §I.2 · p. 22, Table 9 · p. 23

I.3 END-TO-END ROBUSTNESS TO CORRUPTED GUIDANCE

• Even with corrupted NudeNet guidance, Adaptive Noise Sampling beats standard baselines: ESD has 308 failures at σ = 0.15 versus 510 uncorrupted, and ACE has 148 at σ = 0.10 versus 491.Evaluation uses independent Open-NSFW2; EAP and RECELER also outperform their uncorrupted baselines under the reported noisy guidance settings.Source: §I.3 · p. 23, Table 10 · p. 24

J COMPUTATIONAL OVERHEAD AND OPTIMIZATION EFFICIENCY

• ACE and EAP use fewer gradient-update steps with Adaptive Noise Sampling, but candidate scanning increases total wall-clock runtime; the paper characterizes this as step efficiency, not acceleration.The authors describe the additional runtime as hours rather than days for a one-time model edit.Source: J COMPUTATIONAL OVERHEAD AND… · p. 23, Table 11 · p. 24

J.1 HARDWARE AND PEAK MEMORY REQUIREMENTS

• The detached candidate scan limits backward-pass memory to the selected update batch, keeping peak VRAM comparable to the corresponding uniform-sampling baseline.The reported configuration uses N = 20 candidates and M = 4 selected samples on one 40 GB NVIDIA A100 GPU.Source: §J.1 · pp. 23–24

Impact of N and M

• N = 20, M = 4 achieves 60 failures on ACE, near the 52 failures at N = 40, M = 8, while matching a standard batch size of 4 for backward-pass memory.The study describes diminishing returns as exploration and update batch sizes increase computational costs.Source: K ABLATION STUDY: HYPERPARAMETER… · pp. 24–25, Table 13 · p. 25
• Even N = 4, M = 1 outperforms uniform-sampling baselines across methods; on ACE, failures fall from 491 to 119.At N = 20, M = 4, EAP failures fall from 1142 to 638 and RECELER failures from 693 to 326.Source: K ABLATION STUDY: HYPERPARAMETER… · p. 25

Sensitivity to the Regularization Constant (γ)

• Performance remains consistent across tested γ values from 10^-5 to 10^-3, and the experiments use γ = 10^-4.The configuration comparison reports total wall-clock training time separately from the hyperparameter results.Source: K ABLATION STUDY: HYPERPARAMETER… · p. 25, Table 12 · p. 25

L SCALING TO LARGER ARCHITECTURES: STABLE DIFFUSION 2.1

• On Stable Diffusion 2.1, 20-seed failures fall from 1413 to 985 for EAP and from 673 to 250 for ESD; CLIP scores rise from 0.233 to 0.236 and 0.221 to 0.223, respectively.The study uses NudeNet for guidance and Open-NSFW2 for evaluation.Source: L SCALING TO LARGER ARCHITECTURES:… · p. 25, Table 15 · p. 26

M PROOF OF CONCEPT ON A DIT ARCHITECTURE: STABLE DIFFUSION 3

• On Stable Diffusion 3, DUO’s single-seed failures fall from 60 to 46 and 20-seed failures from 1245 to 938; CLIP rises from 0.2385 to 0.2399.This is a proof-of-concept transfer to a Diffusion Transformer architecture without modifying the DUO objective.Source: M PROOF OF CONCEPT ON A DIT… · p. 26, Table 16 · p. 27

N UNLEARNING ON FLUX.1 [DEV]

• On FLUX.1 [dev], 20-seed failures fall from 1002 to 703 and mean nudity score from 0.257866 to 0.214151; CLIP rises to 0.227629 but remains below the original model’s 0.235414.The reported values are point estimates without run-to-run standard deviations, and residual failures remain.Source: N UNLEARNING ON FLUX.1 [DEV] · p. 27, Table 17 · p. 27

O ADVERSARIAL EVALUATION

• The paired attack comparison evaluates standard and adaptive checkpoints under black-box Ring-A-Bell and white-box UDA, reporting attack success rate as the normalized robustness measure.The setup holds each underlying unlearning objective fixed when comparing its checkpoints.Source: O ADVERSARIAL EVALUATION · p. 27, Table 18 · p. 28
• Under Ring-A-Bell, ASR falls for ACE from 0.20% to 0.00%, RECELER from 1.38% to 0.43%, ESD from 45.68% to 35.55%, and EAP from 53.13% to 32.38%.The corresponding UDA ASRs also fall: ACE 2.81% to 2.42%, RECELER 19.14% to 11.88%, ESD 76.35% to 23.28%, and EAP 21.74% to 15.58%.Source: O ADVERSARIAL EVALUATION · p. 27, Table 18 · p. 28

P ADDITIONAL NUDENET TRAINING AND EVALUATION RESULTS

• The additional NudeNet analysis evaluates adaptive checkpoints with NudeNet itself, unlike the main experiment’s independent Open-NSFW2 evaluation.It reports failure counts for one predefined seed and 20 randomly sampled noise seeds per prompt.Source: P ADDITIONAL NUDENET TRAINING AND… · p. 28, Table 19 · p. 28

Q HUMAN EVALUATION.

• Five annotators independently labeled images from 100 nudity prompts, with 20 random seeds per prompt yielding 2,000 images per method.Annotators were informed images might contain nudity and consented before annotation.Source: Q HUMAN EVALUATION · p. 29, Table 20 · p. 29
• The method substantially reduced human-labeled nudity failures versus all baselines, with the lowest failure rates when applied to ACE and ESD.Table 20 reports nude-image counts and failure ratios from binary human annotations.Source: Q HUMAN EVALUATION · p. 29, Table 20 · p. 29
• The evaluation used binary nude/non-nude labels and reported both failure counts and failure ratios, with lower values indicating fewer failures.Source: Table 20 · p. 29

R ADDITIONAL QUALITATIVE RESULTS

• The qualitative results cover both fixed prompt–seed conditions and comparisons across unlearning methods and noise initializations.Source: R ADDITIONAL QUALITATIVE RESULTS · p. 29
• With prompt and seed held constant, outputs remained visually consistent with baseline generations while removing the target concept and preserving scene structure and semantic content.The comparison is presented as targeted unlearning without unnecessary deviations or artifacts.Source: R ADDITIONAL QUALITATIVE RESULTS · p. 29, Figure 7 · p. 30
• Across methods and stochastic noise samples, the approach more reliably suppressed the target concept while maintaining visual plausibility and prompt faithfulness.Baseline examples sometimes retained the concept or showed noticeable image-quality degradation.Source: R ADDITIONAL QUALITATIVE RESULTS · p. 29, Figure 8 · p. 30