Source-linked AI summary
Mitigating Sequential Reappearance in Diffusion Data-Point Unlearning
Donghyun Kim, Taehyuk Lee, Jinyeong Kim, Youngmin Oh, Dohyeong Kim, Jaehyuk Ryu, Sangwoo Hong
TL;DR
Diffusion data-point unlearning is commonly assessed immediately after deletion, leaving target behavior across later model updates underexamined. The paper introduces target-level sequential evaluation and LASTING, and reports greater recovery accessibility for targets that reappear alongside improved forgetting persistence with competitive generation quality.
Problem
Prior evaluations predominantly assess targets immediately after deletion, leaving their behavior under future model updates unmeasured.
Source: §1 · p. 1Method
The authors track each target’s immediate and persistent forgetting and reappearance, and propose LASTING to adaptively identify and suppress locally recoverable representations during unlearning.
Source: §1 · p. 2Results
Targets that later reappear have greater local recovery accessibility, while LASTING improves forgetting persistence with competitive generation quality.
Source: §1 · pp. 1–2Takeaways & Limitations
The findings highlight the need to evaluate diffusion unlearning beyond immediate deletion success and account for forgetting stability under subsequent requests.
Source: §6 · p. 9
Abstract
from arXiv · showhide
Diffusion data-point unlearning is typically evaluated immediately after each deletion, even though subsequent requests may repeatedly update the same model. We identify sequential reappearance, a failure mode in which an instance that is initially judged to be forgotten later returns to the memorized regime without reuse of the deleted data or adversarial fine-tuning. To capture this behavior, we introduce a target-level evaluation protocol that tracks whether each target is forgotten immediately, remains forgotten at the end of the sequence, or reappears during subsequent deletions. We further find that targets that later reappear exhibit sharper local denoising-loss geometry after deletion than targets that remain forgotten.
1 INTRODUCTION
Stage-local evaluations can overestimate deletion reliability because targets initially judged forgotten may reappear during later ordinary deletions. The paper introduces a target-level persistence protocol, links reappearance to post-deletion recovery accessibility, and proposes LASTING to improve forgetting persistence.
1 INTRODUCTION
- Targets that later reappear have greater post-deletion recovery accessibility in local denoising-loss geometry than persistently forgotten targets, despite both meeting the immediate forgetting criterion.This motivates suppressing nearby representations that preserve a route back to memorization.Source: §1 · p. 1
- LASTING adaptively targets the most locally recoverable representation, improving forgetting persistence across long sequential deletion trajectories while maintaining competitive generation quality.Figure 1 shows LASTING maintaining forgetting after the full CelebA-HQ deletion sequence.Source: Figure 1 · p. 2, §1 · p. 2
- The target-level protocol distinguishes immediate forgetting, end-of-sequence persistence, and intervening reappearance, which can occur during ordinary deletions without target reuse, adversarial fine-tuning, or an explicit recovery attack.Source: §1 · pp. 1–2
2 RELATED WORK
Prior diffusion unlearning methods focus primarily on immediate deletion, while this work asks whether deleted instances remain forgotten through subsequent updates and examines their post-deletion denoising-loss geometry.
2.1 MEMORIZATION IN DIFFUSION MODELS
- Diffusion models can reproduce training examples, and extraction attacks have recovered memorized images at scale; this work studies model editing rather than detection or inference-time mitigation.Extraction attacks are reported by Somepalli et al. (2023a) and Carlini et al. (2023); prior work also studies duplication, data coverage, and text conditioning.Source: §2.1 · p. 2
2.2 DIFFUSION MODEL UNLEARNING
- Concept-level methods suppress broad semantic content, whereas data-point methods target individual examples but primarily assess forgetting immediately after deletion.The paper instead examines whether successfully deleted instances remain forgotten through subsequent deletion updates.Source: §2.2 · pp. 2–3
2.3 RECOVERY AND SEQUENTIAL UNLEARNING
- Unlike recovery studies involving explicit interventions, this work examines spontaneous reappearance during ordinary sequential deletion without fine-tuning, recovery attacks, or reuse of removed instances.Sequential diffusion work has focused mainly on semantic concepts and utility; sequential data-point unlearning remains largely unexplored despite its relevance to accumulating copyright and privacy requests.Source: §2.3 · p. 3
2.4 LOCAL GEOMETRY IN MACHINE UNLEARNING
- Prior classifier studies examine residual knowledge near perturbed forget samples or reshape local decision boundaries; this work studies diffusion denoising-loss geometry and its association with later reappearance.It identifies nearby image representations that remain easy to denoise after deletion.Source: §2.4 · p. 3
3 ANALYSIS OF SEQUENTIAL REAPPEARANCE
The analysis distinguishes immediate forgetting from persistence through later deletion requests, showing that immediately forgotten targets can reappear. Higher recovery accessibility after deletion is associated with subsequent reappearance.
3.1 PROTOCOL AND METRICS
- The protocol separates each target’s immediate model state from its final state, uses SSCD < 0.7 for forgetting, and defines distinct Final Success and Final Reappearance fractions.Final Reappearance is the fraction of immediately forgotten targets that return to the memorized regime under the final model.Source: §3.1 · p. 3
3.2 REAPPEARANCE UNDER SEQUENTIAL DELETION
- Across 300 deletion requests, previously forgotten targets can return to the memorized regime as later updates show that immediate forgetting does not ensure persistence.The protocol was applied using SISS, EraseDiff, ReTrack, Prompt-Free, and FU.Source: §3.2 · p. 4
- Recovery accessibility measures the relative denoising-loss reduction available within a bounded data-space neighborhood, estimated by constrained projected-gradient search after freezing the immediate model.The analysis uses a fixed bank of timesteps and noise samples for each target.Source: §3.2 · p. 4
- Among immediately forgotten targets tracked for 30 deletion stages, later reappearing targets show higher recovery accessibility, with positive correlations and meaningful AUROC across methods.The positive association remains after controlling for Immediate SSCD.Source: §3.2 · p. 4, Figure 2 · p. 4, Table 1 · p. 4
4 PROPOSED METHOD
LASTING searches for a nearby representation with low denoising loss, then uses that neighbor in the deletion objective to target the model’s strongest remaining recovery route. It preserves the base unlearning method’s other components and optimization rule.
4 PROPOSED METHOD
- At each optimization step, LASTING selects a low-loss neighbor of the deletion target and uses it to guide the base unlearning objective toward the strongest remaining recovery route.The outer objective uses the selected neighbor in place of the original target while preserving the base method’s remaining components and update rule.Source: §4 · p. 5, §4.2 · p. 5
4.1 ADAPTIVE LOW-LOSS NEIGHBOR SEARCH
- A low diffusion loss signals that a representation remains easy to denoise, so LASTING searches within a feasible target neighborhood using projected sign-gradient steps.The search constrains perturbations to a radius and valid image range, and reuses the same sampled diffusion state for the search and deletion update.Source: §4.1 · p. 5
4.2 RECOVERY-GUIDED UNLEARNING
- LASTING changes only the representation used by the deletion objective, leaving the retain computation and original optimization procedure unchanged.The main implementation instantiates LASTING on SISS (Alberti et al., 2025), retaining its construction, weighting, gradient handling, and optimizer.Source: §4.2 · pp. 5–6
- The selected neighbor is detached before the outer update, avoiding differentiation through the search and the need for second-order derivatives.After each sequence of Q optimization iterations, the resulting model becomes the starting point for the next deletion request.Source: §4.2 · p. 6
5 EXPERIMENTS
Experiments evaluate LASTING’s forgetting persistence across sequential deletions on CelebA-HQ and Stable Diffusion, and analyze how neighborhood construction and unlearning objectives affect reappearance. LASTING shows lower reappearance than key baselines in both settings and reduces recovery accessibility.
5.1 SEQUENTIAL UNLEARNING ON CELEBA-HQ
- At K = 300 on CelebA-HQ, LASTING achieves 96.78% Final Success and 3.12% Final Reappearance, versus SISS’s 91.67% and 7.93%.EraseDiff and ReTrack have Final Reappearance rates of 53.89% and 49.84%; LASTING’s Final SSCD is 0.4034 versus SISS’s 0.4243, with FID 23.14.Source: §5.1 · p. 6, Table 2 · p. 7
- At K = 200, LASTING’s advantage in forgetting persistence also holds across three different CelebA-HQ deletion-target configurations.The comparison uses targets starting at indices 10,000, 15,000, and 20,000.Source: §5.1 · p. 7, Table 3 · p. 8
5.2 EXTENSION TO TEXT-CONDITIONED LATENT DIFFUSION
- At K = 30 on Stable Diffusion, LASTING achieves 88.33% Final Success and 5.88% reappearance, versus SISS’s 66.67%/22.45%, EraseDiff’s 61.67%/31.25%, and ReTrack’s 93.33%/6.90%.LASTING also maintains competitive generation quality; shown qualitative examples suppress memorized content without obvious visual degradation.Source: §5.2 · pp. 7–8, Table 4 · p. 8
5.3 ABLATION AND ANALYSIS
- LASTING reduces mean recovery accessibility from 0.0501 to 0.0309, a 38.35% reduction, reaching 77.72% and 82.83% reductions among the top 10% and 5% most vulnerable targets.This reduction indicates suppression of the strongest remaining recovery geometry.Source: §5.3 · p. 8
- With Forward KL-Constrained Unlearning, LASTING reduces Final Reappearance from 5.50% to 2.50% for SISS and from 34.17% to 4.50% for FU.The improvement under both evaluated objectives supports compatibility beyond the specific SISS objective.Source: §5.3 · pp. 8–9
- LASTING achieves 2.50% Final Reappearance and 97.50% Final Success, versus 5.50% reappearance for SISS, 23.74% for Random Neighbor, and 37.20% for High-Loss Neighbor.The results support low-loss, locally recoverable neighborhood search rather than generic perturbation or high-loss search.Source: §5.3 · p. 8, Table 5 · p. 9
6 CONCLUSION
The paper identifies sequential reappearance in diffusion unlearning and proposes LASTING to improve the persistence of forgetting. Its findings motivate evaluating whether forgetting remains stable across subsequent unlearning requests, not only immediately after deletion.
6 CONCLUSION
- Across diffusion settings, LASTING substantially improves forgetting persistence while maintaining competitive generation quality.LASTING adaptively identifies and suppresses locally recoverable representations during unlearning.Source: §6 · p. 9
- Sequential reappearance occurs when a diffusion training instance forgotten immediately after deletion becomes recoverable after subsequent unlearning updates.The analysis associates reappearance with greater local recovery accessibility around the deleted target.Source: §6 · p. 9
- The findings highlight the need to evaluate forgetting stability under subsequent requests, beyond immediate deletion success.Source: §6 · p. 9
SUPPLEMENTARY MATERIAL CONTENTS … A.2 COMPARISON AT MATCHED COMPUTATION
With increased optimization, SISS achieves the lowest Immediate SSCD at every reported checkpoint but does not match LASTING’s persistence. LASTING’s inner search takes 960.43 seconds per target, while its peak GPU memory is the lowest among the evaluated methods.
A.1 RESULTS ACROSS TRAINING SEEDS
- The CelebA-HQ seed experiments use seeds 0, 42, and 10000 at K ∈ {50, 100, 200, 300}, with targets, order, protocol, and optimization settings fixed across runs.Table 8 reports mean ± sample standard deviation over the three training seeds.Source: §A.1 · p. 14, Table 8 · p. 15
A.2 COMPARISON AT MATCHED COMPUTATION
- LASTING takes 960.43 seconds per target because of its inner search and uses 13.63 GiB peak GPU memory, the lowest among evaluated methods.The comparison uses each method’s reported CelebA-HQ configuration and update budget on one NVIDIA RTX PRO 6000 Blackwell Server Edition GPU.Source: §A.2 · p. 14, Table 9 · p. 15
- The increased-budget SISS run raises training steps per deletion target to approximately match LASTING’s measured per-target computation time.The comparison includes reference SISS, increased-budget SISS, and LASTING along the deletion sequence.Source: §A.2 · p. 14
- At K = 200/300, increased-budget SISS has 29.50%/31.00% Final Reappearance versus reference SISS’s 5.50%/9.73%, while LASTING reaches 2.50%/2.33%.These results show that additional SISS optimization does not reproduce the persistence achieved by LASTING.Source: §A.2 · p. 14
- At K = 300, LASTING has 97.67% Final Success and 0.3917 Final SSCD, versus 69.00% and 0.5377 for increased-budget SISS.Source: §A.2 · p. 14
- Increased-budget SISS obtains the lowest Immediate SSCD at every reported checkpoint, strengthening immediate forgetting without matching LASTING’s persistence.Source: §A.2 · p. 14
A.3 DETAILED ANALYSIS OF REAPPEARANCE … B.2 NEIGHBORHOOD RADIUS
Across deletion trajectories, LASTING reduces target reappearance after immediate forgetting; sensitivity analyses identify how inner-search steps and neighborhood radius affect persistence. The evaluation distinguishes targets that remain forgotten, transiently reappear, or end in a reappeared state.
A.3 DETAILED ANALYSIS OF REAPPEARANCE
- Trajectory evaluation conditions on targets immediately forgotten at SSCD threshold τ = 0.7 and classifies their subsequent status as no, transient, final, or ever reappearance.Ever Reappearance combines transient and final reappearance; final reappearance means the target is at or above the threshold under the final model.Source: §A.3 · p. 15
- Across K = 50–300, LASTING has the lowest Ever Reappearance; at K = 300 it records 7.00% (21 targets: 14 transient, 7 final) versus SISS’s 23.49%.At K = 50, none of LASTING’s immediately forgotten targets reappear; at K = 100 and 200, its rates are 4.00% and 4.50%, versus SISS’s 9.00% and 13.50%. At K = 300 it has a 93.00% No Reappearance rate, while other baselines range from 65.02% to 80.81% Ever Reappearance.Source: §A.3 · p. 16, Table 11 · p. 17
- The reappearance metrics focus on losing an already achieved forgetting state, distinct from immediate forgetting failures that later deletion updates may correct.Source: §A.3 · p. 16
B.1 NUMBER OF INNER-SEARCH STEPS
- The inner-search ablation tests J ∈ {1, 4, 8, 12} with fixed diffusion timestep and noise reused in the subsequent outer update.Source: §B.1 · p. 16
- At J = 12, further inner-search steps provide no benefit and increase Final SSCD and endpoint reappearance; therefore, J = 8 is used for CelebA-HQ experiments.The comparison varies J ∈ {1, 4, 8, 12} on the same K = 300 deletion sequence, with ϵ_nbr = 4/255.Source: §B.1 · pp. 16, 18
B.2 NEIGHBORHOOD RADIUS
- At ϵ_nbr = 8/255, Final Reappearance is absent and Final SSCD is 0.3495; radii 12/255 and 16/255 raise Final SSCD to 0.5300 and 0.7030 and produce more final reappearance.Although 8/255 performs best in this K = 100, J = 4 ablation, 4/255 is the conservative default for longer main experiments.Source: §B.2 · p. 18, Table 13 · p. 18
B.3 EVALUATION TIMESTEP … LASTING SISS EraseDiff
The appendix reports that LASTING’s forgetting-persistence advantage remains consistent across the tested evaluation timesteps and recovery-noise seeds, and supplies implementation protocols and additional qualitative examples for CelebA-HQ and Stable Diffusion.
B.3 EVALUATION TIMESTEP
- LASTING maintains its forgetting-persistence advantage across evaluation timesteps and recovery-noise seeds, with small standard deviations indicating stable performance.The sensitivity evaluation uses K = 300 and compares t_eval ∈ {250, 350}; Table 14 reports means and sample standard deviations over three seeds.Source: §B.3 · pp. 18–19, Table 14 · p. 19
C IMPLEMENTATION DETAILS
- Implementation details are provided for both the CelebA-HQ and Stable Diffusion experiments.These details accompany the respective experiment protocols and configurations.Source: C IMPLEMENTATION DETAILS · p. 19
C.1 CELEBA-HQ
- CelebA-HQ experiments sequentially delete 300 targets from a fixed 29,700-image retain pool, with each updated model initializing the next deletion stage.LASTING leaves SISS’s retain construction and outer deletion objective unchanged, and uses projected sign-gradient descent to search within an ℓ∞ neighborhood for a low-loss neighbor.Source: §C.1 · pp. 19–20
C.2 STABLE DIFFUSION
- Stable Diffusion experiments sequentially process 30 memorized prompt–image targets and evaluate forgetting separately for each target’s original and modified prompts.A target–prompt case counts as forgotten only if all 16 generations are classified as non-memorized; final success and reappearance are evaluated at the sequence’s final checkpoint.Source: §C.2 · p. 21
D.1 CELEBA-HQ
- Six additional CelebA-HQ targets illustrate how reconstruction resemblance can change between immediate and final checkpoints.The examples provide target-level context for the aggregate persistence results.Source: §D.1 · p. 23, Figure 4 · p. 23
D.2 STABLE DIFFUSION
- Four additional Stable Diffusion prompts show immediate and final generation composites for visual inspection of whether reference-specific content remains suppressed.Each checkpoint shows four generations per prompt, making within-prompt variation visible.Source: §D.2 · p. 23, Figure 5 · p. 25
LASTING SISS EraseDiff
- Further Stable Diffusion qualitative examples show immediate and final outputs for four prompts, including “Beige on White Watercolor Skull Bedding” and “Talks on the Precepts and Buddhist Ethics.”The prompt examples also include “Prince Reunites With Warner Brothers, Plans New Album.”Source: Figure 5 · pp. 25–27, LASTING SISS EraseDiff · pp. 26–28