Source-linked AI summary
eval-unlearn: Benchmarking unlearning in Text-to-Image Diffusion Models
Mansi, Nikhil Raghavan, Zixia Huang, Kai Sheng Ong, Ji Shen Lim, Brandon Siao Xiang Ling, Francesco Leofante
TL;DR
Heterogeneous evaluation conditions make reliable comparison among T2I concept-unlearning methods difficult. eval-unlearn provides a unified benchmark spanning twelve techniques and nine metrics, with a plugin architecture and reproducible evaluation pipeline. Its conclusion states that standardization enables fair cross-method comparison that is otherwise impractical.
Problem
T2I unlearning methods are evaluated with different datasets, metrics, and hyperparameter regimes, making reliable cross-method comparison difficult.
Source: §1 · p. 1Method
eval-unlearn provides a shared benchmarking pipeline for twelve unlearning techniques and nine metrics, with plugin registration for external techniques and metrics.
Source: §1 · p. 1, §2 · pp. 1–2, §3 · p. 2Results
Standardizing evaluation across twelve techniques and nine metrics enables fair cross-method comparison that is otherwise impractical.
Source: §4 · p. 2Takeaways & Limitations
The MIT-licensed framework is intended to lower barriers to community adoption and contribution, with future work targeting broader model, metric, and concept coverage.
Source: §4 · p. 2
Abstract
from arXiv · showhide
The rising number of concept unlearning techniques for text-to-image (T2I) diffusion models has produced a fragmented evaluation landscape. Methods are assessed under heterogeneous experimental conditions making principled cross-method comparison difficult. We present eval-unlearn, an open-source Python library providing a unified, reproducible benchmarking framework for concept unlearning in T2I Diffusion models. eval-unlearn integrates twelve published unlearning techniques spanning fine-tuning, closed-form model editing, and inference-time intervention, alongside nine complementary evaluation metrics covering erasure efficacy, adversarial robustness, generative quality, and concept retention. Its plugin architecture lets third-party techniques and metrics self-register without modifying the core framework, and its streaming, batched pipeline supports efficient evaluation of both standard NSFW concepts and arbitrary general concepts. As a further contribution, we release a public leaderboard on HuggingFace along with an interactive tool for real-time evaluation of unlearning techniques. The leaderboard compares nudity concept erasure case study across all twelve techniques, exposing significant accuracy-quality trade-offs that are obscured by heterogeneous evaluation. eval-unlearn is released under the MIT license; the package, code, leaderboard, and documentation are all available at https://eval-unlearn.readthedocs.io.
1 Introduction
T2I unlearning offers a targeted alternative to costly, often ineffective retraining, but heterogeneous evaluations make reliable cross-method comparison difficult. eval-unlearn addresses this gap with a common framework spanning three technique categories.
1 Introduction
- Machine unlearning selectively suppresses concepts through targeted weight changes or inference-time interventions, avoiding fully retraining on curated data that is expensive and often ineffective due to compositionality.The motivation is that T2I models trained on internet-scraped corpora can generate inappropriate content with negative implications.Source: §1 · p. 1
- The surveyed techniques fall into fine-tuning, closed-form weight editing, and inference-time intervention, which respectively adjust model weights iteratively, update them in one step, or reshape guidance or latent representations.Examples include ESD and CA for fine-tuning, UCE and MACE for closed-form editing, and SLD and SAFREE for inference-time intervention.Source: §1 · p. 1
- Heterogeneous datasets, metrics, and hyperparameters make reliable cross-method evaluation difficult, while existing frameworks cover a narrow set of techniques and lack extension interfaces.eval-unlearn provides a shared pipeline across all three categories, nine standardized metrics, extensible plugins, and a public leaderboard with logged hyperparameters.Source: §1 · p. 1
2 Implemented Techniques and Metrics
The framework implements twelve unlearning techniques and nine metrics to evaluate erasure, robustness, image quality, and concept retention. These methods and metrics can be configured for benchmarking.
2 Implemented Techniques and Metrics
- Twelve techniques spanning fine-tuning, closed-form editing, and inference-time intervention are implemented, generally targeting Stable Diffusion v1.4; SLD instead uses its safety-fine-tuned variant.Custom checkpoints and HuggingFace model IDs can also be evaluated without wrapper code, and technique packages are separately installable.Source: §2 · p. 1
- Nine metrics cover erasure efficacy, adversarial robustness, generative quality, and concept retention.They include I2P attack success rate; attack success rates under Ring-A-Bell, MMA-Diffusion, and P4D; FID and CLIP Score on COCO 2017; TIFA; ERR; and UA-IRA.Source: §2 · pp. 1–2
- Figure 1 presents an abstract overview of the eval-unlearn pipeline.The figure is described as showing the pipeline at an abstract level.Source: Figure 1 · p. 2
- Technique and metric configurations are exposed through runner variables, with defaults based on standardized conventions or values reported in the original publications.The documentation records the configuration conventions and source values.Source: §2 · p. 2
3 Software Design
eval-unlearn separates orchestration from technique and metric internals through fixed interfaces and wrappers. Its runner streams generated-image batches through metrics, while entry points support registering external implementations.
3 Software Design
- Fixed abstract interfaces and wrapper classes follow the Adapter design pattern, keeping orchestration independent of technique and metric internals.Technique versions are pinned to their most recent release for reproducibility.Source: §3 · p. 2
- Installed techniques register through Python entry points, which the runner discovers and makes addressable by name without manual imports or framework changes.The same registration mechanism applies to metrics and dataset loaders.Source: §3 · p. 2
- Metrics process generated images batch by batch through update(), then produce a final score through compute().Dataset loaders can generate and cache prompts or stream fixed benchmark datasets from HuggingFace without holding the complete dataset in memory.Source: §3 · p. 2
- Frozen dataclass configurations validate hyperparameters at initialization, and each run writes concise and extended reports to a uniquely identified output directory.The extended report includes scores and full configuration.Source: §3 · p. 2
- MultiBenchmarkRunner evaluates one technique across multiple metrics in a single pass, reusing the loaded model while streaming batches and aggregating metric results.Metric objects are deleted and garbage-collected between evaluations to reclaim VRAM; inference uses FP16, while fine-tuning uses FP32 where numerical stability requires it.Source: §3 · p. 2
4 Conclusion
eval-unlearn provides a unified, reproducible framework for concept-unlearning benchmarks in T2I diffusion models. The conclusion frames standardization across techniques and metrics as enabling fairer comparison, while identifying future expansions.
4 Conclusion
- Standardizing evaluation across twelve techniques and nine metrics enables fair cross-method comparison that is otherwise impractical.The framework is released under an MIT license, and future work includes newer Stable Diffusion variants, computational-overhead metrics, and broader concept coverage.Source: §4 · p. 2
- Planned extensions include Stable Diffusion v2 and SDXL, latency and peak-VRAM metrics, and concepts beyond nudity and violence.Source: §4 · p. 2
- The framework is released under an MIT license to minimize barriers to community adoption and contribution.Source: §4 · p. 2