PaperlayerView exact revision on arXiv

Source-linked AI summary

Compositional Concepts Erasing: Disentangling Object-Style via Domain Manifold Unlearning

Yang Xiao, Kaiyuan Deng, Gen Li, Zinan Ling, Zhiwen Fan, Xiaolong Ma, Bo Hui

arXiv:2610.14514v1cs.CVcs.AI

TL;DR

Object- or style-level unlearning is insufficient for fine-grained, selective removal of object–style combinations [9] [10]. DMU models target-domain manifolds with a non-parametric object proxy and combines manifold unlearning with denoising loss on retained and anchor sets. DMU is reported to outperform state-of-the-art methods for batch and sequential unlearning and to improve robustness across multiple requests.

  • Problem

    Object- or style-level unlearning is insufficient for fine-grained, selective removal of object–style combinations [9] [10].

    Source: §1 · pp. 1–2

  • Method

    DMU models target-domain manifolds with a non-parametric object proxy and combines manifold unlearning with denoising loss on retained and anchor sets.

    Source: §1 · p. 2, §2.2 · p. 3, §2.3 · p. 4

  • Results

    DMU is reported to outperform state-of-the-art methods for batch and sequential unlearning and to improve robustness across multiple requests.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The authors present DMU as a framework for selectively removing object–style domains while preserving core utility and generalization.

    Source: §1 · p. 3

    Experiments unlearn one target object across multiple styles and report averaged results across objects.

    Source: §4.1 · p. 8

Abstract

from arXiv · show

Diffusion models pose significant risks regarding the generation of illicit content and copyright violations. This has catalyzed the development of machine unlearning to erase harmful concepts or copyrighted styles from a model's generative repertoire without compromising its general performance. However, previous unlearning approaches on diffusion models have largely focused on either object-level or style-level removal and failed to address the challenge of object-style combination unlearning. At the same time, multiple unlearning requests often lead to catastrophic forgetting. To address these challenges, we propose Domain Manifold Unlearning (DMU), which uses an EMA-updated feature proxy extracted from intermediate diffusion features to represent the retained target-object direction. As a lightweight estimating representation of an object, DMU enables selective unlearning of domain-specific components in a specific object while preserving other semantic representations, thereby significantly enhancing robustness in multiple combination unlearning scenarios. Extensive experiments show that DMU significantly mitigates catastrophic forgetting, outperforming state-of-the-art methods of batch unlearning and sequential unlearning.

1 Introduction

Fine-grained object–style unlearning is needed because removing only an object or style can over-suppress content, while repeated requests can degrade retained representations. DMU models domain manifolds to isolate targeted distributions, and the authors report that it outperforms state-of-the-art batch and sequential unlearning methods.

1 Introduction

  • Object–style combination removal offers finer control than removing an entire object or style, which can over-suppress permitted content [9] [10].Examples include removing weapons in cartoon styles while retaining weapons in real-world images and benign cartoon content.Source: §1 · pp. 1–2
  • Batch unlearning gradients may interfere, while sequential requests can amplify gradient noise and degrade accuracy on retained data [10].The reported retention decline affects both the target object in remaining styles and other objects.Source: §1 · p. 2
  • DMU models target-domain manifolds to decouple domain-specific components from shared semantic representations, addressing gradient conflicts and noise amplification.The stated aim is to excise targeted distributions while preserving core utility and generalization.Source: §1 · pp. 2–3
  • DMU is reported to outperform state-of-the-art methods for batch and sequential unlearning.The authors also report improved robustness across multiple unlearning requests.Source: §1 · p. 3

2 Method

DMU separates unlearned target-object styles from retained semantics using an object proxy built from intermediate diffusion features. Its objective combines manifold-based unlearning with denoising-loss preservation, while the proxy’s EMA stabilizes estimates across batches.

2.2 Non-parametric Object Proxy in the Manifold

  • DMU estimates an object proxy from intermediate U-Net features and updates it by EMA using retained-style positive samples only.EMA stabilizes estimation when batches may not contain enough positive samples; with sufficient positive sampling, the current-batch variant achieves comparable performance.Source: §2.2 · p. 3

2.3 The view of von Mises-Fisher (vMF) distribution

  • The normalized proxy serves as a semantic reference: retained samples align with it, while samples from unlearned styles are pushed away.Under the vMF interpretation, greater concentration corresponds to greater cosine similarity to the proxy; denoising loss preserves retained samples’ stylistic variance.Source: §2.2 · p. 3, §2.3 · pp. 4–5
  • The final objective combines manifold loss for decoupling target-object features with denoising loss on retained-style and anchor data to preserve general semantics.The dataset is divided into retained-style target examples, unlearned-style target examples, and examples of other objects across styles.Source: §2.1 · p. 3, §2.3 · p. 4
  • The method estimates feature concentration from mean cosine similarity, using a high-dimensional approximation that is described as accurate and numerically stable at d = 1280.A larger mean cosine similarity indicates tighter concentration around the proxy; a smaller value indicates more dispersed features.Source: §2.3 · p. 5

3 Experiments

Experiments evaluate Domain Manifold Unlearning (DMU) in sequential and batch unlearning, using object–style benchmarks and measures of unlearning and retention. DMU achieves the strongest overall balance in the main UnlearnCanvas comparison, while visualizations and ablations examine its behavior and robustness.

3.1 Experiment Setups

  • Evaluations cover sequential requests processed chronologically and batch requests optimized together, with UA, retained accuracies, HA, style accuracy, and CLIP score measuring unlearning and preservation.The stated goal is to suppress target-object generation in specific domains while maintaining other-object performance across domains.Source: §3.1 · p. 5

3.2.1 Main Experiments

  • Across sequential and batch unlearning on UnlearnCanvas, DMU achieves the highest HA while improving UA and preserving retained accuracy, with competitive style consistency and semantic alignment.The default request removes six domains.Source: §3.2.1 · p. 6, Table 3 · p. 6
  • Experiments use UnlearnCanvas, OfficeHome, and PACS with SD1.5, SD2, and SDXL; DMU maintains strong unlearning and retention across the tested datasets and model architectures.OfficeHome and PACS are evaluated with SD1.5; larger backbones SD2 and SDXL are tested on UnlearnCanvas.Source: §3.1 · p. 5, §3.2.1 · pp. 6–7, Table 4 · p. 6

3.2.2 Visualization

  • DMU features begin tightly overlapped, start separating by step 500, and form distinct positive and negative clusters by step 1000, while three compared methods do not fully decouple them.The visualization uses features from 100 U-Net bottleneck batches; positive and negative samples each make up 25% of the plotted samples, with anchors making up 50%.Source: §3.2.2 · p. 7, Figure 4 · p. 7, Figure 3 · p. 7

3.3 Ablation Studies

  • Removing positive alignment sharply reduces RApos, removing negative repulsion collapses UA, and removing denoising preservation slightly degrades retained accuracies and style consistency.Jointly removing positive alignment and positive-set denoising preservation further collapses RApos; the full DMU has the highest HA.Source: §3.3 · pp. 7–8, Table 6 · p. 8
  • HA remains robust without significant decline for momentum values from 0.1 to 0.9, though it oscillates; values 0 and 1 are excluded because they fail to capture dynamic proxy updates.A momentum of 0 uses only current-batch features, while 1 leaves the proxy static at initialization.Source: §3.3 · p. 8
  • At 1000 training timesteps, UA decreases as unlearning requests increase, but rises to approximately 85% at 2000 timesteps for 25 and 30 requests.The authors attribute the degradation to insufficient optimization steps and state that more requests require more iterations.Source: §3.3 · p. 8, Figure 5 · p. 8

4 Discussion and Limitation

The discussion identifies a path toward joint object–style unlearning but bounds current evidence: experiments use one target object, omit latent-space auditing, and do not compare feature layers.

4.1 Extension to multi-domain unlearning.

  • Experiments cover one target object with multiple styles; object-specific proxies could support joint multi-object/style requests, but comprehensive evaluation is future work.The proposed settings include jointly unlearning N objects across M styles or assigning different style subsets to different objects.Source: §4.1 · p. 8

4.2 Auditing and Concept Identifiability

  • Evaluation of forgetting and retention under tested object–style requests does not establish resistance to latent-space recovery or concealment of erased concepts.Latent-space auditing and erasure-footprint identification are described as complementary tests for future investigation.Source: §4.2 · p. 8

4.3 Feature Extraction in Different Layers

  • Feature-layer effects remain untested despite prior diffusion work's use of intermediate layers; Figure 6 illustrates selective domain erasure with retained-domain preservation.The cited background describes intermediate layers as balancing low-level details and high-level semantics.Source: §4.3 · pp. 8–9, Figure 6 · p. 9

5 Related Works

The section maps multi-concept erasure, erased-concept auditing, and domain unlearning, noting unresolved scalability and manifold-retention concerns.

5.1 Multiple Concept Erasing

  • Multi-concept erasure includes negative-guidance ESD, closed-form UCE, LoRA-based MACE, representation-decoupling SepME [29], mask-based methods [4] [13], and video-oriented FADE [30].Sculpting Memory [4] and FIA [13] use masks; CUIG [11] stabilizes sequential updates; FADE [30] addresses residual leakage across video frames.Source: §5.1 · p. 9

5.2 Auditing and Identifying Erased Concepts

  • AVCE [22] audits recoverable erased concepts, while Tracer [23] identifies erased concepts and estimates their number using weight footprints and limited forward probes.These approaches motivate evaluation beyond output-based forgetting and retention metrics.Source: §5.2 · p. 9

5.3 Domain Unlearning

  • ADU [9] uses MMD to disentangle domain distributions, while SSDA uses classification, uncertainty loss, and Gaussian-guided latent alignment; neither tracks retained domains within general knowledge.The passage links this omission to less robust unlearning performance.Source: §5.3 · pp. 9–10

6 Conclusion

DMU uses an EMA-updated non-parametric proxy to model data manifolds and decouple domain-specific features from shared semantics. Evaluations span benchmarks and models, while extensions to other generative modalities are proposed as future work.

6 Conclusion

  • Evaluations across benchmarks and models present DMU as a versatile approach for large-scale and continuous unlearning requests.The conclusion frames DMU as addressing robustness and catastrophic forgetting in multiple-domain unlearning for diffusion models.Source: §6 · p. 10
  • DMU models data manifolds with an EMA-updated non-parametric proxy; the von Mises-Fisher account links manifold-loss optimization to semantic concentration and dispersion.Combining the manifold loss with standard denoising loss is described as preserving core utility and intra-class diversity on retained data.Source: §6 · p. 10
  • The authors propose extending the proxy to the discussed limitation and to other generative modalities, including large language models.Source: §6 · p. 10

A Detailed Experiment Setups

The setup uses pretrained classifiers for UnlearnCanvas metrics and CLIP scores for generated image–prompt pairs.

A Detailed Experiment Setups

A.1 GPU, Training and Test Latency

The experiments use four L40S GPUs; BU and SU have different experiment-time estimates, while image generation takes about one hour.

  • All experiments run on four L40S GPUs.Source: §A.1 · p. 12
  • BU experiments require approximately 210 minutes each.Source: §A.1 · p. 12
  • The entire image generation process typically takes approximately 1 hour.Source: §A.1 · p. 12
  • SU experiments require approximately Number of styles× 210 minutes each.Source: §A.1 · p. 12

A.2 How many objects and how many styles?

The datasets cover several object and style combinations: UnlearnCanvas varies its style set, while OfficeHome and PACS specify their own objects and styles.

  • UnlearnCanvas uses six objects: Birds, Trees, Cats, Horses, Jellyfish, and Human.Source: §A.2 · p. 12
  • UnlearnCanvas style sets are listed at sizes 6, 12, 15, 25, and 30, with their specific styles enumerated for each size.Source: §A.2 · p. 12
  • OfficeHome uses six objects—Bike, Calculator, Monitor, Refrigerator, Telephone, and TV—and two styles: Art and Clipart.Source: §A.2 · p. 12
  • PACS includes all objects in PACS and three styles: art painting, cartoon, and sketch.Source: §A.2 · p. 12

A.3 Image Generation

Image generation uses specified random seeds and produces four images per object–style pair and seed. The section also reports a lambda ablation and presents case examples.

A.3 Image Generation

  • For positive and negative sets, image generation uses seven random seeds; the anchor set uses five different seeds.The positive and negative seeds are 42, 144, 79, 27, 599, 123, and 577; anchor seeds are 188, 288, 388, 488, and 588.Source: §A.3 · p. 12
  • For each object–style pair and seed, the protocol generates four images using that seed plus 0–3, then aggregates samples across seeds.Source: §A.3 · p. 12
  • Across λ values from 0.05 to 1.00, DMU maintains strong unlearning, retention, and generation-quality results; removing its objective at λ = 0.0 degrades unlearning.The main-text experiments use λ = 0.15.Source: Table 7 · p. 13, §A.3 · p. 13

B Case Study

C Application in LLM Unlearning.

DMU is extended to LLM unlearning using retain-derived reference representations. It achieves strong forgetting and competitive general capability, with a better forgetting–retention trade-off than prior methods.

Related papers

Loading 2610.14514v1…