HyperErase: Scale-Calibrated Hypernetwork for Multi-Concept Erasure in Text-to-Image Models

1 INTRODUCTION

• Internet-scale training data can leave text-to-image models capable of generating objectionable or harmful imagery.Full retraining is prohibitively costly, motivating concept erasure as a parameter-efficient alternative that selectively suppresses target concepts while aiming to preserve broader generative fidelity.Source: §1 · pp. 1–2
• Static adapters cannot adapt to open-ended prompt phrasing, and multi-concept use can accumulate interference between erased concepts and preserved classes.Storing separate LoRAs requires manual routing, while merging can introduce crosstalk and direction drift.Source: §1 · p. 2
• HyperErase reframes concept erasure as prompt-conditioned parameter amortization, training a hypernetwork to generate prompt-specific LoRAs from textual descriptions.The approach uses ESD-derived prompt–checkpoint pairs and avoids per-prompt gradient optimization and manual LoRA merging.Source: §1 · p. 2
• HyperErase adds decoupled rectification to stabilize generated adapters and reports erasure performance comparable to gold single-concept baselines with stronger generative capability.The rectification strategy uses square-root scale encoding and pattern and scale corrections.Source: §1 · p. 2

2 RELATED WORKS

• Concept-erasure work spans dataset filtering, model retraining, training-based fine-tuning, and training-free approaches.Filtering can exclude sensitive content at the dataset level, while retraining from scratch carries computational and temporal costs.Source: §2 · pp. 2–3
• HyperErase trains on gold-baseline checkpoint–prompt pairs, then generates pattern and scale checkpoints from a new input and recouples them into a LoRA.The overview also describes decoupled magnitude rectification as a way to mitigate accumulated decay across concepts.Source: Figure 2 · p. 3
• Because target concepts may persist through benign prompts, direct suppression in pretrained models has developed into training-based and training-free branches.The cited passage notes that residual target concepts can prompt iterative retraining cycles.Source: §2 · p. 3
• Training-based examples include ESD's noise-prediction alignment, UCE's closed-form attention-weight editing, and DVE's timestep-wise semantic projection.ESD and UCE are described as training-based approaches; DVE is presented as parameter-free with dynamically adaptive suppression.Source: §2 · p. 3

3.1 PARAMETER GENERATOR

• A generator trained on concept-specific prompt–checkpoint pairs produces a corresponding LoRA for each inference prompt, erasing the target when present and otherwise generating normally.Training uses ESD checkpoints collected across training snapshots and prompt pools for each concept.Source: §3.1 · pp. 3–4
• LoRA weights are chunked into tokens that separate normalized patterns from scale statistics, and importance-weighted MSE emphasizes valid positions with higher cross-checkpoint variance.The token grid stacks chunks across U-Net layers; the variance weights emphasize positions carrying erasure-relevant information and down-weight near-constant padding.Source: §3.1 · pp. 3–4
• In multi-concept training, prediction error under convex exponential decoding biases decoded scales upward, and independent Aℓ and Bℓ inflation compounds in their product.The resulting over-scaling preserves erasing direction but can push the U-Net off its learned denoising manifold and corrupt generation.Source: §3.1 · p. 4
• Square-root scale encoding replaces logarithmic encoding, with a fixed constant set to the median standard deviation across weight chunks.The decode compresses LoRA standard deviations into an O(1) interval and equalizes loss contributions across chunks with widely differing raw magnitudes.Source: §3.1 · p. 4

3.2 DECOUPLED MAGNITUDE RECTIFICATION

• MSE regression shrinks predicted pattern norms relative to individual teacher patterns, so detokenization renormalizes each predicted chunk pattern to unit Frobenius norm.The shrinkage is approximately uniform across concepts because per-checkpoint pattern variance is comparable throughout the concept set.Source: §3.2 · p. 5
• Canonical scale anchoring replaces predicted scale rows with concept-specific priors averaged from teacher checkpoints in raw weight space and then re-encoded.This addresses cross-concept mean pull and residual variance-dependent displacement in predicted scales.Source: §3.2 · p. 5
• Pattern renormalization and canonical scale anchoring operate on disjoint token subspaces, yielding adapters described as bias-stable across training-set sizes and close to the gold baseline.The generator is relieved of scale regression, while pattern channels continue to be learned from prompt semantics.Source: §3.2 · pp. 4–5

4.1 EXPERIMENTAL SETUP

• The experiments compare static and sequential ESD and HyperErase variants, evaluate three erasure tasks across 14 concepts, and use CLIP and FID for alignment and image quality.Source: §4.1 · pp. 5–6

4.2.1 NUDITY ERASURE

• HyperErase Mix approaches ESD Single’s nudity-erasure performance while achieving better FID and CLIP scores; it also remains close to the gold baseline against adversarial attacks.Source: §4.2.1 · pp. 6–7, Table 1 · p. 7, Table 2 · p. 7
• HyperErase remains close to the gold baseline against multiple nudity attacks; in sequential training, later concepts preserve generation capability, but early erasure can degrade and suffer catastrophic forgetting without regularization.Source: Table 2 · p. 7, §4.2.1 · pp. 7–8
• HyperErase’s LPIPS and t-SNE distributions align more closely with the gold baseline than ESD Mix, and its LPIPS is closer to real images than the gold baseline’s.Source: Table 3 · p. 7, Figure 4 · p. 7, §4.2.1 · p. 8

4.2.2 OBJECT ERASURE

• For object erasure, HyperErase approaches ESD Single’s erasure effect while improving CLIP and FID and retaining non-target concept generation capability.Source: Table 4 · p. 8, §4.2.2 · p. 8

4.2.3 ARTIST STYLE ERASURE

• For artist-style erasure, HyperErase exhibits an erasure effect close to ESD Single.Source: §4.2.3 · p. 8, Table 5 · p. 9

4.3 FURTHER ANALYSIS

• On FLUX.1-dev, HyperErase still erases nudity, although erasure declines somewhat; FID and CLIP remain consistent with SD v1.4 results.Source: §4.3 · p. 9, Table 6 · p. 9
• Without decoupled magnitude rectification, decay accumulates across 14 concepts and significantly degrades erasure; with it, erasure approaches the gold baseline.Source: Table 7 · p. 9, §4.3 · p. 9

• Across nudity, artist-style, and object erasure, HyperErase matches the gold single-concept baseline in erasure effectiveness while improving the trade-off with image quality and semantic alignment.Source: §5 · pp. 9–10

• Across all 14 tasks, HyperErase uses Stable Diffusion v1.4 and trains the parameter generator on 50 ESD-u teacher checkpoints per concept, sampling 32 text prompts per step.Source: §A.1 · p. 13
• Text conditions use a frozen all-MiniLM-L12-v2 Sentence-BERT encoder with hidden dimension 384, while LoRA checkpoints are represented as token tensors with N = 7,502, h = 10, and tw = 130.Source: §A.1 · p. 13
• Training uses an importance-weighted mean-squared objective, AdamW, and a learning rate of 3×10^-5; inference uses DDIM with 50 denoising steps and guidance scale 7.5.Source: §A.1 · p. 13, Table 8 · p. 13

• Nudity erasure is evaluated on all 4,703 I2P prompts and seeds, with adversarial robustness tested using four attack methods and exposed body parts measured by Nudenet.Source: §A.2 · p. 13
• Object erasure covers 10 ImageNet concepts, with 500 images generated per object and target and non-target ASR measured using a ResNet-50 classifier.Source: §A.2 · p. 13
• Artist-style erasure tests Van Gogh, Da Vinci, and Picasso using the Conceptprune validation dataset and UnlearnDiff’s style classifier.Source: §A.2 · p. 14
• For general usefulness, each method generates 10,000 MS-COCO images for CLIP and FID evaluation; across 10 object concepts, HyperErase has comparable erasure and improved CLIP and FID versus ESD Single.Source: §A.2 · p. 14, Table 9 · p. 14

B.1 DETAILS OF OBJECT ERASURE

• Across object concepts, HyperErase Mix outperforms ESD Single on CLIP and FID, while ASRe and ASRk are close to or partly better than the gold baseline.The authors interpret these comparisons as evidence that HyperErase learns from the gold baseline for multi-concept erasure.Source: §B.1 · p. 14

Ablation on The Epoch

• At 2000 epochs, erasure is already close to the gold baseline; increasing epochs decreases exposed body while FID and CLIP remain unchanged.The authors report that erasure improves as epochs increase, without requiring a very large epoch count to approach the gold baseline.Source: §B.2 · p. 14, Table 11 · p. 15

Ablation on The Number of Concepts

• Across 3, 5, 10, and 14 erased concepts, exposed body decreases only slightly and stays near baseline, while CLIP and FID remain basically consistent.The authors speculate that lower exposed-body values with fewer concepts may reflect overfitting when all configurations use the same epoch count; Table 10 covers these comparisons.Source: §B.2 · pp. 14–15, Table 10 · p. 15

Ablation on The Gold Baseline and Modules

• With EraseAnything as the gold baseline and its text-projection layers as targets, HyperErase achieves concept-erasure performance close to the baseline.This test supports the authors’ claim that the approach is not tied to ESD-U or non-cross-attention layers.Source: §B.2 · p. 15, Table 12 · p. 15

Ablation on non-LoRA Gold Baseline

• Using RECE as a non-LoRA gold baseline, HyperErase can generate erased LoRAs, but fitting RECE is not always successful.Some experiments reach 301 exposed parts, which the authors speculate may stem from the weight-to-LoRA conversion and may require targeted tuning; Table 13 reports these comparisons.Source: §B.2 · p. 15, Table 13 · p. 15

B.3 ADDITIONAL VISUAL RESULTS

• Supplementary visual examples cover additional Nudity, Object, and Artist Style erasure, plus results on MS-COCO.The section assigns Tables 7–9 to the three erasure categories and Table 10 to MS-COCO visual results.Source: §B.3 · p. 15

C.1 ALGORITHM

• Algorithm 1 trains a generator on prompt–LoRA examples with a frozen BERT encoder, then maps a new prompt to an adapter that is applied to the pretrained U-Net.Inference repeats the prompt across M slots and reconstructs each chunk from its predicted pattern and scale channels.Source: §C.1 · p. 16

C.2 CANONICAL SCALE ANCHORING

• Canonical scale anchoring assigns each checkpoint for a concept the same pre-encoding scale target, removing within-concept target variance and making scale regression concept-level.The stated result applies to the scale-only regression subproblem.Source: §C.2 · pp. 16–17

C.3 SQUARE-ROOT DECODER BIAS

• The square-root decoder turns zero-mean encoded noise into an additive bias proportional to encoded-error variance and Sσ.For symmetric error, the stated leading term of the second moment is linear in σ; the derivation motivates reducing within-concept scale variation before optimization.Source: §C.3 · p. 17

C.4 INFERENCE-TIME RENORMALIZATION

• Inference-time renormalization removes a predicted pattern chunk’s affine magnitude and offset while preserving its orientation, so calibration changes effective layer magnitude rather than LoRA direction or subspace.The separately decoded or canonical pair (µ, σ) determines reconstructed chunk magnitude.Source: §C.4 · p. 18

C.5 FIRST-ORDER LORA FACTOR ERROR

• For small factor errors, scale-row errors are amplified by factor norms; the decoupled corrections address magnitude and pattern deviations separately.The calibration branch corrects residual magnitude without modifying the learned LoRA subspace.Source: §C.5 · p. 19

D AI DISCLOSURE SECTION

• The disclosure states that generative-AI tools were used for tasks including synthetic data, writing assistance, translation, code, literature work, and editing, with the authors taking responsibility for the final content.The disclosure also lists categories of tasks for which such tools were not used or were not applicable.Source: D AI DISCLOSURE SECTION · p. 19
• Supplementary figures show additional adversarial-attack results, object-erasure results, Van Gogh erasure results, and method comparisons on MS-COCO.The figure captions identify the content but do not state comparative outcomes.Source: Figure 7 · p. 20, Figure 8 · p. 20, Figure 9 · p. 21