PaperlayerView exact revision on arXiv

Source-linked AI summary

Data Unlearning via Inverse Distillation

Aleksei Leonov, Nikita Kornilov, Zhenhe Zhang, Evgeny Burnaev, Iaroslav Koshelev, Alexander Korotin

arXiv:2609.36099v1cs.LG

TL;DR

Data unlearning must remove selected training data even when retained examples are unavailable, while preserving retained-data quality. IDU combines inverse distillation with a forget/generated-data mixture to train a one-step generator from a full-data teacher. Across the evaluated datasets and backbones, it suppresses forgotten modes while retaining quality close to retrain-distillation references in three settings.

  • Problem

    Data unlearning must remove selected samples whose influence is entangled in model parameters while preserving retained-data quality, even when retained examples are unavailable.

    Source: §1 · p. 2

  • Method

    IDU applies inverse distillation to a mixture of forget samples and generated data, using a pretrained full-data teacher without retained examples, extra feature extractors, or classifiers.

    Source: §1.1 · p. 2, §3.1 · p. 4

  • Results

    Across four dataset–backbone combinations, IDU suppresses forgotten modes; forgotten-class FGR is at most 0.56% for FM and 1.25% for SiD, with retained quality close to retrain-distillation in three settings.

    Source: §4 · p. 7

  • Takeaways & Limitations

    IDU is suitable for unlearning multi-step teacher matching models with additional distillation and for unlearning pretrained generators.

    Source: §5 · p. 9

    In extended FM runs, forgotten-class generation frequency can rise again after roughly 20k generator updates, so ρ and checkpoint selection must be considered jointly.

    Source: §5 · p. 10

Abstract

from arXiv · show

Multi-step matching models, including flow and diffusion models, produce high-quality outputs but incur substantial inference costs and may reproduce unwanted components of their training datasets. We introduce Inverse Distillation Unlearning (IDU), a unified framework that simultaneously distills a teacher multi-step matching model into an efficient one-step student generator and suppresses outputs corresponding to a designated training subset. We first formulate distillation as a min-max objective over a data distribution and then represent this distribution as a mixture of the forget-set and the generated distributions. This allows us to compare this mixture with the teacher's training distribution and recover only the retained data at the optimum. Our method requires only a pretrained full-data teacher and data from the forget set, without access to retained training examples, extra feature extractors or classifiers. Extensive experiments on MNIST and CIFAR-10 datasets under flow-matching and score-based diffusion settings demonstrate that IDU substantially reduces the generation frequency of forgotten classes while preserving generation quality on the retained classes. To the best of our knowledge, IDU is the first unified framework for simultaneous unlearning and distillation in unconditional flow-matching and score-based models.

1 INTRODUCTION

Data unlearning seeks to remove selected samples’ influence without retraining, while preserving retained-data quality despite unavailable retained examples. IDU addresses the gap in model-agnostic one-step unlearning by combining unlearning with distillation.

1 INTRODUCTION

  • Data unlearning uses selected samples without class or prompt anchors, whereas class unlearning typically uses labels and aims to remove unwanted classes while preserving others.Concept unlearning extends removal to prompt-triggered concepts or styles.Source: §1 · pp. 1–2
  • Forget and retained data are entangled in model parameters, and retained samples may be unavailable even though their generation quality must be preserved.The ideal result behaves as if trained only on remaining data, without retraining from scratch.Source: §1 · p. 2
  • Existing data-unlearning work has made little progress toward effective, easy-to-tune, model-agnostic approaches for one-step generators.Prior approaches use varied losses and optimization frameworks to distinguish forget from remaining data.Source: §1 · p. 2

1.1 CONTRIBUTIONS

  • IDU combines data unlearning with distillation for one-step generators, using a pretrained full-data teacher and requiring neither retained examples nor extra feature extractors or classifiers.It compares the mixture of forget and generated data with the teacher’s training distribution to penalize reproducing unwanted samples.Source: §1.1 · p. 2

2 RELATED WORK

The related work frames matching models as approximations of conditional reverse-process functions, with distillation offering a route to one-step generation. Existing unlearning methods span data and class settings, but one-step approaches remain limited.

2.1 DIFFUSION, FLOW AND MATCHING MODELS

  • Diffusion, score-based, and flow models approximate reverse-process functions through conditional estimates, with flow models learning an ODE drift between data and noise.Matching models collectively refer to models matched with these conditional function estimates.Source: §2.1 · pp. 2–3

2.2 DISTILLATION AND ONE-STEP MODELS

  • Matching distillation trains a one-step generator under a multi-step teacher by fitting a fake model to generated data while reducing its discrepancy from the teacher.The inverse-distillation view recovers the teacher’s data distribution at its optimum.Source: §2.2 · p. 3

2.3 DATA UNLEARNING

  • Data unlearning seeks to replace training on the original distribution with training on retained data, while weighted forget and remaining losses trade forgetting against retained-sample quality.Methods vary the losses and optimization, including gradient masking, importance sampling, score-function orthogonality, and constrained optimization.Source: §2.3 · pp. 3–4
  • Other data-unlearning methods use variational guidance or energy weighting; UOT-Unlearn shifts a one-step generator but requires feature extraction and tuning, with reported scalability and diversity limits.The cited consistency-model adaptations use classifiers and do not directly extend to non-self-consistent generators.Source: §2.3 · p. 4

2.4 CLASS UNLEARNING

  • Class unlearning removes unwanted classes from conditional models while preserving other classes, often by combining forget and remaining losses or redirecting class attributes.These techniques are not always adaptable to data unlearning.Source: §2.4 · p. 4
  • SFD performs class forgetting during conditional inverse distillation by replacing the teacher score for forgotten classes with a safe-class score in the generator loss.Source: §2.4 · p. 4

3 INVERSE DISTILLATION UNLEARNING

IDU applies inverse distillation to a mixture of forget samples and generated data, comparing it with a full-data teacher. At the correct forget proportion, the optimum generator recovers only retained data; practical training alternates fake-model and generator updates.

3.1 METHOD DESCRIPTION

  • IDU parameterizes the distribution optimized by inverse distillation as a mixture of forget samples and generated data, substituting generated data for the unavailable retained distribution.The mixture is compared with the teacher’s original-data distribution through the min-max objective.Source: §3.1 · p. 5
  • At the correct forget proportion ρ = π, the optimal generator retrieves only the remaining data because the forget component is already represented by forget samples.The theorem formalizes this forgetting property.Source: §3.1 · p. 5

3.2 PRACTICAL DETAILS

  • IDU alternates between updating a fake model on noised generated and forget samples and updating the generator with the fake model fixed.The generator update uses the frozen teacher and fake model; its modified loss scales the second term by αSiD, whose useful value depends on model type and architecture.Source: §3.2 · p. 5
  • The only method-specific hyperparameter is ρ, which controls forgetting strength; its theoretically justified value is the forget-data proportion, but other values may trade forgetting against retention quality.The paper recommends trying values in practice; the practical pipeline weights the fake-model branches by ρ and 1 −ρ.Source: §3.2 · p. 5, Figure 1 · p. 6

4 EXPERIMENTS

Experiments test IDU across flow-matching and SiD backbones on MNIST and CIFAR-10, using forget samples without labels or prompts. IDU suppresses forgotten classes while generally retaining quality near retrain-distillation references, with results depending on the forget-mixture weight.

Experimental setup

  • The experiments use FM and EDM-VP SiD on MNIST and CIFAR-10; IDU receives forget samples but not labels or prompts, and retained-data FID and classifier-based FGR evaluate outcomes.FID references differ: full-data for teachers and ordinary distillation, retained-data for IDU and retraining; a retained-data teacher and distilled student provide retraining baselines.Source: §4 · pp. 6–7

Main results

  • At most 0.56% FGR for FM and 1.25% for SiD, while IDU retains quality close to retrain-distillation in three settings and improves on it for FM–MNIST.The FM–MNIST oracle comparison is sensitive to the retained-only retraining baseline.Source: Table 1 · p. 7, §4 · p. 7

Robustness across forget classes

  • Mean Retain FID/FGR across ten single-class tasks is 4.54/0.16% for FM–MNIST, 7.73/0.84% for FM–CIFAR-10, and 3.61/0.75% for SiD–CIFAR-10.Low FGR is not confined to the class pairs used in the main experiments.Source: §4 · pp. 7–8, Table 2 · p. 8

Effect of the forgetting weight

  • For CIFAR-10 FM, ρ = 0.6 is the empirical quality–forgetting balance; increasing ρ to 0.8 improves forgetting at substantial FID cost, while ρ = 0.9 diverges.For SiD, larger ρ worsens Retain FID and class-wise FGR varies non-monotonically; at ρ = 0.9, FGRs approach the full-data teacher while Retain FID rises to 10.36.Source: Table 3 · p. 8, §4 · p. 8

5 DISCUSSION AND COMPARISON

IDU uses teacher guidance and a generated–forget data mixture to penalize reproduction of unwanted samples, rather than relying on adversarially combined forget and retain losses. The discussion compares its scope and results with other unlearning approaches and identifies FM optimization stability as a limitation.

Training and evaluation requirements

  • IDU training uses a frozen teacher and forget samples only; retained images and external classifiers do not enter its losses or gradient updates.Without retained-data evaluation, ρ can start from the forget-set proportion, but larger values do not necessarily improve forgetting and may degrade fidelity.Source: §5 · p. 9

Data unlearning methods

  • IDU supports unlearning multi-step matching teachers with additional distillation and unlearning pretrained generators.Distilled generators provide efficient one-step inference rather than the teacher’s multi-step sampling.Source: §5 · p. 9
  • IDU models generated and forget data jointly as a mixture within the fake model, avoiding the adversarial multi-objective optimization used by many matching-model unlearning methods.The generator loss compares the mixture-trained fake model with the teacher; excess generated forget data moves the fake model away from the teacher, which the generator loss counteracts.Source: §5 · pp. 8–9
  • For CIFAR-10 classes 1, 6, and 8, IDU’s Retain FIDs are 2.96, 4.16, and 3.22 versus UOT-Unlearn’s 9.90, 5.11, and 5.88; IDU’s FGRs are 0.61, 0.53, and 0.54 versus around 2, 0.9, and 1.5.UOT-Unlearn uses an unbalanced-optimal-transport cost requiring a feature extractor and manual tuning; its code was unavailable, so the comparison uses its best unlearned CTM model with similar initial FID.Source: §5 · p. 9

Class unlearning methods

  • Unlike SFD’s conditional class-unlearning setup, IDU uses forget samples and does not prompt the generator with class labels, while reporting comparable results under the same conditions.For CIFAR-10 class 0, SFD reports Retain FID 3.1 and FGR 0.36 versus IDU’s 3.54 and 0.58.Source: §5 · p. 9

Optimization stability across backbones

  • FM can suppress target classes early but see their generation frequency rise again after roughly 20k updates, so ρ and checkpoint selection must be coordinated.This reversal was not observed during the evaluated SiD training horizon; the authors identify it as a limitation of the current FM instantiation.Source: §5 · p. 10

A.1 PROOF OF THEOREM 1

The proof states that inverse distillation reaches its zero optimum exactly when it retrieves the teacher data. The accompanying sections describe the minimized loss, SFD’s class-specific losses, model architectures, and teacher sampling.

A.1 PROOF OF THEOREM 1

  • The inverse scheme attains optimum 0 if and only if the teacher data is retrieved.The proof describes this retrieval as p0 = p∗0.Source: §A.1 · p. 16

A.2 THE MINIMIZED DISTANCE

  • For fixed t and x_t, the UM loss is minimized by the conditional mean, while its bias–variance decomposition term is independent of f.Source: §A.2 · p. 16

A.3 SFD’S DETAILS

  • SFD aligns forget-class conditional scores with safe-class teacher scores, leaves remaining classes unswapped, and uses α_SiD usually in [0.5, 1.2].Its fake-model loss remains the same for all classes.Source: §A.3 · p. 17

B.1 ARCHITECTURES AND TEACHER CHECKPOINTS

  • CIFAR-10 uses TorchCFM U-Net for FM and DDPM++ for SiD; MNIST adapts both architecture families to 28 × 28 grayscale inputs.Source: §B.1 · p. 18

B.2 TEACHER SAMPLING

  • FM teachers use Dopri5 on CIFAR-10 and 100-step Euler on MNIST, while SiD teachers use a deterministic 18-step EDM sampler.Every distilled generator produces a sample in one step.Source: §B.2 · p. 18

B.3 EVALUATION CLASSIFIERS

Forgotten-class generation rate is measured by classifying 50,000 generated images; the classifiers serve only as external evaluation instruments.

B.3 EVALUATION CLASSIFIERS

  • FGR is the percentage of 50,000 generated images classified as each forgotten class, measured with public LeNet-5 for MNIST and ResNet-56 for CIFAR-10.The classifiers are not used in IDU training and supply no loss, gradient, feature representation, or conditioning signal.Source: §B.3 · p. 18

B.4 FID PROTOCOL AND EVALUATION DATA

The section specifies FID references and evaluation conventions, optimization horizons and checkpoint selection, and the two IDU initialization routes.

B.4 FID PROTOCOL AND EVALUATION DATA

  • FID uses 50,000 generated images and real training images from the target distribution; teacher sampling is multi-step, while distilled and IDU generators use one-step inference.Test images are not used as FID references; CleanFID’s sample covariance differs from the original evaluator’s population covariance by a scale factor of about 1.00002 at 50,000 samples.Source: §B.4 · p. 18

B.5 OPTIMIZATION HYPERPARAMETERS

  • FM generator horizons are approximately 30,000 iterations on MNIST and 50,000 on CIFAR-10; SiD distillation and IDU use a 100 Mimg horizon.Horizons are maxima; reported checkpoints minimize observed FID, with FM selection before the late forgetting reversal.Source: §B.5 · p. 19

B.6 INITIALIZATION

  • IDU can fine-tune a pretrained generator or distill from scratch, differing in whether Gθ starts from that generator or the one-step teacher inference scheme.Source: §B.6 · p. 19

B.7 FINE-TUNING EXPERIMENTS

CIFAR-10 fine-tuning reduces forgotten-class generation relative to pure distillation, with different Retain FID and FGR trade-offs for FM and SiD.

B.7 FINE-TUNING EXPERIMENTS

  • FM fine-tuning yields Retain FID 5.17 ± 0.08 and FGRs 0.89 ± 0.02% and 0.42 ± 0.03% for forgotten classes 1 and 9.Source: §B.7 · p. 19
  • SiD fine-tuning yields Retain FID 3.12 ± 0.06 and FGRs 2.69 ± 0.06% and 2.60 ± 0.04% for forgotten classes 1 and 9.Source: §B.7 · p. 19
  • Relative to IDU trained from scratch, FM fine-tuning improves Retain FID but slightly increases FGR, while SiD has comparable Retain FID but higher FGR.Source: §B.7 · p. 19

B.8 CODE AND CHECKPOINT RELEASE

  • The authors plan to release the source code, configurations, evaluation scripts, and checkpoints used for the main results upon publication.Source: §B.8 · p. 19

B.9 MNIST FM RETRAINING AND SENSITIVITY TO ρ

This appendix examines the retained-only FM/MNIST retraining baseline and how the forgetting weight ρ affects quality and forgetting. The observed trade-off is sensitive to ρ, including collateral suppression at higher weights.

B.9 MNIST FM RETRAINING AND SENSITIVITY TO ρ

  • The retained-only FM/MNIST baseline is sensitive: its retrained teacher has Retain FID 0.34, while poor digit-2 generation raises its distilled counterpart to 5.72.The authors suggest the teacher's low FID may reflect overfitting to the smaller retained subset.Source: §B.9 · pp. 19–20
  • ρ = 0.4 gives the best observed Retain FID–FGR trade-off; at ρ ≥ 0.6, target FGR reaches zero, extra digits are suppressed, and Retain FID exceeds 10.Additional suppressed digits are most consistently 2 and 5, with 8 or 9 affected in some runs.Source: §B.9 · p. 20, Table 5 · p. 20
  • Below ρ = 0.4, forgetting digits 3 and 7 weakens without improving Retain FID; SiD remains stable at ρ = 0.2 without the same collateral suppression.The authors describe this sensitivity as specific to their low-dimensional MNIST setting and FM architecture and training recipe.Source: §B.9 · p. 20

C VISUAL RESULTS

Across MNIST and CIFAR-10, the visual comparisons show teacher samples containing the target classes and IDU samples visibly suppressing them while retaining other classes. The figures illustrate results for both FM and SiD.

C VISUAL RESULTS

Related papers

Loading 2609.36099v1…