PaperlayerView exact revision on arXiv

Source-linked AI summary

CoUn: Empowering Machine Unlearning via Contrastive Learning

Yasser H. Khalil, Mehdi Setayesh, Hongliang Li

arXiv:2509.16391v3cs.LGcs.AIcs.CV

TL;DR

Machine unlearning must remove selected training data’s influence while preserving knowledge of the remaining data, but existing approaches based on label manipulation or weight perturbation have limitations. CoUn uses contrastive and supervised learning on retain data to adjust representations through semantic similarity. It consistently outperforms state-of-the-art unlearning baselines, and its contrastive-learning module improves existing methods.

  • Problem

    Existing label-manipulation and weight-perturbation methods can have limited unlearning effectiveness while machine unlearning must remove forget-data influence and preserve retain knowledge.

    Source: §1 · p. 2

  • Method

    CoUn applies contrastive and supervised learning exclusively to retain data, indirectly adjusting forget representations through semantic similarity while mitigating cluster collision among retain representations.

    Source: §1 · p. 2

  • Results

    CoUn consistently outperforms state-of-the-art machine-unlearning baselines, and integrating its contrastive-learning module improves existing baselines’ unlearning effectiveness.

    Source: §4.2 · p. 8, §5 · p. 10

  • Takeaways & Limitations

    CoUn adjusts forget representations toward semantically similar retain clusters while using supervised learning to preserve separation among retain clusters.

    Source: §5 · p. 10

    Evaluation is limited to vision-based classification on relatively small CIFAR-10, CIFAR-100, and TinyImageNet datasets.

    Source: H Limitations and Future Work · p. 22

Abstract

from arXiv · show

Machine unlearning (MU) aims to remove the influence of specific "forget" data from a trained model while preserving its knowledge of the remaining "retain" data. Existing MU methods based on label manipulation or model weight perturbations often achieve limited unlearning effectiveness. To address this, we introduce CoUn, a novel MU framework inspired by the observation that a model retrained from scratch using only retain data classifies forget data based on their semantic similarity to the retain data. CoUn emulates this behavior by adjusting learned data representations through contrastive learning (CL) and supervised learning, applied exclusively to retain data. Specifically, CoUn (1) leverages semantic similarity between data samples to indirectly adjust forget representations using CL, and (2) maintains retain representations within their respective clusters through supervised learning. Extensive experiments across various datasets and model architectures show that CoUn consistently outperforms state-of-the-art MU baselines in unlearning effectiveness. Additionally, integrating our CL module into existing baselines empowers their unlearning effectiveness.

1 Introduction

Machine unlearning seeks to remove forget data’s influence while preserving retain knowledge; retraining only on retain data is exact but computationally inefficient. CoUn adjusts representations using contrastive and supervised learning on retain data, and the paper reports stronger unlearning than existing baselines.

1 Introduction

  • Retraining only on retain data is the gold-standard exact unlearning approach, but it is computationally inefficient.Approximate unlearning seeks efficiency while performing approximately like the Retrain model.Source: §1 · p. 1
  • Retrain classifies forget samples into retain clusters with the highest semantic similarity: class-wise forgotten trucks map mainly to automobile (69.32%), airplane (13.47%), and ship (12.60%).With random forgetting, 97.42% of forgotten truck samples are correctly classified as truck, with most others assigned to semantically similar classes.Source: §1 · p. 1
  • Label manipulation can reduce retain utility and requires forget data, while weight perturbations trade forget quality against utility.The paper notes that forget data may be unavailable in federated unlearning when clients leave after requesting unlearning.Source: §1 · p. 2
  • CoUn applies contrastive and supervised learning exclusively to retain data, indirectly shifting forget representations through semantic similarity while preserving retain clusters.The paper attributes the retain-cluster preservation role to supervised learning, which mitigates cluster collision.Source: §1 · p. 2
  • CoUn consistently outperforms state-of-the-art baselines across datasets and architectures, and adding its contrastive-learning module improves baseline unlearning effectiveness.The paper also presents theoretical analysis of higher forget-data misclassification while maintaining low retain-data misclassification.Source: §1 · p. 2

2 Related Work

Prior machine-unlearning methods manipulate forget-data labels or perturb model weights, each with trade-offs for retain utility and forget quality. CoUn instead uses semantic similarity and cluster collision rather than forcing forget samples into incorrect classes.

Label Manipulation

  • Label-manipulation methods alter forget-data outputs through targeted gradients or relabeling, but assigning semantically inconsistent or random classes can reduce retain utility.NegGrad uses gradient ascent [24] [25], while NegGrad+ combines retain-data fine-tuning with forget-data gradient ascent.Source: §2 · pp. 2–3

Model Weight Perturbation

  • Weight-perturbation methods target forget influence, but excessive perturbations erase essential knowledge and insufficient ones leave forget influence inadequately removed.Examples include ℓ1 regularization [10], Fisher-information damping [28], and layer-wise weight negation [8].Source: §2 · p. 3

Contrastive Learning

  • A prior contrastive-unlearning method pushes forget samples away from same-class retain samples and toward different-class samples, which the paper says harms utility.CoUn differs by using semantic similarity and cluster collision rather than forcing forget samples to be misclassified.Source: §2 · p. 3
  • CoUn seeks a higher misclassification rate for forget data while maintaining a low rate for retain data, aligning its aim with exact unlearning.Source: §2 · p. 3

3 Methodology

CoUn formulates approximate unlearning as adjusting retain and forget representations toward the behavior of a model retrained on retain data, while preserving retain knowledge. It combines contrastive and supervised learning on retain data and analyzes the resulting representation changes empirically and theoretically.

3.1 Machine Unlearning: Background

  • Machine unlearning transforms a model trained on all data to remove forget-data influence; exact unlearning retrains on retain data, while approximate methods initialize from the original model for greater efficiency.The forget-data ratio is |Du|/|D| × 100.Source: §3.1 · p. 3

3.2 CoUn Overview

  • Two augmented views of retain samples receive contrastive learning, while supervised cross-entropy maintains retain-cluster separation.Contrastive learning can create cluster collisions that affect retain utility; supervised learning mitigates this, while representation adjustments indirectly push forget samples toward semantically similar retain clusters.Source: §3.2 · pp. 3–5

3.3 Empirical Analysis

  • CoUn’s forget-sample predictions align more closely with Retrain than FT in both class-wise and random forgetting.Table 1 reports predictions for forgotten truck samples on CIFAR-10 with ResNet-18 and their differences from Retrain.Source: Table 1 · p. 5, §3.3 · p. 5

3.4 Theoretical Analysis

  • Under its stated assumptions, CoUn’s theoretical analysis bounds retain misclassification and concludes that forget misclassification is higher than retain misclassification.The analysis uses an upper bound involving alignment, R[ϵ], and establishes Rr[ϵ] < Ru[ϵ].Source: §3.4 · pp. 5–6

4 Experiments

Experiments compare CoUn with established MU baselines across datasets, architectures, and random-forgetting settings, then examine its CL module and hyperparameters. CoUn outperforms baselines, while ablations show that key training choices affect its performance.

4.1 Experimental Setup

  • Experiments cover CIFAR-10, CIFAR-100, and TinyImageNet with ResNet-18, VGG-16, and ViT; baselines include Retrain, FT, NegGrad+, ℓ1-sparse, SalUn, and NoT.Unlearning runs for 50 epochs, and reported results average 10 trials.Source: §4.1 · pp. 6–7

4.2 Results

  • 16.7% on CIFAR-10, 15.8% on CIFAR-100, and 6.3% on TinyImageNet are CoUn’s improvements over the best baseline with ResNet-18; it also outperforms baselines at 50% forgetting.For CIFAR-100, improvements are 53.4% with VGG-16 and 4.5% with ViT.Source: Table 2 · p. 7, §4.2 · p. 8, Table 3 · p. 8
  • Adding CoUn’s CL module improves baseline performance, with larger gains at a 50% forget ratio; gains persist under matched computation budgets.On CIFAR-10 with ResNet-18, NegGrad+, ℓ1-sparse, and NoT improve by 44.8%, 42.1%, and 43.3% at 10% forgetting, and by 50.0%, 87.7%, and 73.1% at 50%.Source: §4.2 · p. 8, Figure 4 · p. 9, Figure 5 · p. 10
  • CoUn consistently outperforms baselines across five sequential removal stages, and integrating its CL module further improves their unlearning effectiveness.Each stage removes 10% of random data, up to 50% total.Source: §4.2 · p. 9, Figure 6 · p. 10

4.3 Ablation Study

  • A high λ weakens supervised learning’s constraint on retain representations, increasing their susceptibility to cluster collisions and degrading utility and unlearning effectiveness.The scaling factor λ controls the relative contribution of the CE and CL losses.Source: §4.3 · p. 9, Figure 7 · p. 10
  • CL temperature τ performs best at 0.1; performance worsens when τ moves away from this value, despite generally improving as τ decreases.The study reports that excessively low temperatures can also harm performance.Source: §4.3 · p. 9, Figure 8 · p. 10
  • Using the same CHN transformation for CL and supervised learning performs best; stronger transformations can degrade performance, while overly simple ones may insufficiently adjust forget representations.Sharing an augmented image between the two learning objectives also reduces computation cost.Source: §4.3 · pp. 9–10, Figure 9 · p. 10
  • CoUn performs best with batch size 256, matching the batch size used for the Retrain model.Ablations use CIFAR-100, ResNet-18, and a 50% forget ratio.Source: §4.3 · pp. 9–10

5 Conclusion

CoUn uses contrastive learning and supervised learning on retain data to adjust representations according to semantic similarity while preserving retain clusters. The paper concludes that CoUn outperforms state-of-the-art unlearning baselines and that its contrastive-learning module improves existing baselines.

5 Conclusion

  • CoUn consistently outperforms state-of-the-art machine-unlearning baselines, and adding its contrastive-learning module improves existing baselines.Contrastive learning adjusts representations using semantic similarity, while supervised learning on retain data mitigates cluster collision for retain representations.Source: §5 · p. 10

A Related Work: Further Details

  • CoUn uses augmented views as positive examples in self-supervised contrastive learning, avoiding forget-data access and the need to force samples out of their original clusters.Unlike the related approach in [33], CoUn does not require forget and retain samples from the same cluster in each batch; [33] modifies its objective for class-wise unlearning.Source: A Related Work: Further Details · p. 14

B.1 Pseudo-Code

  • Algorithm 1 updates the Original model using a weighted sum of contrastive and cross-entropy losses computed from retain-data batches.It generates two transformed views per batch, computes the losses, and updates the model parameters.Source: §B.1 · p. 14

B.2 PyTorch Code

  • The PyTorch implementation extracts embeddings from the penultimate layer and combines supervised cross-entropy with contrastive loss.It uses retain data and two transformed views to compute the losses.Source: §B.2 · p. 15

C Further Details on Empirical Analysis

On CIFAR-10 with ResNet-18, CoUn’s forget representations are closer to semantically similar retain clusters and its distance statistics are closer to Retrain than those of other baselines. Prediction-level results likewise show more concentrated redirection toward semantically related clusters.

Representation Space of Original Model

Statistical Comparisons

  • CoUn’s forget representations are closer to semantically similar retain clusters, and its distance statistics are closer to Retrain than those of other baselines.For truck samples, automobile has the highest semantic similarity; smaller L2 distance indicates higher similarity.Source: Table 4 · p. 16, C Further Details on Empirical Analysis · p. 16

Prediction-Level Results

  • CoUn redirects forget samples toward semantically similar clusters more effectively than label-manipulation or weight-perturbation baselines.Its prediction distributions are closer to Retrain, while baseline misclassifications are less concentrated on semantically related classes.Source: C Further Details on Empirical Analysis · p. 16

D Proof of Lemma 1

The proof uses Lipschitz assumptions and probabilities of transformed inputs exceeding thresholds to relate retain and forget data. It notes that flat minima on retain data do not guarantee flatness on forget data.

D Proof of Lemma 1

  • Inequality (8) relies on an L-Lipschitz feature extractor, and inequality (9) follows by taking the supremum of both sides..Source: D Proof of Lemma 1 · p. 16
  • Training on retain data tends toward flat minima with low local Lipschitz behavior, but flatness is not guaranteed on forget data.The proof distinguishes Lipschitz constants evaluated on retain and forget data.Source: D Proof of Lemma 1 · pp. 16–17
  • The Original CIFAR-10 ResNet-18 model has perfect accuracy on both retain and forget samples because it was trained on both.Figure 11 depicts its representation space, with training samples clustered by class.Source: Figure 11 · p. 17
  • For random forgetting, IID sampling and epsilon/Lu <= epsilon/Lr imply Rr[epsilon] <= Ru[epsilon].The threshold for Ru[epsilon] is lower than the threshold for Rr[epsilon].Source: D Proof of Lemma 1 · p. 17

E.1 Model Architectures

The architecture details specify VGG-16 and ViT configurations, while the prediction-table captions identify experiments on CIFAR-10 with ResNet-18 and CIFAR-100 with VGG-16.

E.1 Model Architectures

  • The ViT uses 4 × 4 patches, 512-dimensional embeddings, a 1024-dimensional MLP hidden layer, 12 attention heads of size 64, and six transformer layers.Dropout and embedding dropout are both 0.1.Source: §E.1 · p. 17
  • Table 5 reports predictions for forgotten truck samples on CIFAR-10 with ResNet-18.The caption says it reports differences from Retrain.Source: Table 5 · p. 18
  • Table 6 reports predictions for forgotten man samples on CIFAR-100 with VGG-16.The caption says it reports differences from Retrain.Source: Table 6 · p. 18

E.2 Training and Unlearning Configurations

The paper specifies training and unlearning schedules, optimizer settings, and method-specific hyperparameter tuning. Experiments were implemented in PyTorch and run on NVIDIA Tesla V100 GPUs.

E.2 Training and Unlearning Configurations

  • Original models were trained for 182 epochs with batch size 256, SGD, momentum 0.9, and weight decay 5 × 10−4.The learning rate started at 0.1 and dropped by a factor of 10 at 50% and 75% of training; VGG-16 used a 75-epoch linear warmup.Source: §E.2 · p. 18
  • Unlearning runs lasted 50 epochs using SGD, a cosine annealing schedule, momentum 0.9, and weight decay 5 × 10−4.Learning rates were tuned per method within [0.01, 0.1], with a minimum learning rate of 1 × 10−4.Source: §E.2 · p. 18
  • Method-specific hyperparameters were tuned, including BadT temperature, SSD weight selection, and SCRUB maximization steps and loss weights.CoUn’s scaling constant λ was tuned within [0.1, 6] and its CL temperature τ within (0, 0.3].Source: §E.2 · pp. 18–19
  • Experiments were implemented with PyTorch and run on NVIDIA Tesla V100 GPUs.Source: §E.2 · p. 19

E.3 Data Augmentation

The section describes image augmentation operations and reports comparisons with additional baselines for random and class-wise forgetting. CoUn performs strongly in the reported random-forgetting comparison, while class-wise forgetting yields small gaps from retraining across baselines.

E.3 Data Augmentation

  • Image augmentation sequentially applies random cropping, horizontal flipping, and color normalization; some ablations add color jitter and grayscale.Cropping uses 32 × 32 outputs for CIFAR-10/100 and 64 × 64 for TinyImageNet; horizontal flip is applied with probability 0.5.Source: §E.3 · p. 19

F.1 Random Forgetting: Forget Ratio 10% (Additional Baselines)

  • CoUn consistently achieves superior performance to BadT, SSD, SCRUB, and CU under 10% random forgetting.BadT and SCRUB require more computational resources because they use two teacher models.Source: §F.1 · p. 19, Table 7 · p. 20

F.2 Class-wise Forgetting

  • Class-wise forgetting shows minimal gaps from Retrain across baselines, with CoUn achieving competitive results.The authors characterize class-wise forgetting as relatively easy and focus most experiments on the more challenging random-forgetting setting.Source: §F.2 · p. 19, Table 8 · p. 20

F.3 Integration of CoUn’s CL Module with Baselines: Detailed Results

Adding CoUn’s contrastive-learning module improves existing methods’ performance as measured by average gap under random forgetting. The transformation analysis finds that weaker CL transformations support more effective unlearning, and the broader-impacts discussion describes the work’s intended societal relevance.

F.4 Effect of CL Transformation on Representation Space

  • Weaker CL transformations lead to more effective unlearning because looser forget representations can overlap with semantically similar retain clusters.Figure 12 contrasts tighter forget-data clustering under CHJGN with looser clustering under CHN; the paper reports that tight clustering impedes unlearning.Source: §F.4 · p. 20, Figure 12 · p. 21

F.3 Integration of CoUn’s CL Module with Baselines: Detailed Results

G Broader Impacts

  • The paper frames machine unlearning as supporting users’ requests to remove their data and eliminating harmful or outdated information to enhance safety and fairness.The authors state that they foresee no direct negative societal impacts from this work.Source: G Broader Impacts · p. 22

H Limitations and Future Work

The authors identify limits in CoUn’s evaluation scope, its dependence on retain data, and the computational overhead of contrastive learning. They propose broader-domain evaluation and cost reduction as future directions.

H Limitations and Future Work

  • Evaluation is limited to vision-based classification on the relatively small CIFAR-10/100 and TinyImageNet datasets.The authors suggest exploring larger datasets such as ImageNet and other domains, including natural language processing.Source: H Limitations and Future Work · p. 22
  • CoUn relies heavily on access to retain data, leaving performance under partial retain-data access for future study.Source: H Limitations and Future Work · p. 23
  • Contrastive learning adds computational overhead from augmented data and extra forward passes, although CoUn outperforms baselines under matched computational budgets.The authors suggest exploring strategies to reduce this cost without sacrificing performance.Source: H Limitations and Future Work · p. 23

Related papers

Loading 2509.16391v3…