Source-linked AI summary
Contrastive Unlearning: A Contrastive Approach to Machine Unlearning
Hong kyu Lee, Qiuchen Zhang, Carl Yang, Jian Lou, Li Xiong
TL;DR
Machine unlearning seeks to remove the influence of selected training samples without substantially reducing model performance, but existing approaches may neglect latent representations or use forgotten and remaining samples ineffectively. The paper contrasts their embeddings to move forgotten samples away from their class and toward other classes. It reports stronger unlearning effectiveness, model performance, and efficiency than state-of-the-art algorithms across class and sample unlearning.
Problem
Machine unlearning must remove selected training samples' influence while preserving overall model performance, and existing methods may neglect latent representations or use forgotten and remaining samples ineffectively.
Source: §1 · p. 1Method
Results
The paper reports that contrastive unlearning outperforms state-of-the-art algorithms in model performance, unlearning effectiveness, and efficiency across class and sample unlearning.
Source: §6 · p. 7Takeaways & Limitations
The approach is designed to remove forgotten-sample influence while keeping remaining-sample embeddings relatively intact, supporting model utility during unlearning.
Source: §1 · p. 2The formal setup assumes a cross-entropy-trained classifier and a disjoint test set sampled from a distribution analogous to the training set.
Source: §3.2 · p. 3
Abstract
from arXiv · showhide
Machine unlearning aims to eliminate the influence of a subset of training samples (i.e., unlearning samples) from a trained model. Effectively and efficiently removing the unlearning samples without negatively impacting the overall model performance is still challenging. In this paper, we propose a contrastive unlearning framework, leveraging the concept of representation learning for more effective unlearning. It removes the influence of unlearning samples by contrasting their embeddings against the remaining samples so that they are pushed away from their original classes and pulled toward other classes. By directly optimizing the representation space, it effectively removes the influence of unlearning samples while maintaining the representations learned from the remaining samples. Experiments on a variety of datasets and models on both class unlearning and sample unlearning showed that contrastive unlearning achieves the best unlearning effects and efficiency with the lowest performance loss compared with the state-of-the-art algorithms.
1 Introduction
The paper frames machine unlearning as removing training-sample influence while preserving model performance, and proposes contrastive unlearning to address limits in how prior methods use representations and remaining samples. Its experiments report strong unlearning effectiveness, utility, and computational efficiency across class and sample unlearning.
1 Introduction
- Unlearning effectiveness can be assessed through low accuracy on forgotten samples or low membership-prediction rates, alongside preserved accuracy on original tasks and computational efficiency.The motivation includes privacy requests such as the GDPR right to be forgotten, since models can retain training data in their parameters [Arpit et al., 2017].Source: §1 · p. 1
- Prior approaches often neglect latent representations or use unlearning and remaining samples separately or ineffectively, risking utility or unlearning effectiveness.For example, NegGrad [Golatkar et al., 2020] uses only unlearning samples, while Finetune [Golatkar et al., 2020] uses only remaining samples; SCRUB uses both but requires costly repeated passes with poor unlearning effectiveness.Source: §1 · p. 1
- The method pushes unlearning-sample embeddings away from same-class remaining samples and pulls them toward different-class samples, while keeping remaining embeddings relatively intact.It also uses an auxiliary classification loss on contrasted remaining samples to maintain model accuracy.Source: §1 · p. 2
- Figure 1 compares representation spaces for contrastive unlearning, gradient ascent, and finetuning.The accompanying explanation describes contrastive unlearning as making forgotten embeddings geometrically distant from every class while leaving remaining embeddings relatively intact.Source: Figure 1 · p. 2, §1 · p. 2
- The paper designs contrastive losses for both single-class and sample unlearning, with contrasting pairs tailored to unlearning rather than conventional contrastive learning.It also reports that classification loss can help keep remaining-sample embeddings in place and reciprocally improve unlearning effectiveness.Source: §1 · p. 2
- Contrastive unlearning is reported as the most effective approach while maintaining model utility and high computation efficiency across class and sample unlearning.Its membership-inference evaluation also reports the lowest member prediction rate on unlearning samples compared with all baselines.Source: §1 · p. 2
2 Related Works
Prior work includes exact methods that seek complete removal and approximate methods that reduce influence to a negligible level, with empirical and certified variants. The paper positions its approximate approach as covering both sample and class unlearning and evaluates it against both method types.
2 Related Works
- Exact unlearning seeks completeness, while approximate unlearning seeks to reduce the forgotten data's influence to a negligible level.Certified unlearning adds a quantifiable approximation guarantee [Gupta et al., 2021; Guo et al., 2020].Source: §2 · p. 3
- Sharding-based exact methods such as SISA retrain affected partitions, but require partitioned training and can be costly, with performance dependent on the partition strategy.ARCANE partitions by class.Source: §2 · p. 3
- Certified methods face scaling and computation challenges for neural networks, as guarantees often require convex loss and methods may compute expensive Hessian or Fisher matrices.LCODEC [Mehta et al., 2022] selectively generates Hessians to alleviate computation cost.Source: §2 · p. 3
- Empirical approximate methods include noisy-gradient updates for class removal, decision-boundary changes, specialized architecture, and teacher-student transfer excluding the forgotten class.These examples are UNSIR, boundary unlearning, ERM-KTP, and SCRUB.Source: §2 · p. 3
- The proposed approximate unlearning method covers both sample and class unlearning and is evaluated against empirical and certified approaches.The paper reports empirical superiority, without specifying individual benchmark values in this passage.Source: §2 · p. 3
3 Preliminaries and Problem Definition
The preliminaries describe supervised contrastive learning and formalize the model and two unlearning settings. Sample unlearning targets forgotten samples, while single-class unlearning targets all samples of a class, with retraining used as the reference for desired behavior.
3.1 Contrastive Learning
- Supervised contrastive learning uses class labels to pull same-class embeddings together and push different-class embeddings apart; the paper adapts a SupCon-inspired loss for unlearning.Unlike SimCLR's self-supervised setup, SupCon contrasts samples based on their classes.Source: §3.1 · p. 3
3.2 Problem Definition
- The setup defines a classifier as an encoder that produces embeddings followed by a classification head, trained with cross-entropy on a training set and evaluated on a disjoint, analogous test distribution.The encoder is parameterized by θ.Source: §3.2 · p. 3
- For sample unlearning, the unlearned model should match a model retrained without the selected samples on both those samples and test samples.For single-class unlearning, accuracy should approach zero on the removed class's training and test samples while matching retraining on remaining classes.Source: §3.2 · p. 3
4 Contrastive Unlearning
Contrastive unlearning treats training influence as geometric structure in representation space, contrasting unlearning samples with remaining samples to alter the former while preserving the latter. Its algorithm combines this loss with classification loss on remaining samples and uses task-specific evaluation to decide when to stop.
4 Contrastive Unlearning
- Unlearning samples are pushed away from same-class remaining samples and pulled toward different-class remaining samples, moving their embeddings toward the middle of the remaining samples.The approach treats class-based geometric similarity in trained representations as information to modify for unlearning.Source: §4 · p. 4
- The unlearning loss contrasts each unlearning anchor with remaining samples, minimizing similarity for same-class pairs and maximizing it for different-class pairs.The paper describes this as the opposite of conventional contrastive learning.Source: §4 · p. 4
- For single-class unlearning, no remaining samples belong to the target class, so there are no positive samples to push away.The single-class setup therefore has an empty positive set.Source: §4 · p. 4
- Classification loss on remaining samples restores their embeddings because unlearning optimization also changes them, risking model performance and unlearning effectiveness.The final objective combines unlearning loss and remaining-sample classification loss; the single-class loss denominator is changed to damp excessive pulling when no positives exist.Source: §4 · p. 4
- The algorithm repeatedly samples remaining batches for each unlearning batch, with up to four repetitions used to limit computational overhead while maintaining stable unlearning.It terminates using task-specific evaluation criteria: class-unlearning accuracy falls below a threshold tied to random-guess accuracy, while sample-unlearning accuracy falls below test-sample accuracy.Source: §4 · pp. 4–5
5 Experiments
Experiments compare contrastive unlearning with retraining and other methods on class and sample removal, measuring unlearning effectiveness, retained-task accuracy, and efficiency. The reported results show strong unlearning with relatively small performance loss, alongside fast execution.
5.1 Experiment Setup
- Experiments use CIFAR-10 and SVHN with ResNet-18/34/50/101; class removal targets class 5 and sample removal 500 random samples, evaluating accuracy, unlearning, runtime, and MIA.Source: §5.1 · p. 5
5.2 Results on Single Class Unlearning
- For single-class unlearning, contrastive unlearning reaches zero unlearning-class accuracy while preserving remaining-class accuracy on CIFAR-10 and consistently removes influence with negligible loss on SVHN.Source: Table 1 · p. 6, Table 2 · p. 6, §5.2 · p. 6
- Contrastive unlearning is reported fastest for class and sample unlearning; it uses one iteration for class removal and fewer than 20 sample-unlearning iterations.Source: Table 3 · p. 6, §5.2 · p. 6, Table 7 · p. 7
5.3 Results on Sample Unlearning
- On CIFAR-10, contrastive unlearning most closely matches retraining; on SVHN, it consistently matches retrained unlearning accuracy with relatively small test-accuracy loss.Source: Table 4 · p. 7, Table 5 · p. 7, §5.3 · p. 7
- MIA shows a larger member-prediction-rate gap between unlearning and retained samples for contrastive unlearning than other methods; the attack reports around 60% on unlearning samples due to a technical limitation.Source: Table 6 · p. 7, §5.3 · p. 7
6 Conclusion
The paper uses contrastive unlearning to reconfigure embedding-space geometry by contrasting unlearning and remaining samples. Experiments show it outperforms state-of-the-art methods in model performance, unlearning effectiveness, and efficiency.
6 Conclusion
- Contrastive unlearning reconfigures embedding-space geometry by contrasting unlearning samples with remaining samples.The paper proposes examining the method in different model architectures and settings such as graph and correlated-sequence unlearning.Source: §6 · p. 7
- Experiments show contrastive unlearning outperforms state-of-the-art algorithms in model performance, unlearning effectiveness, and efficiency.Source: §6 · p. 7
- Future work will examine different model architectures and settings such as graph and correlated-sequence unlearning.Source: §6 · p. 7