Source-linked AI summary
Entangled Representations Amplify Collateral Damage in Unlearning
Evžen Wybitul, Tim G. J. Rudner, Christian Schroeder de Witt
TL;DR
The paper addresses the untested intuition that representational entanglement makes targeted unlearning harder. It repurposes Selective Gradient Masking to train models with graded disentanglement and evaluates three unlearning methods, finding consistently better retain–forget trade-offs for more disentangled models. The results support entanglement as a cause of collateral damage, while the training intervention may also change other model properties.
Problem
Whether model-level representational entanglement makes targeted unlearning harder had not been directly tested in a controlled experiment.
Method
The study trains six 254M-parameter language models on English Wikipedia with graded biology–non-biology disentanglement using Selective Gradient Masking, then applies three unlearning methods to each model.
Results
More disentangled models consistently achieve better retain–forget trade-offs across three unlearning methods, with roughly 4× lower retain cost under WGA and RMU and 1.3× lower under WDR at fixed forgetting.
Takeaways & Limitations
The findings turn a long-standing interpretability intuition about entanglement into an empirical finding about unlearning trade-offs.
Takeaways & Limitations
The intervention cannot change entanglement alone because altering training may also change other model properties.
Abstract
from arXiv · showhide
A long-held intuition in interpretability research is that representational entanglement, the sharing of structure between knowledge domains in a neural network, makes unlearning harder. While the intuition is widespread, it has never been directly tested in a controlled experiment. We present a way to do so: by repurposing Selective Gradient Masking (SGTM), we train a suite of six 254M-parameter language models on English Wikipedia with graded levels of disentanglement between biology and non-biology knowledge. Applying three standard unlearning methods to every model in the suite, we find that more disentangled models consistently achieve better retain-forget trade-offs: at a fixed level of forgetting, the most disentangled models incur roughly $4\times$ lower retain cost under two of the three methods, and $1.3\times$ lower under the third. Because our intervention changes only the model, not the data or the unlearning algorithm, this is direct evidence that representational entanglement is one of the causes of collateral damage in unlearning, as interpretability researchers have long suspected. A similar design could be used to test other structural claims from interpretability.
1 INTRODUCTION
The paper directly tests whether representational entanglement makes targeted unlearning harder by varying entanglement in models while holding data and algorithms fixed. More disentangled models achieve better retain–forget trade-offs, though the intervention may also change other model properties.
- Entanglement is the degree to which retain and forget domains share representations, processing pathways, or parameters in a network.
- Prior work had not directly tested whether model-level entanglement affects unlearning because existing studies varied data composition or algorithms while holding the model fixed.
- The experiment trains six 254M-parameter language models on English Wikipedia with graded biology–non-biology disentanglement using Selective Gradient Masking, then evaluates three common unlearning methods.
- More disentangled models consistently achieve better retain–forget trade-offs across the tested unlearning methods.
- At a fixed forgetting level, the most disentangled models incur roughly 4× lower retain cost under WGA and RMU, and 1.3× lower under WDR.
- The intervention changes the training procedure and may affect properties beyond entanglement, but it rules out the earlier data and algorithm confounds.
2 EXPERIMENTAL SETUP
The experiment creates models with graded domain disentanglement, measures that separation, and evaluates three unlearning algorithms using retain–forget trade-offs. SGTM controls disentanglement during training while unlearning uses only forget and retain data.
- Training models with varying disentanglement: SGTM specializes parameters on the forget domain and thereby increases its disentanglement from retain; varying its training frequency creates different model conditions.The study activates SGTM for fractions p ∈ {0, 20, 40, 60, 80, 100} of training steps.
- Training models with varying disentanglement: The training data are English Wikipedia divided into forget, adjacent, and retain domains using article-level topic labels.Forget contains STEM.Biology articles; adjacent contains related topics; retain contains the remaining topics.
- Training models with varying disentanglement: Route-bio updates only the biology subnetwork, route-other trains without that subnetwork, and route-unchanged performs standard training.These interventions modify forward or backward passes according to each example’s routing category.
- Measuring disentanglement between domains: Disentanglement is measured between domain pairs using VES, MMD2, and SW2, with lower VES and higher MMD2 and SW2 indicating greater separation.Because the metrics have different scales, results are reported as log ratios relative to the p=0% model.
- Measuring the retain–forget trade-off: Three standard unlearning algorithms are applied independently to every model using forget and retain data, while adjacent data is reserved for evaluation.The algorithms receive no information about the models’ training procedure.
- Measuring the retain–forget trade-off: Retain–forget Pareto frontiers compare within-method trade-offs using changes in test loss relative to each model’s own pre-unlearning value.Each method is evaluated across multiple hyperparameter configurations and five random seeds per configuration.
3 RESULTS
Increasing SGTM training produces graded disentanglement, and more disentangled models achieve better retain–forget trade-offs across all three unlearning methods. The effect is strongest for WGA and RMU, while adjacent knowledge remains the hardest case.
- 3.1 MODELS TRAINED WITH MORE SGTM ARE MORE DISENTANGLED: All three metrics agree that increasing SGTM steps separates forget from retain and adjacent, with the largest change between p=40% and p=60%.Models cluster into less disentangled p ≤40% and more disentangled p ≥60% groups.
- 3.1 MODELS TRAINED WITH MORE SGTM ARE MORE DISENTANGLED: Retain–adjacent disentanglement also increases with p, but roughly three to four times less than forget–retain disentanglement.Forget–adjacent disentangles about two-thirds as much as forget–retain.
- 3.2 MORE DISENTANGLED MODELS ACHIEVE BETTER RETAIN–FORGET TRADE-OFFS: More disentangled models consistently achieve better retain–forget trade-offs across all three unlearning methods at ∆ℓforget = 0.4.The direction is consistent, although the effect size varies by method.
- 3.2 MORE DISENTANGLED MODELS ACHIEVE BETTER RETAIN–FORGET TRADE-OFFS: 4× lower retain cost: the most disentangled models outperform p=0% under WGA and RMU.For WGA, the frontiers form the same less- versus more-disentangled clusters identified in Section 3.1; RMU shows the same trend except for p=20%.
- 3.2 MORE DISENTANGLED MODELS ACHIEVE BETTER RETAIN–FORGET TRADE-OFFS: 1.3× lower retain cost: the most disentangled models outperform p=0% under WDR, although frontiers are close and standard errors overlap.The trend is in the same direction but weaker than for WGA and RMU.
- 3.2 MORE DISENTANGLED MODELS ACHIEVE BETTER RETAIN–FORGET TRADE-OFFS: Adjacent knowledge is hardest to protect because it receives greater spillover damage and is excluded from optimization.WGA and WDR preserve the same ordering seen for retain; RMU results are less clear.
4 CONCLUSION
The paper turns entanglement from an interpretability intuition into an empirical finding through a controlled model intervention. Its experimental recipe could also test other structural claims about neural networks.
- 4 CONCLUSION: Holding data and unlearning algorithms fixed while varying the model, the study finds better retain–forget trade-offs for more disentangled models across three algorithms.The design controls the data and algorithm confounds addressed in earlier work.
- 4 CONCLUSION: A recipe of controlling a representational property during training, verifying the control, and testing its predicted consequence could apply to other interpretability claims.
A TRAINING HYPERPARAMETERS
The cited training-hyperparameter section points readers to Table 2 for the full configuration.
- A TRAINING HYPERPARAMETERS: Table 2 contains the full training hyperparameters.
B UNLEARNING HYPERPARAMETERS
Unlearning configurations use AdamW without weight decay and repeat every configuration across five seeds.
- B UNLEARNING HYPERPARAMETERS: All unlearning methods use AdamW with no weight decay and five seeds, 42–46, per configuration.Full configurations are provided in Tables 3 to 5.
C DISENTANGLEMENT METRICS
The paper compares three metrics for quantifying disentanglement between domain representations, with metric-specific directions indicating greater disentanglement.
- VES compares within-group spread with between-group separation; lower VES indicates more disentanglement.
- MMD2 measures the squared distance between two distributions’ mean embeddings in a reproducing kernel Hilbert space.The paper uses an unbiased estimator with five Gaussian kernels.
- Higher MMD2 indicates more disentanglement between the compared distributions.
- SW2 2 is an optimal-transport distance averaging squared 2-Wasserstein distances across 512 fixed random projections.Higher SW2 2 indicates more disentanglement.
D UNLEARNING TRADE-OFFS ON ADJACENT KNOWLEDGE
Adjacent–forget Pareto frontiers reproduce the ordering seen for retain–forget trade-offs under WGA and WDR, but not under RMU.
- WGA: 4×: WGA’s most entangled model incurs roughly four times the adjacent cost of the disentangled cluster at ∆ℓforget = 0.4.The disentangled cluster consists of models with p ≥60%, while the most entangled model has p=0%.
- WDR: WDR reproduces the ordering across the model suite, although its frontiers are close together and standard errors overlap.The means remain correctly ordered.
- RMU: RMU’s adjacent–forget frontiers largely overlap, providing no clear ordering across models.The authors cannot resolve an ordering at this scale of adjacent loss rather than conclude that none exists.