PaperlayerView exact revision on arXiv

Source-linked AI summary

Unmerge: Efficient Machine Unlearning via Task Arithmetic

Haoran Tang, Andrew Tan, Rajiv Khanna

arXiv:2609.38895v1cs.LGcs.AI

TL;DR

Approximate unlearning seeks to remove a forget set’s influence without full retraining, while existing approaches face fragile tuning, entangled knowledge, and limited insight into layerwise behavior. UNMERGE learns and subtracts a low-rank forget component from a merged task vector, using an objective tied to activation-space leakage and damage. On class-level forgetting, it improves the retain/forget tradeoff over gradient-based and task-vector baselines, while matching retraining closely under membership-inference attacks.

  • Problem

    Approximate unlearning methods face fragile tuning, difficulty with entangled forget and retain knowledge, and limited insight into where unlearning occurs within a network.

    Source: §1 · p. 2

  • Method

    UNMERGE learns a forget component from the merged task vector and subtracts it, using layerwise activation subspaces and a low-rank objective.

    Source: §1 · p. 2

  • Results

    On class-level forgetting, UNMERGE improves the retain/forget tradeoff over gradient-based and task-vector baselines, avoids gradient-ascent collapse, and keeps membership-inference exposure at the level of retraining.

    Source: §1 · p. 2, §5 · p. 10

  • Takeaways & Limitations

    Per-layer bases, overlaps, and task-vector magnitudes provide diagnostics of unlearning difficulty and delimit the method’s scope to concept-level requests.

    Source: §1 · p. 2, §5 · p. 10

    The task-vector input-subspace property has an O(η^2) residual under finetuning assumptions and is used heuristically for from-scratch training.

    Source: §3.3 · p. 5

Abstract

from arXiv · show

Approximate machine unlearning seeks to remove the influence of a forget set from a trained model without full retraining. Existing gradient-based methods require data-dependent hyperparameter search, struggle when forget and retain knowledge are entangled, and offer little insight into where unlearning actually happens inside the network. We recast unlearning through the lens of task arithmetic: if finetuning produces a merged task vector $\tau_m$ that combines learning on forget and retain sets, unlearning is the inverse operation that subtracts a learned forget component $\tau_F$ to recover the retain task vector $\tau_R$. The forget signal is concentrated: at every layer, forget activations lie in a subspace spanned by a handful of dominant directions, so we factorize $\tau_F$ in a low-rank forget basis, which is faithful up to a small tail-eigenvalue residual and limits how far the correction can perturb retain. We then optimize three intuitive goals (match the merged vector inside the forget span, suppress leakage into the retain span, and bound the correction size) that provably bound forget leakage and retain damage in activation space. The resulting algorithm, Unmerge, is fast and powerful: on class-level unlearning with ResNet-50 on CIFAR-100 and Tiny ImageNet, it improves Tug-of-War by up to ~24% over a baseline of comparable runtime and by up to ~18% over stronger baselines that run ~5x slower, keeps membership-inference exposure at the level of retraining, and shrinks the feature-distribution gap to the retrained model, where relabeling methods leave forget features cleanly separable. Further studies show that Unmerge also applies to ViT-S/16 and scales to Llama-3.2-3B. The per-layer basis geometry that drives the algorithm also serves as a layerwise diagnostic for when and where unlearning becomes structurally hard.

1 INTRODUCTION

Approximate unlearning methods face fragile tuning, entangled forget and retain knowledge, and opaque layerwise behavior. UNMERGE reframes unlearning as subtracting a learned forget component from a merged task vector, with subspace diagnostics and reported improvements over baseline methods.

1 INTRODUCTION

  • Existing methods share fragility, entanglement blindness, and opacity: small tuning changes can trigger collapse, shared features invite retain damage, and scalar metrics obscure layerwise behavior.The four dominant families—gradient ascent, relabeling, teacher-student distillation, and saliency- or Fisher-based methods—all reshape weights through forget-loss optimization and trade forgetting against retention through hyperparameter tuning.Source: §1 · pp. 1–2
  • Per-layer bases, overlaps, and task-vector magnitudes expose where unlearning occurs and can predict requests beyond subspace editing’s reach.The introduction identifies late layers as high-leverage and notes that requests without a separable activation subspace, such as random instance subsets, cannot be unmerged.Source: §1 · p. 2
  • UNMERGE treats unlearning as the inverse of task merging: it learns a forget component from the merged task vector and subtracts it to recover a retain-only vector in activation space.Its per-layer objective is convex and solved in a projected space without gradients on data; the remaining scale can usually be set from forget-set accuracy alone.Source: §1 · p. 2
  • UNMERGE improves the retain/forget tradeoff over gradient-based and task-vector baselines on ResNet-50 and ViT-S, avoids gradient-ascent collapse, and recovers output behavior closer to retraining.Ablations attribute the gain to the retain-leakage term, and a preliminary study applies the method to a 3B-parameter language model.Source: §1 · p. 2

2 RELATED WORK

Related work spans post-hoc unlearning, task arithmetic, and representation editing. UNMERGE learns a forget component from a finetuned model’s own activation bases, rather than requiring a separately trained forget task vector or imposing a hard projection.

2 RELATED WORK

  • Task-vector removal methods subtract an explicit forget vector, whereas UNMERGE constructs and learns one from data-free bases on the finetuned model itself.Recent alternatives obtain vectors by finetuning on the forget set, combining vectors under a sign-consensus mask, or constraining finetuning to permit exact unlearning at scale.Source: §2 · p. 3
  • Model-merging work studies combining task vectors while limiting cross-interference; UNMERGE reverses that direction by decomposing an already-merged vector and subtracting its forget component.The paper describes the relationship to Cheng et al. (2025) as an analogy between objectives, not a formal duality.Source: §2 · p. 3
  • Post-hoc approximate methods include influence- or Fisher-based scrubbing, gradient ascent, saliency updates, dampening, boundary shifting, relabeling, and distillation; UNMERGE uses no training on forget or retain data at unlearning time.Its optimization is data-free, although data are used to collect activation second-moment statistics.Source: §2 · p. 3
  • UNMERGE differs from gradient-free projection methods by editing the task vector with an explicit forgetting-versus-leakage objective, supervised bases, and diagnostics rather than a hard orthogonal projection.It retains natural overlap between forget and retain bases and softly penalizes retain leakage.Source: §2 · p. 3

3 UNLEARNING VIA UNMERGING

UNMERGE represents a finetuned model as a merged task vector and learns a low-rank forget component to subtract. Its layerwise objective targets forget behavior while limiting retain leakage and correction size, with activation-space bounds explaining those design terms.

3.1 PRELIMINARIES

  • Joint finetuning on retain and forget data is treated as producing a merged vector, but the retain vector is defined as the remainder rather than as an additive vector from a separately trained expert.The objective selects a forget component that makes the remainder approximately orthogonal to the forget span; “unmerging” names this objective, not the training provenance.Source: §3.1 · pp. 3–4
  • The learned forget vector is parameterized in layerwise activation bases, using either an unsupervised high-energy basis or supervised discriminative concept-activation vectors.Retain bases are constructed from retain inputs as well.Source: §3.1 · p. 4

3.2 UNMERGING

  • The per-layer objective is a strictly convex quadratic in the projected coefficients, balancing fit within the forget span, retain leakage, and correction size.The retain term acts along retain-basis directions, and the low-rank form uses k-dimensional projected parameters per layer.Source: §3.2 · p. 4
  • A low-rank forget basis is faithful up to a tail-eigenvalue residual and limits perturbations outside the forget subspace, where larger rank can hurt retain accuracy.The forget eigenspectrum decays sharply in the considered layers.Source: §3.2 · p. 4
  • UNMERGE builds activation bases in a forward pass, takes a fixed budget of 100 Adam steps, then applies the correction, optionally only to deep layers.The edit is optimization-time data-free: no gradient is taken on a training sample; early stopping acts as an implicit regularizer.Source: §3.2 · p. 4

3.3 THEORETICAL MOTIVATION

  • The low-rank approximation is motivated by the input-subspace property, under which task-vector rows lie approximately in the span of layer inputs and approximation error decays with tail eigenvalues.The stated O(η^2) residual is a finetuning idealization used heuristically for from-scratch training.Source: §3.3 · p. 5
  • The objective’s terms control activation-space forget leakage, dominant retain damage, and tail-direction retain damage; the bounds apply to the budgeted update, with α trading forgetting strength against retain damage.Term 2 controls the dominant retain-damage component, while Term 3 protects directions beyond the top-k retain subspace.Source: §3.3 · p. 5
  • The minimizer preserves forget-private directions more strongly and shrinks directions shared with retain according to their subspace overlap.The overlap is represented by cosines of principal angles; Term 2 acts where the spans meet, matching the reported ablation mechanism.Source: §3.3 · p. 5

4 EXPERIMENTS

Experiments compare UNMERGE with retraining and other unlearning methods on image-classification benchmarks, using performance, feature-distribution, and privacy measures. UNMERGE leads the tug-of-war comparisons, while layerwise diagnostics, feature distances, and instance-level tests characterize where it works and where its scope is limited.

4.1 SETUP

  • The evaluation uses ResNet-50 on CIFAR-100 and TinyImageNet, with class-level forget requests and both pretrained and zero initializations.It also compares against retraining, gradient-based, task-vector, and projection baselines; evaluation includes performance, feature entanglement, and membership privacy.Source: §4.1 · p. 6

4.2 MAIN UNLEARNING RESULTS

  • UNMERGE variants lead tug-of-war on both datasets and both initializations; CAV-CLASS leads on TinyImageNet for both initializations at runtime comparable to NegGrad+.The comparisons cover CIFAR-100 superclasses and TinyImageNet Dogs & Cats.Source: §4.2 · p. 6, Table 2 · p. 7, Table 1 · p. 7
  • UNMERGE-CAV-KMEANS leads every axis, with ToW 0.9975 versus SalUn’s 0.9949 at 14× lower runtime, and MIA gap and entanglement 1.8× and 7× closer to retraining.Preliminary TOFU experiments found that UNMERGE on Llama-3.2-3B-Instruct reached forget quality 0.13 and model utility 0.630 without training; the retain-only model had utility 0.650.Source: §4.2 · p. 7
  • UNMERGE is up to ~24% better than comparable-runtime NegGrad+ and ~18% better than Random Label and SalUn, which run about 5× slower.Runtime is 8–105 seconds on one H100, with no retain-loop fine-tuning.Source: §4.2 · p. 7

4.3 WHERE DOES UNLEARNING HAPPEN?

  • Skipping ResNet-50’s first 35 of 40 convolutional layers retains the forget drop while improving retain/test accuracy by 2–5 points.For class- or concept-level requests, forget signals are separable where semantics are encoded; low-level attributes such as color or texture call for earlier layers.Source: §4.3 · p. 7
  • Forget–retain overlap declines with depth, from 0.8–1.0 in layers 1–2 to 0.4–0.6 at layer 4, while PCA is more entangled than CAV bases.The passage reports that bases are nearly initialization-invariant; task-vector magnitude, rather than basis entanglement, changes between initialization regimes.Source: §4.3 · p. 8

4.4 FEATURE-LEVEL ENTANGLEMENT

  • UNMERGE’s CAV variants are closest to retraining in feature-distribution gap, whereas Random Label and SalUn leave forget representations cleanly separable from retain.On W0 the CAV variants are within ~0.10 relative gap; on WPT they are at 0.25–0.32.Source: §4.4 · pp. 8–9, Table 3 · p. 9

4.5 MEMBERSHIP PRIVACY, SCOPE AND FIDELITY

  • At class level, UNMERGE’s membership-inference exposure remains close to retraining; random-subset requests remain unresolved, and UNMERGE and NegGrad+ are near-inert on them.For random subsets, relabeling methods score by marking the forget set, with attack accuracy 0.84–0.88 versus 0.63–0.68 under retraining; the reported separability score is 0.17.Source: §4.5 · p. 9

5 CONCLUSION

The conclusion presents UNMERGE as task-vector subtraction with activation-space bounds on forget leakage and retain damage. It also frames the approach as a soft subspace trade-off, documents its scope and evaluation context, and relates it to prior subspace and representation-editing methods.

5 CONCLUSION

  • UNMERGE learns a low-rank forget component of the merged task vector and subtracts it without gradient updates on training data.Its objective bounds forget leakage and retain damage in activation space.Source: §5 · pp. 9–10

AI USE STATEMENT

  • The paper reports using Claude for debugging, experimental-design feedback, utility code, result interpretation, and literature search, with human review of assisted work.Source: AI USE STATEMENT · p. 11

ETHICS STATEMENT

  • The work uses public image-classification benchmarks and pretrained weights, involves no human subjects or personal data, and cautions that approximate unlearning is not a deletion guarantee.Source: ETHICS STATEMENT · p. 11

REPRODUCIBILITY STATEMENT

  • The authors report complete algorithm pseudocode, assumptions and proofs, experiment configurations, seed variance, and plans to release code and evaluation materials upon acceptance.Experiments ran on a single NVIDIA H100 GPU.Source: REPRODUCIBILITY STATEMENT · p. 11

A ADDITIONAL RELATED WORK

  • UNMERGE softly penalizes retain leakage while preserving natural overlap between forget and retain bases, and lifts CAV-style directions into weight-level task-vector subtraction.The related-work discussion contrasts this soft trade-off with hard orthogonal projection and representation-level erasure.Source: A ADDITIONAL RELATED WORK · pp. 17–18

B.1 ALGORITHM AND IMPLEMENTATION DETAILS

UNMERGE builds layerwise retain and forget activation bases, then optimizes a low-rank forget correction in projected space. Its analysis connects the correction to activation-space leakage and damage, while its implementation uses forward-pass statistics and keeps optimization inexpensive.

B.1 ALGORITHM AND IMPLEMENTATION DETAILS

  • UNMERGE collects layer inputs and builds a PCA retain basis plus a PCA or CAV forget basis, with CAV using clusters or forget-class labels.CAV uses ridge probes against retain activations and orthogonalizes their directions; KMeans needs no class labels, while Class uses one direction per forget class.Source: §B.1 · p. 18
  • Projecting the objective into the forget basis reduces each layer’s optimization to a strictly convex quadratic in B, with an exact minimizer and a fixed 100-step Adam budget.The fixed-budget iterate generally does not reach the exact minimizer; early stopping can regularize against over-shrinking the forget correction on high-overlap bases.Source: §B.1 · pp. 18–19, §B.2 · p. 19

F end for return WUL

  • For CAV forget bases that are not eigenbases, the dominant forget-leakage component remains controlled, but the tail constant depends on variance outside the selected span.The captured-variance diagnostic reports this residual; bounds are per layer, use unedited-network bases, and do not track downstream activation shifts from upstream edits.Source: F end for return WUL · pp. 21–22

B.2 PROOFS AND SCOPE OF THE ANALYSIS

  • The analysis assumes accumulated layer updates lie in the span of training inputs, with an O(η^2) input-drift term treated as a small-learning-rate idealization for finetuning.For from-scratch training, the authors use the property heuristically and check low-rank structure empirically.Source: F end for return WUL · pp. 21–22

B.3 REQUIREMENTS, RUNTIME AND COST

  • UNMERGE reads forget and retain data in forward passes to form statistics, takes no sample gradients, and can reuse cached retain statistics across forget requests on the same checkpoint.PCA uses one forward pass, while CAV uses two over the retain set.Source: §B.3 · p. 22
  • At E=10, measured UNMERGE runtime ratios versus Random Label and SalUn are 4–23× on ResNet-50 and ViT-S/16; ratios vary with time spent on basis construction.The baselines’ timers include test-set evaluation after each epoch, whereas UNMERGE’s timer stops before evaluation.Source: §B.3 · p. 22

C EXPERIMENTAL DETAILS

The experiments use class-level forget requests on CIFAR-100 and TinyImageNet, with ResNet-50 runs from pretrained and from-scratch initializations and ViT-S/16 used to probe architectural generality. The section also reports the per-layer task-vector magnitude pattern across initializations and specifies training, baseline, and configuration-transfer protocols.

C EXPERIMENTAL DETAILS

  • Forget tasks are CIFAR-100 superclasses of five classes and ImageNet semantic groups of ten classes; within each task, F is the union of chosen classes’ training images and R = S \ F.Source: C EXPERIMENTAL DETAILS · pp. 22–23
  • Models use SGD with momentum 0.9 and batch size 128; ResNet-50 schedules differ by dataset and initialization, while ViT-S/16 is finetuned for 10 epochs at learning rate 0.01.Each retrained reference uses its corresponding finetuning schedule on the retain set only.Source: C EXPERIMENTAL DETAILS · p. 23
  • Gradient baselines run for E=10 epochs, task-vector baselines receive a Trees grid search before transfer, and UNMERGE settings transfer from Trees or Dogs & Cats with limited changes.UNMERGE always uses 100 Adam steps at learning rate 10−3, never tuned; transfer changes at most α, with a few W0 cells also changing skip-layers.Source: C EXPERIMENTAL DETAILS · p. 23
  • At layer4 and the classifier, the merged-task-vector magnitude is ∼1.5–2× larger on W0 than WPT, while layers1–2 are slightly larger on WPT.The Figure 2 curves coincide across variants because τm is basis-independent; the shaded band spans three finetuning seeds.Source: Figure 2 · p. 24, C EXPERIMENTAL DETAILS · p. 24

D.1 PER-LAYER DIAGNOSTICS AND HYPERPARAMETERS

The diagnostics turn basis overlap, layerwise task-vector magnitude, and forget-signal rank into practical hyperparameter choices. Ablations show that retaining the leakage penalty is central to the objective’s gains.

D.1 PER-LAYER DIAGNOSTICS AND HYPERPARAMETERS

  • PCA reaches ToW 0.9851 on WPT but 0.9230 on W0, while CAV-Class reaches 0.9909 on W0 by allowing a larger α with less retain overlap.The shared basis entanglement leaves deep task-vector magnitude as the deciding factor across initialization regimes.Source: §D.1 · p. 24
  • CAV rank elbows lie near 16 on ResNet-50 and 16–32 on ViT-S, while PCA prefers ranks 64–128 to cover more forget signal.Beyond the CAV elbow, retain accuracy degrades by 0.5–2% at fixed α; CAV-Class uses one direction per forget class.Source: §D.1 · pp. 24–25
  • Skip-layers at the layer3-to-layer4 overlap elbow preserve the forget drop while improving retain/test accuracy by 2–5 points over all-layer updates.CAV uses cutoffs 35/40, while PCA uses 45–50 because its variance-driven directions overlap more with retain.Source: §D.1 · pp. 24–25
  • The retain-leakage penalty is most binding for overlapping PCA bases, whereas CAV-Class is insensitive to λ across an order of magnitude.Both regularizers transfer across superclasses without retuning.Source: §D.1 · p. 25
  • The α rule selecting the smallest value with at most 1% forget accuracy matches the ToW-optimal grid point in five of six Trees configurations, losing 0.003 ToW in the sixth.It fails to reach the target for CAV-KMeans on TinyImageNet/W0 and overshoots for PCA on Arthropods, so the hardest regime still needs a reference.Source: §D.1 · p. 25

D.2 OBJECTIVE TERMS

  • With Term 1 alone, PCA collapses retain accuracy to 71–94%; adding Terms 1 and 2 puts every cell within 0.02 of the full objective.Term 3 alone or with Term 1 closes at most part of the gap, while its effect at the reported λ is within 0.02 everywhere.Source: §D.2 · p. 25

D.3 THE BUDGETED SOLVER AND THE EXACT MINIMIZER

The budgeted iterate admits a thresholded closed-form model that closely reproduces its behavior, while the exact minimizer is an alternative for CAV bases but remains budget-dependent for PCA. The initialization ablation finds zero initialization a safe fallback, but using a different pretrained checkpoint is catastrophic.

D.3 THE BUDGETED SOLVER AND THE EXACT MINIMIZER

  • Bδ is closest to the Adam iterate in every cell and reproduces reported ToW within 0.005 in five of six cells (0.023 in the sixth).Its relative distance is 0.01–0.32, versus 0.05–0.96 for B⋆ and 0.74–1.85 for B0.Source: §D.3 · p. 26, Table 8 · p. 26
  • For CAV bases, the residual is nearly empty and ToW shifts by at most 0.02 over δ ∈ [0.05, 0.1]; for PCA, the residual carries the edit and depends on the step budget.On W0, the PCA minimizer is a no-op at reported λ, while 0.36% of coefficients do the forgetting.Source: §D.3 · p. 26
  • At iterate-tuned hyperparameters, the exact minimizer reproduces CAV cells on WPT, under-forgets CAV cells on W0, and is a no-op for both PCA cells.When tuned for itself, it prefers γ=0 and larger λ (10–100) with correspondingly larger α.Source: §D.3 · pp. 26–27
  • At a fixed budget, the budgeted iterate is the only estimator that works in all six cells; the exact minimizer is a budget-free equivalent for CAV variants, but PCA variants depend on budget.The subsection's ablations use one seed on Trees, and validating an explicit-form default across seeds and datasets is left for future work.Source: §D.3 · p. 27

D.4 Winit ABLATION

  • With zero initialization and retuned α, the method matches CAV cells and trails only on PCA; substituting a different pretrained checkpoint is catastrophic for every variant.The authors identify zero initialization as the safe fallback and release the exact WPT checkpoint used as input.Source: §D.4 · p. 27, Table 9 · p. 27

E.1 DETAILED PER-TASK RESULTS

Across the reported superclass and initialization cells, at least one UNMERGE variant leads every baseline on ToW, with variant rankings differing by setting. Three-seed results preserve most method orderings, and ViT-S/16 results extend the reported performance pattern to an attention-based backbone.

E.1 DETAILED PER-TASK RESULTS

  • In every superclass/init cell, at least one UNMERGE variant leads every baseline on ToW; on CIFAR-100 at least two of three lead every gradient-based baseline.On TinyImageNet/W0, only CAV-CLASS leads, while PCA and KMeans trail SalUn; CAV-CLASS is strongest on W0, with the WPT leader varying by superclass.Source: §E.1 · pp. 27–28, Table 10 · p. 28, Table 11 · p. 28

E.2 SEED VARIANCE

  • Across three seeds, every ordering between UNMERGE variants and SalUn/Random Label is preserved in the means; on W0, UNMERGE standard deviations (0.008–0.021) are 1.7–4.5× smaller than Random Label and SalUn (0.036–0.037).On WPT, SalUn is most stable (0.003) but trails UNMERGE-PCA and CAV-KMEANS by 1.0–1.3 points.Source: §E.2 · p. 28

E.3 UNLEARNING WITH VIT

  • On ViT-S/16, UNMERGE-CAV-KMEANS leads every baseline on every reported axis: ToW 0.9975, runtime 34s, MIA gap 0.061, and feature-entanglement gap 0.101.The method edits linear projections in the attention and MLP blocks, with the best cells skipping the first 32 PCA or 40 CAV-KMeans layers; leaving biases at finetuned values barely changes the two cells.Source: §E.3 · pp. 28–29

E.4 PRELIMINARY STUDY ON A LANGUAGE MODEL

In a preliminary Llama-3.2-3B study, a retain-deflated PCA basis yields selective unlearning at utility comparable to the retrained reference, while basis choice strongly affects selectivity. TinyImageNet feature comparisons show UNMERGE variants near retrained feature distances, with other metrics exposing differences that entanglement alone misses.

E.4 PRELIMINARY STUDY ON A LANGUAGE MODEL

  • UNMERGE reaches forget quality 0.131 and utility 0.630, versus RMU's 0.094 and 0.633, while requiring one activation-statistics pass and no training.The edited model is not separated from WRT by the Kolmogorov–Smirnov test at the 5% level; baselines train for 70–290s.Source: §E.4 · p. 30, Table 16 · p. 31
  • Deflating the top 1024 retain directions restores selectivity for PCA, whereas plain PCA bases overlap the retain basis by 0.87–0.91 and lower answer probabilities on forget and retain in lockstep.CAV bases overlap the retain basis by less than 0.13 but capture only 1–8% of forget-token variance, so retain degrades before forgetting completes.Source: §E.4 · p. 30
  • The language-model study uses one split, one model, and one seed; UNMERGE is lightly tuned on a proxy using WRT, whereas baselines are untuned.The deflated-direction edit uses α=64, far above the vision range, and extrapolates along forget-private input directions rather than removing a finetuning delta.Source: §E.4 · p. 30

E.5 FEATURE-LEVEL ENTANGLEMENT ON TINYIMAGENET

  • On TinyImageNet/W0, CAV-CLASS has mean feature gap 0.062, while the projection baseline's smaller 0.036 mean gap coexists with 18% forget accuracy and a 7-point test loss.The projection's MIA gap is 0.157 versus 0.058 for CAV-CLASS; on WPT, CAV-CLASS's gap of 0.082 is slightly tighter than the projection's 0.099.Source: §E.5 · p. 31, Table 17 · p. 31
  • MMD2 and Sliced-W2 rank the best UNMERGE variant ahead of every gradient and task-vector baseline on both initializations, while W2-Bures is mean-shift dominated when D≫N.The feature-entanglement axis alone does not reveal the projection baseline's forget accuracy and test-accuracy costs.Source: §E.5 · p. 31

E.6 MEMBERSHIP INFERENCE: DETAILED RESULTS

On class-level tests, UNMERGE variants show tighter MIA gaps on TinyImageNet WPT and competitive gaps on CIFAR-100 Trees, but attack-protocol choices affect interpretation.

E.6 MEMBERSHIP INFERENCE: DETAILED RESULTS

  • The class-level tables average absolute MIA gaps across signals, while attack accuracy averages retain true-positive and forget true-negative rates.SVC attackers use an unbalanced shadow set, and correctness is counted twice; these details affect interpretation of the reported gaps.Source: §E.6 · p. 31
  • On CIFAR-100 Trees W0, Kodge et al. (2024) has the smallest mean gap (0.017), while UNMERGE-CAV-KMEANS (0.026) and UNMERGE-CAV-CLASS (0.029) beat SalUn (0.047).On WPT, Kodge et al. (2024) leads again at 0.028; NegGrad+’s 0.067 is close to UNMERGE-CAV-CLASS at 0.070 but costs 2.8 retain and 6.2 test points.Source: Table 18 · p. 32, §E.6 · p. 32
  • UNMERGE-PCA’s 0.009 MIA gap on TinyImageNet WPT is 2–5× tighter than saliency baselines and 13–29× tighter than task-vector baselines and Kodge et al. (2024).The CAV variants report gaps of 0.018/0.020; on W0, however, Random Label and SalUn appear to lead because they under-unlearn.Source: §E.6 · p. 32

E.7 INSTANCE-LEVEL FORGETTING: DETAILS

Instance-level subsets expose a boundary: UNMERGE treats coherent subsets as partial concept removal, while random subsets offer no separable target. Output-space comparisons are sharper than weight-space validation in pretrained runs.

E.7 INSTANCE-LEVEL FORGETTING: DETAILS

  • On coherent instance subsets, UNMERGE lowers the retained half of the class along with the forget half, and no tested method reproduces retraining on that protocol.The Trees test accuracy falls 18–26 points below retraining, while the other 95 classes remain within 2.2 test points; aggregate retain accuracy hides the loss because the retained half is only 2.6% of R.Source: §E.7 · pp. 33–34, Table 21 · p. 34
  • On random subsets, forget and retain accuracy move together; relabeling methods reach ToW up to 0.96 while making forget samples more identifiable than under retraining.Their relative entanglement gap is 16–82, compared with ≤0.7 for UNMERGE; the retrained model itself barely forgets random samples.Source: Table 20 · p. 33, §E.7 · p. 34
  • Untraining reverses training on the forget samples, whereas concept unlearning removes the represented concept; these targets diverge when that concept remains in R.The section describes random subsets as an extreme case: retraining retains test-like forget accuracy, leaving little beyond per-sample memorization to remove.Source: §E.7 · p. 34
  • The separability score distinguishes class-level from random CIFAR-100 regimes within a dataset, but does not transfer directly across datasets.For TinyImageNet, class-level scores are 0.07–0.10 despite successful unlearning; normalizing against a same-dataset random-subset score addresses basis and feature-structure dependencies at the cost of an extra forward pass.Source: §E.7 · p. 34, Table 22 · p. 35

E.8 DECOMPOSITION VALIDATION: DETAILS

  • On WPT, UNMERGE variants stay at the unedited weight-space level (0.750–0.754), while UNMERGE-CAV-KMEANS has the smallest forget-set divergence: 2.89 on W0 and 5.29 on WPT.Weight-space comparisons on W0 are uninformative because independent retrains converge to unrelated basins; CAV-CLASS at α=2.5 on WPT matches forget accuracy but has divergence 7.15.Source: Table 23 · p. 35, §E.8 · p. 35, §E.7 · p. 35

F LIMITATIONS AND FUTURE WORK

UNMERGE is scoped to concept-level requests with separable activation subspaces and requires data for forward passes. Its reported studies have narrow evaluation breadth, and its fidelity claims are limited to outputs and feature distributions.

F LIMITATIONS AND FUTURE WORK

  • UNMERGE targets concept-level requests with separable activation subspaces; on coherent instance subsets it lowers retained and forget halves alike, while random subsets lack such a subspace.Its Phase-1 separability ratio requires a same-size random-subset control for calibration.Source: F LIMITATIONS AND FUTURE WORK · p. 36
  • The method is optimization-time data-free, not data-free: it needs forward passes over F ∪ R, and CAV-CLASS needs forget labels.The WPT regime also requires the same pretraining checkpoint; substituting another is catastrophic, while Winit=0 is a safe fallback.Source: F LIMITATIONS AND FUTURE WORK · p. 36
  • UNMERGE claims fidelity to retraining in outputs and feature distribution, not that class information becomes undecodable.A linear probe still recovers forget classes from every tested approximate method, including UNMERGE.Source: F LIMITATIONS AND FUTURE WORK · p. 36
  • Instance-level, task-vector-validation, and seed-variance studies use one CIFAR-100 superclass; language-model evidence is preliminary.The cost model is validated on ResNet-50 and ViT-S/16.Source: F LIMITATIONS AND FUTURE WORK · p. 36
Loading 2609.38895v1…