PaperlayerView exact revision on arXiv

Source-linked AI summary

RA$^2$: Retain-Anchored Attraction Preserves Forget-Adjacent Utility in LLM Unlearning

Zinan Ling, Yang Xiao, Ruimeng Ye, Sijia Liu, Bo Hui

arXiv:2610.19385v1cs.LG

TL;DR

Unlearning can suppress target behavior and preserve clean retain performance while still degrading benign reasoning near the forgotten domain. RA2 redirects forget-token states toward local retain anchors while preserving higher-layer retain compatibility. Across WMDP, ablations, and MUSE, the paper reports improved target suppression versus nearby benign utility compared with separative updates.

  • Problem

    Standard unlearning criteria can miss loss of benign reasoning when harmless prompts lie near the forgotten region.

    Source: §1 · p. 1

  • Method

    RA2 redirects forget-conditioned states toward local retain-anchor neighborhoods and preserves higher-layer retain compatibility.

    Source: §4 · pp. 4–5

  • Results

    Across WMDP, ablations, and MUSE, RA2 improves target suppression versus nearby benign utility compared with separative updates.

    Source: §6 · p. 10

  • Takeaways & Limitations

    Unlearning should control not only whether displaced forget representations leave removed computation, but also where they land afterward.

    Source: §6 · p. 10

    RA2 is modestly slower than RMU because it maintains and queries the anchor field, though its end-to-end cost remains much lower than RMU-LAT.

    Source: §5.6 · p. 10

Abstract

from arXiv · show

Large Language Model (LLM) unlearning is often evaluated primarily by target suppression and clean retain utility. These metrics can miss a local failure mode: a model may perform well on standard retain prompts, yet degrade on benign reasoning when prompts lie near the forgotten domain. This phenomenon is termed forget-adjacent utility loss. A fundamental question is: where should forget representations go after they leave the removed behavior? Successful unlearning depends not only on whether forget-conditioned representations depart from the harmful computation, but also on the direction in which they are redirected afterward. We propose RA$^2$ (Retain-Anchored Attraction), a latent-space unlearning method that redirects forget-token states toward nearby retain-supported representations while preserving higher-layer behavior on retain data. On WMDP and MUSE, RA$^2$ yields higher forget-adjacent utility than strong unlearning baselines at similar broad-utility levels. These results show that the migration direction of forget representations materially affects benign behavior near the forget boundary. Our code is available at https://github.com/ZacLing/retain_anchored_attraction.

1 Introduction

Standard unlearning metrics can miss benign reasoning failures when prompts approach the forgotten domain. RA2 redirects forget-conditioned states toward local retain support, and the aligned WMDP comparison reports stronger forget-adjacent utility than the cited baselines at similar general-utility levels.

1 Introduction

  • Forget-adjacent utility measures whether benign target behavior survives when prompts move toward the removed domain through nearby cues, entities, citations, or mixed-query context.The paper evaluates this regime with perturbed MMLU, WMDP-prefixed MMLU, and a cue-swapping Bio-MNCP variant.Source: §1 · p. 1
  • Clean retain performance can conceal local brittleness: forget-conditioned states may leave the harmful computation yet fail to follow a representation trajectory later layers can use for benign reasoning.A harmless prompt can still activate forget-adjacent features despite success on clean retain prompts.Source: §1 · p. 2
  • RA2 redirects each forget-token state toward a local neighborhood supported by retain-token representations rather than only pushing it away from removed behavior.The intended direction keeps forget-conditioned states near the retain representation manifold while changing their propagation through the model.Source: §1 · p. 2
  • 56.6 MMLU and 55.7 MMLU Perturbed, 43.0 WMDP→MMLU, and 56.2 harmful-cue Bio-MNCP are reported for RA2 in the aligned WMDP comparison.The corresponding RMU scores are 56.4, 51.5, 29.4, and 50.0; RMU-LAT scores are 57.5, 46.5, 24.6, and 42.2, respectively.Source: §1 · pp. 1–2
  • A qualitative biology example keeps the chlorophyll question unchanged while adding a bio-adjacent preface; RMU and RMU-LAT lose the target under that prefix, while RA2 retains it.The example illustrates the benign-query failure mode at aligned forget strength.Source: §1 · p. 2, Table 1 · p. 2

2 Related Works

Prior work spans benchmark suites and robustness-style tests, while this paper isolates a distinct gap: strong global retain utility may coexist with degradation near forget-related context. RA2 remains a representation-level intervention but redirects states toward local retain support.

2 Related Works

  • Benchmark conclusions can vary with metric and prompting setup, and the paper distinguishes its focus on near-forget retain utility from benchmark coverage of other unlearning contracts.TOFU, WMDP, MUSE, and OpenUnlearning cover fictitious-author forgetting, hazardous-knowledge removal, corpus removal, and unified cross-benchmark evaluation, respectively.Source: §2 · p. 3
  • Robustness-style evaluations already examine mixed prompts, overlap effects, retain-set dependence, context-aware behavior, and input-transformation robustness.The paper positions its forget-adjacent evaluations within this line of work.Source: §2 · p. 3
  • RA2 belongs to representation-level unlearning but changes its target geometry by attracting forget-side states toward local retain support rather than a largely separative direction.The passage contrasts this approach with RMU and RMU-LAT and notes a conceptual relation to ROME, while distinguishing unlearning from factual rewriting.Source: §2 · p. 3

3 Forget-adjacent utility evaluation

Forget-adjacent utility evaluates benign targets under context moved toward the removed domain, separating ordinary retain performance from local stability. The section defines the target and transformations, then describes three evaluation settings and how their results indicate neighborhood-sensitive utility loss.

3 Forget-adjacent utility evaluation

  • Forget-adjacent utility measures retain-side behavior after a prompt moves toward the removed domain through nearby context, while requiring harmful completions to be suppressed and harmless targets preserved.A model can retain high ordinary utility yet have low utility under these transformations.Source: §3 · pp. 3–4
  • The setup distinguishes a benign retain distribution from the removed domain and treats a retain example as an input-target pair whose target answer is harmless.A benign target neither conveys designated-for-removal knowledge nor assists its inference or retrieval; admissible context transformations preserve the question and retain status.Source: §3 · p. 3
  • The evaluation family injects or prefixes forget-adjacent context while keeping the benign target answer unchanged.The three settings move from local prompt perturbation to hazardous-prefix conditioning and then a matched cue-level intervention; Table 2 frames benign-target preservation alongside suppression of designated forget targets.Source: Table 2 · p. 3, §3 · p. 4
  • MMLU Perturbed tests whether forget-related content inserted into a retain question disrupts its original harmless answer.It follows RNA and scores only the original answer under the perturbation.Source: §3 · p. 4
  • WMDP→MMLU tests the trailing benign MMLU answer after a WMDP example is prefixed, while Bio-MNCP holds the target fixed and changes its cue.The prefix setting follows SEPS; Bio-MNCP contrasts harmless and harmful-adjacent cues and reports harmful-side benign accuracy and a centered-logit drift gap.Source: §3 · p. 4
  • A high retain utility paired with falling utility under one or more forget-adjacent estimates indicates preserved broad performance but fragile benign behavior near forget-related content.The utility gap is small when retain behavior is stable under tested transformations and large when neighborhood-sensitive utility loss is present.Source: §3 · p. 4

4 RA2: Retain-Anchored Attraction via Local Redirection

RA2 redirects forget-conditioned states toward local retain-supported representations while preserving higher-layer retain compatibility. It constructs local anchors and uses them to control where forget states move after leaving removed computations.

4 RA2: Retain-Anchored Attraction via Local Redirection

  • RA2 edits a selected parameter subset to redirect forget-token states at a reshape layer while preserving higher-layer retain compatibility with the frozen model.Figure 2 distinguishes frozen-retain, aligned-retain, and reshape layers alongside retain states, anchors, forget states, and redirected states.Source: §4 · pp. 4–5, Figure 2 · p. 5

4.1 A Local Criterion for Retain-Supported Redirection

  • The local retain-patch criterion distinguishes leaving the harmful computation from reaching a destination supported by nearby retain representations.A destination is retain-supported up to locality error δz when its distance to the local patch is at most δz; this separates forgetting strength from destination control.Source: §4.1 · p. 5

4.2 Constructing the Retain Anchor Field

  • RA2 builds its anchor field from retain-token states, initializing anchors with Farthest-Point Sampling after the retain buffer reaches its budget.The field is updated outside the model parameters, so gradients act only on θS; retain-only warm-up and continuing retain batches stabilize and refresh it.Source: §4.2 · pp. 5–6

4.3 Anchor-Weighted Redirection and Retain Consistency

  • The forget-side objective attracts each forget state toward its local anchor destination, using assignment weights and anchors as stop-gradient targets.This differs from repulsive or random-target latent unlearning, which does not direct states toward a retain-anchor neighborhood.Source: §4.3 · p. 6
  • A higher-layer retain-consistency term matches updated retain states to the frozen model, and the final objective combines it with anchor attraction over θS.The local-control analysis states that exact anchor locality contracts retain-normal distance, while nonzero locality error contracts the bound toward a δz-tube; Table 3 reports an aligned WMDP profile averaged over ten seed runs.Source: §4.3 · p. 6, Table 3 · p. 7

5 Experiments

The experiments compare unlearning methods on WMDP, test how forget-side update direction affects nearby benign utility, and examine MUSE-Books knowledge removal. Across WMDP settings, the results distinguish retain-anchored attraction from updates that only separate forget representations.

5.1 WMDP Main Results

  • On Zephyr, RA2, RMU, and RMU-LAT remain in a similar broad-utility band, but RA2 better preserves forget-adjacent utility; on Qwen3-8B, it is more stable on clean and adjacent metrics.GA, GradDiff, and JensUn show broad-utility or truthfulness collapse at this forget scale.Source: §5.1 · p. 7
  • On WMDP→MMLU, RA2's sentence log-likelihood was −11.00 versus the original model's −10.99, and gold-answer ROUGE-L was 20.99 versus RMU's 18.13.RA2 achieved the highest gold-answer ROUGE-L among evaluated unlearning methods on both tasks.Source: §5.1 · p. 8, Table 4 · p. 8

5.2 Direction, Not Just Strength, Controls Forget-Adjacent Utility

  • Repulsive directions improve direct forgetting but sharply lower MMLU-Perturbed and WMDP→MMLU, while attraction is the only region that keeps local metrics high across useful strengths.Signed-strength sweeps show that repel-like branches retain the local collapse across strengths, so strength alone does not explain the difference.Source: §5.2 · pp. 8–9, Figure 3 · p. 8

5.3 Mechanistic Ablations

  • Changing only the forget-side force sign preserves the contrast: repulsion still forgets but loses benign WMDP→MMLU accuracy, while removing attraction leaves the forget side barely moved.The retain-consistency and anchor ablations indicate that lower-layer redirection needs a stable retain-compatible pathway above it.Source: §5.3 · p. 9, Table 5 · p. 9

5.4 MUSE-Books Knowledge Removal

  • Table 6 reports MUSE-Books knowledge-removal results.The supplied passage identifies the table's scope but does not state its results.Source: Table 6 · p. 9

5.5 RA2 Behaves More Like a Retrained Model

  • Among unlearning methods on 3000 WMDP→MMLU questions, RA2's centroid is closest to the original–retrain reference neighborhood.The filtered retrain and unfiltered original serve as references; large departures on benign forget-adjacent questions indicate shortcut-like behavior introduced by the unlearning update.Source: §5.5 · p. 9

5.6 Discussion

  • The experiments support destination control: once forget states move enough to suppress removed behavior, their landing region matters for forget-adjacent utility.GA/NPO/RMU-style objectives do not specify a retain-supported destination, whereas RA2 uses local anchors.Source: §5.6 · p. 10
  • RA2 is modestly slower than RMU because it maintains and queries the anchor field, but its end-to-end cost is much lower than RMU-LAT's.The passage describes the wall-clock comparison in Table 7.Source: §5.6 · p. 10, Table 7 · p. 10

6 Conclusion

The conclusion frames unlearning as a destination-control problem: displaced forget-token states should be redirected toward local retain anchors while preserving higher-layer retain compatibility. The appendices provide notation, implementation procedure, and supporting geometry.

6 Conclusion

  • RA2 redirects forget-token states toward local retain anchors, and experiments across WMDP, ablations, and MUSE show improved target suppression versus nearby benign utility compared with separative updates.The conclusion argues that methods should make the landing region of displaced representations explicit.Source: §6 · p. 10

Appendix

  • The appendices cover notation, RA2 optimization and implementation scope, local geometric analysis, evaluation scoring and prompts, experimental details, and related work.Appendix E includes the MUSE-News split.Source: Appendix · p. 16

A Notation Summary

  • The notation summary defines symbols for evaluation targets, the RA2 objective and method, and the local geometric analysis.It includes forget-adjacent utility and utility gap notation, along with anchor assignments and attraction and retain-consistency losses.Source: A Notation Summary · pp. 17–18

B RA2 Optimization Procedure and Scope

  • RA2 first initializes and warms up its anchor field using retain data, then refreshes anchors, computes forget attraction at ℓr, and applies higher-layer retain consistency at ℓt during joint training.The procedure selects a trainable parameter subset and freezes the rest.Source: B RA2 Optimization Procedure and Scope · pp. 18–19

C Local Geometry of Unlearning Objectives

The section compares how unlearning objectives move forget-token representations relative to local retain support. GA, NPO, and RMU can separate states from that support, while RA2 selects a retain-anchor destination and provides local control under anchor-locality conditions.

Local retain patch and retain-normal distance

  • From a state on the retain patch, any update with a nonzero retain-normal component moves it away from local retain support to leading nontrivial order.The analysis considers one forget token-position, a small update, and a fixed local neighborhood.Source: C Local Geometry of Unlearning… · p. 19

GA and NPO

  • GA is locally separative when prediction-loss ascent has a nonzero retain-normal component; NPO scales the leading GA displacement without redirecting its normal direction.A retain loss may protect clean retain examples, but does not choose a retain-supported destination for the forget-token displacement.Source: C Local Geometry of Unlearning… · p. 20

RMU

  • RMU pulls forget representations toward a random high-norm target, whose retain-normal component is nonzero almost surely under the stated nondegeneracy condition.With an isotropic preconditioner, expected normal energy is proportional to the normal dimension, dh − dim Tz.Source: C Local Geometry of Unlearning… · p. 20

Local direction claim

  • GA, NPO, and RMU can produce direct forgetting without specifying a retain-supported destination, whereas RA2 redirects forget states toward local retain anchors.GA and NPO use prediction-loss directions; RMU uses a random latent target. RA2’s destination is selected from retain anchors before local preconditioning.Source: C Local Geometry of Unlearning… · pp. 20–21

Preconditioned RA2 direction

  • The parameter-induced RA2 direction is preconditioned toward a retain-anchor destination, and its normal-distance bound depends on locality and tangent-to-normal leakage.When leakage vanishes and the local preconditioner is compatible with the retain-patch geometry, normal distance contracts up to the locality floor.Source: C Local Geometry of Unlearning… · p. 21

RA2 local control

  • RA2’s fixed-destination representation-space step contracts retain-normal distance toward a δz floor, with exact locality contracting it toward zero.The repeated-step recurrence requires the same fixed-neighborhood and anchor-locality assumptions at each step; the main text relies on the one-step local statement.Source: C Local Geometry of Unlearning… · pp. 20–21

Attract-versus-repel interpretation

  • Reversing attraction into repulsion can still separate forget states, but no longer specifies a retain-supported destination.This local interpretation is consistent with signed-strength results in which stronger repulsion can improve direct forgetting and clean utility while damaging forget-adjacent behavior.Source: C Local Geometry of Unlearning… · p. 21

D Scoring Details and Prompt Examples for Forget-Adjacent Evaluations

This appendix defines three forget-adjacent evaluation routes that keep the scored target benign while moving prompt context toward the removed domain. It also gives scoring details and examples, including suppression checks under benign context and a generated-answer illustration.

D Scoring Details and Prompt Examples for Forget-Adjacent Evaluations

  • Across the three settings, the target answer remains benign while the prompt approaches the forget domain through different routes.The appendix specifies scoring definitions and representative prompts for these evaluations.Source: D Scoring Details and Prompt Examples… · p. 22
  • MMLU Perturbed locally edits a harmless MMLU question while keeping its correct answer unchanged, and scores the model’s answer on the perturbed prompt.The perturbation changes one local answer option.Source: D Scoring Details and Prompt Examples… · p. 22
  • WMDP→MMLU moves a benign MMLU target toward the forget domain with a full hazardous WMDP prefix, but scores only the trailing MMLU answer.The mixed-query examples concatenate the hazardous prefix and benign question.Source: D Scoring Details and Prompt Examples… · pp. 22–23
  • Bio-MNCP holds the benign target and answer fixed while swapping a short neighboring cue; its paired scoring reads answer accuracy alongside harmful-side centered-logit drift.Positive cue gaps indicate extra answer-logit movement under the harmful-adjacent cue.Source: D Scoring Details and Prompt Examples… · p. 23
  • The qualitative bio-prefix example adds bio-adjacent context to an unchanged elementary-biology question, illustrating the local failure pattern behind forget-adjacent utility.Table 10 contains output excerpts for the example.Source: D Scoring Details and Prompt Examples… · p. 24, Table 10 · p. 25

D.1 Suppression under benign context

  • With a benign MMLU prefix, RA2’s WMDP suppression scores remain close to its direct controls: 29.6% on Bio versus 29.3%, and 25.3% on Cyber versus 27.3%.Table 11 compares direct WMDP targets with the same targets after a benign MMLU prefix; the scored target is WMDP, for which lower is preferred.Source: §D.1 · p. 25, Table 11 · p. 25

D.2 A benign biological target under adjacent context

  • Under biosecurity context, RA2 answers the benign blood-cell question with repetition and truncation, while RMU produces an unrelated response.The reverse-target evaluation separately measures suppression when the scored target concerns forget knowledge.Source: §D.2 · p. 25

E Additional Experimental Details

The additional experiments report implementation and runtime details alongside robustness, sequential unlearning, baseline, and MUSE comparisons. They also delimit the tested backbone scale and note that other adjacency forms may produce different neighborhoods.

E Additional Experimental Details

  • The forget-adjacent evaluation family covers local perturbations, hazardous-prefix conditioning, and paired cue swaps, while other adjacency forms such as longer dialogues or retrieval contexts may induce different neighborhoods.WMDP uses WikiText retain data and aligned bio/cyber forget corpora; MUSE-Books and MUSE-News use benchmark-provided target and retrain models based on Llama-2-7b.Source: E Additional Experimental Details · p. 25
  • RA2 adds training overhead relative to RMU but remains close in end-to-end train-plus-evaluate time because evaluation dominates, and is substantially faster than RMU-LAT.Runtime is measured on one NVIDIA A100 SXM4 80GB under the reported batch-size and sequence-length settings, which differ across models.Source: E Additional Experimental Details · p. 26, Table 13 · p. 26
  • Both 8-bit and NF4 quantization remain close to the unquantized RA2 reference across WMDP, MMLU, and MMLU-Perturbed.The reported robustness applies to the tested post-training quantization settings.Source: E Additional Experimental Details · p. 27, Table 15 · p. 27
  • Across four sequential WMDP-Bio stages, WMDP-Bio falls from 54.60 after the first partial update to 27.42 after the fourth, while MMLU remains above 53%.Each stage processes a disjoint forget-data subset using 150 training steps.Source: E Additional Experimental Details · p. 27, Table 16 · p. 27
  • The signed-strength and planar-angle sweeps examine forget-side direction choices, with the angle sweep spanning the canonical attract and repel endpoints.The reported angle-sweep metrics are WMDP average, MMLU, MMLU-Perturbed, and WMDP→MMLU.Source: E Additional Experimental Details · pp. 27–28, Table 17 · p. 27, Table 18 · p. 28
  • The experiments focus on 7B–8B backbones; performance and anchor coverage at 14B, 32B, or 72B remain unevaluated.The authors note that larger models may expose more complex representation structure and that scaling behavior remains an empirical question.Source: E Additional Experimental Details · p. 29

E.1 Additional baseline comparisons

  • On Zephyr-7B-β, RA2 has higher forget-adjacent accuracy than the other compared methods near its 29.4% WMDP accuracy; KUDA has 28.2% WMDP accuracy and 22.9% MMLU-P and 22.7% WMDP→MMLU.On Bio-MNCP, KUDA scores 40.6%, compared with 50.0% for RMU, 42.2% for RMU-LAT, and 56.3% for RA2.Source: §E.1 · p. 29, Table 20 · p. 29

E.2 Additional MUSE comparisons

  • On MUSE-Books, RA2 reduces verbatim memorization to 12.89 while retaining a knowledge score of 56.57; on News, its corresponding scores are 31.62 and 48.97.The comparison uses official target checkpoints and reports verbatim removal, forget-side knowledge, retained knowledge, and privacy leakage.Source: E Additional Experimental Details · p. 28, Table 19 · p. 28, §E.2 · p. 29

E.3 Answer likelihood and generation

The supplementary evaluations compare answer likelihood and generation, combined-query response preservation, and combinations of retain-anchor routing with other objectives. Together, they report complementary outcomes across these evaluation settings.

E.3 Answer likelihood and generation

  • RA2 obtains the highest gold-answer ROUGE-L among unlearned models on both tasks, and its WMDP-prefix generation is closer to the original model than RMU’s, with reference ROUGE-L of 43.73 versus 27.96.MMLU-P reference ROUGE-L is similar for these methods: 49.00 versus 49.16; GA and GradDiff produce degenerate outputs throughout this evaluation.Source: §E.3 · p. 30, Table 23 · p. 31

E.4 BLUR combined queries

  • On BLUR Retain vs. Combined, RA2 preserves 70.0% of the original model’s combined-query score, compared with 54.4% for RMU.This measure compares preservation of the original retain-only answer under a combined query.Source: §E.4 · p. 31, Table 24 · p. 31

E.5 Combining retain anchors with other objectives

  • Adding retain-anchor routing and retain consistency improves all three forget-adjacent metrics and MMLU for GA, NPO, and SimNPO on Zephyr-7B-β.MMLU-P and WMDP→MMLU gains are 24.10 and 18.84 points for GA, 3.29 and 3.79 for NPO, and 7.47 and 10.40 for SimNPO.Source: §E.5 · p. 31, Table 25 · p. 32

E.6 Retain-data volume

With example presentations held constant, shrinking the retain subset did not produce a monotone decline in MMLU; adjacent metrics also varied across subsets.

E.6 Retain-data volume

  • At 5% volume, 25 distinct examples were repeatedly presented, and MMLU was 56.03.The experiment used nested subsets of 500 WikiText examples with the same number of presentations in each condition.Source: §E.6 · p. 32, Table 26 · p. 32
  • MMLU ranged from 52.85 to 56.56 across subset sizes, with no monotone decline as the subset shrank.The table reports sensitivity to retain-data volume with a fixed number of example presentations.Source: §E.6 · p. 32, Table 26 · p. 32
  • Adjacent metrics varied across subsets, including increased WMDP→MMLU accuracy for several smaller subsets.The passage does not report a uniform direction of change for these adjacent metrics.Source: §E.6 · p. 32

E.7 Anchor construction and sensitivity

Anchor budget and assignment choices affect the balance between local utility and suppression, while initialization and routing variants have different effects on evaluation metrics.

E.7 Anchor construction and sensitivity

  • Reducing the anchor budget from 5000 to 512 lowered WMDP→MMLU from 41.77 to 36.95 while leaving clean MMLU nearly unchanged.Table 27 varies anchor budget, neighborhood size, and assignment distance; values are percentages.Source: §E.7 · p. 32, Table 27 · p. 32
  • A single anchor gave 42.13 on WMDP→MMLU and a higher WMDP accuracy of 31.09.The authors present these results as illustrating the trade-off between local utility and suppression when configuring the anchor field.Source: §E.7 · p. 32, Table 27 · p. 32
  • Changing FPS initialization or softmax routing to hard assignment changed evaluation metrics by at most 1.5 points.Replacing nearest-neighbor routing with a weighted average over all anchors raised WMDP accuracy from 29.4 to 47.1, supporting local anchor selection.Source: §E.7 · p. 32

E.8 Training time and memory

RA2's runtime and memory are reported for its main configuration and a matched comparison; anchor storage and search are used during training, while inference uses edited parameters.

E.8 Training time and memory

  • In the main configuration, RA2 took 308.87 seconds, compared with 103.10 for RMU and 2261.33 for RMU-LAT.Table 28 reports training runtime and peak GPU memory on a single NVIDIA A100 SXM4 80GB GPU; main configurations use each method's training schedule and sequence length.Source: §E.8 · p. 33, Table 28 · p. 33
  • With batch size 4, sequence length 512, and 20 outer batches matched, RA2 runtime was 1.23 times RMU's and peak memory increased by 1.72 GiB.The comparison fixes these settings across methods.Source: §E.8 · p. 33, Table 28 · p. 33
  • Anchor storage and search are used during training; inference uses the edited model parameters.The reported resource measurements concern training runtime and peak GPU memory.Source: §E.8 · p. 33

F Extended Related Work

This section situates RA2 among benchmark, method, robustness, and editing work, highlighting its retain-supported destination for forget-side updates. It also summarizes the work’s stated limitations and theoretical contribution, experimental and societal-impact disclosures, and use of existing public resources without human-subject or crowdsourcing research.

F Extended Related Work

  • Benchmarks span controlled fictitious-author facts, hazardous knowledge, multidimensional utility and privacy criteria, and shared-fact settings, while analyses warn results depend on benchmark construction and metrics.TOFU [37], WMDP [28], MUSE [47], and DUSK [22] exemplify these settings.Source: F Extended Related Work · pp. 33–34
  • Unlearning methods range from output- and objective-level updates to latent interventions; RA2 differs from separation, feature-selective perturbation, or constrained deviation by specifying a retain-supported destination.Examples include NPO, ULD [24], SPUL [1], RMU [28], CLReg [51], SRMU, and KUDA [11].Source: F Extended Related Work · p. 34
  • Robustness studies show clean retain utility can miss side effects; RA2 tests benign, unchanged tasks with prompts moved near the forget boundary rather than into another harmful domain.Related evaluations also examine perturbed retain prompts, mixed queries, forget–retain overlap, alternate target forms, and adjacent versus remote retain samples.Source: F Extended Related Work · p. 34
  • RA2’s unlearning contract suppresses a target behavior or knowledge region while preserving retain capability, directing forget-side updates toward retain-supported local anchors near the boundary.Unlike editing, which replaces facts, RA2 uses a representation-space intervention under an unlearning contract.Source: F Extended Related Work · p. 35

NeurIPS Paper Checklist

  • The checklist acknowledges limitations in anchor coverage and evaluation scope, including the absence of evaluations on larger 14B/32B models and additional LLM families due to resource constraints.Source: NeurIPS Paper Checklist · p. 36
  • The checklist says the paper formalizes local unlearning geometry, contrasting separative GA-, NPO-, and RMU-style updates with RA2’s redirection toward retain-supported destinations.Source: NeurIPS Paper Checklist · p. 36
  • The checklist reports reproducibility details and open-access assets, ten-seed means and standard deviations, and partial compute information without a comprehensive total-compute estimate.The repository is described as anonymous; compute details include GPU type, memory, GPU count, and approximate training and inference time.Source: NeurIPS Paper Checklist · pp. 37–39
  • The checklist identifies safer removal of undesirable knowledge as a potential benefit and selective information suppression in deployed models as a possible misuse.Source: NeurIPS Paper Checklist · p. 39
  • The paper introduces no new datasets, benchmarks, or standalone assets and uses existing public resources while crediting sources and following their terms and licenses.The checklist marks new-asset documentation as not applicable.Source: NeurIPS Paper Checklist · p. 40
  • The work involves neither crowdsourcing nor research with human subjects, so participant instructions and IRB approval are not applicable.The checklist marks both crowdsourcing and IRB-related items as N/A.Source: NeurIPS Paper Checklist · pp. 40–41
  • LLMs are evaluated target models, not an undisclosed or non-standard tool used for the core methodology, analysis, or results.Source: NeurIPS Paper Checklist · p. 41

Related papers

Loading 2610.19385v1…