Source-linked AI summary
TULIP: Targeted LLM Unlearning at Layers Identified Per-input
Yejin Kim, William F. Shen, Seokwon Jung, Daeun Park, Seong Joon Oh
TL;DR
A single fixed intervention layer may not fit a forget set because the layer where a target forms varies by input. TULIP locates each input’s formation–readout boundary and outperforms output- and representation-level baselines across benchmarks, with reported robustness to attacks. The authors note that the logit lens is not established as the optimal boundary estimator.
Problem
Existing representation-level unlearning methods apply one fixed intervention layer across the forget set, although the formation–readout boundary varies across inputs.
Source: §1 · p. 1Method
TULIP uses the logit lens to locate each input’s formation–readout boundary and removes hidden-state alignment with the target’s unembedding vector at that layer.
Source: §1 · p. 1Results
TULIP outperforms output- and representation-level baselines across TOFU, PISTOL, and WMDP while preserving utility and remaining robust to paraphrase and quantization attacks.
Source: §1 · p. 2Takeaways & Limitations
Per-input layer selection also works as a plug-and-play component that improves existing unlearning methods.
Source: §1 · p. 2The results do not establish the logit lens as the optimal formation–readout boundary estimator.
Source: §5.2 · pp. 7–8
Abstract
from arXiv · showhide
Representation-level unlearning modifies the intermediate hidden states of LLMs. Although knowledge is distributed across layers, existing methods operate at a single fixed layer for the entire forget set. We ask whether such a fixed layer is sufficient for unlearning. To answer this, we design a hijacking experiment that grafts hidden states of the target model into an oracle model. The oracle cannot produce the forget answer on its own, yet it produces the answer from the grafted state. This indicates the answer is already formed at an intermediate layer and read out afterward. Moreover, the layer where formation ends varies widely across inputs. Motivated by these findings, we propose Targeted Unlearning at Layers Identified Per-input (TULIP). For each input, TULIP uses the logit lens to locate the formation-readout boundary and removes the hidden state's alignment with the forget answer's unembedding vector at that layer. TULIP outperforms output- and representation-level baselines on TOFU, PISTOL, and WMDP across Llama, Qwen, and Zephyr models. It also remains robust to paraphrase and quantization attacks. Beyond standalone use, its per-input layer selection serves as a plug-and-play component that further improves existing methods.
1. Introduction
TULIP selects an intervention layer for each input because the layer where a target is formed varies across inputs. It outperforms output- and representation-level baselines across benchmarks while preserving utility and robustness to attacks.
1. Introduction
- Representation-level unlearning modifies internal representations, but existing methods apply one selected layer uniformly across the forget set.RMU, Adaptive RMU, and LUNAR use different selection criteria, yet retain the fixed-layer setting.Source: §1 · p. 1
- The hijacking experiment shows that an oracle can produce a target token from the target model’s grafted hidden state, indicating formation precedes readout.The formation–readout boundary varies widely across inputs, making a single intervention layer insufficient.Source: §1 · p. 1
- TULIP uses the logit lens to locate each input’s formation–readout boundary, then removes hidden-state alignment with the target’s unembedding vector while preserving retain-set states.The retain term preserves hidden states on the retain set.Source: §1 · p. 1
- The evaluation spans TOFU, PISTOL, and WMDP and Llama, Qwen, and Zephyr backbones ranging from 1B to 8B parameters.The supplied evaluation summary reports performance across benchmarks and model sizes.Source: §1 · pp. 1–2
- TULIP outperforms output- and representation-level baselines across TOFU, PISTOL, and WMDP while preserving utility and robustness to paraphrase and quantization attacks.Its per-input layer selection also improves existing unlearning methods as a plug-and-play component.Source: §1 · p. 2
2. Related Work
Output-level methods suppress forget-data responses but may leave the underlying information recoverable. Representation-level methods modify internal representations, though the methods described still intervene at one fixed layer for all forget data.
Output-level unlearning
- Output-level methods suppress target-token probabilities through forget-set loss, reference-relative likelihood, or localized updates, but can change responses without removing knowledge.Examples include GA, GradDiff, NPO, and SimNPO; methods also restrict updates using Fisher information or selected tokens.Source: §2 · p. 2
Representation-level unlearning
- Representation-level methods modify encoded information by steering forget representations toward random vectors, norm-scaled targets, or refusal-style representations.RMU, Adaptive RMU, and LUNAR are the described examples.Source: §2 · p. 2
- The described representation-level methods intervene at a single fixed layer for all forget data, and their effectiveness depends on the chosen layer.This is the shared layer-selection limitation despite differences in intervention strategy.Source: §2 · p. 2
3. Method
TULIP estimates where each forget token is formed and applies unlearning at that per-input boundary, while preserving later readout layers. The method is motivated by a hijacking experiment and uses the logit lens to estimate the boundary without an oracle.
3. Method
- Each response is analyzed token by token, pairing the query and preceding response tokens as context with the next response token as target.The forget and retain sets collect these context–target pairs.Source: §3 · p. 2
3.1. Unlearn formation, not readout
- 0.61 to 1.00: oracle target accuracy rose when grafted target-model hidden states supplied the forget information, supporting formation before readout.The experiment uses a target fine-tuned on the full dataset and an oracle trained only on retain data.Source: Figure 1 · p. 3, §3.1 · p. 3
3.2. Where does formation end?
- The formation–readout boundary is the earliest layer where hijacking succeeds at every later layer, and it varies widely across forget inputs.TULIP uses the boundary to identify layers to modify while leaving later readout layers intact.Source: §3.2 · pp. 3–4
3.3. Estimating FRB with logit lens
- TULIP estimates each boundary with the logit lens, projecting the target model’s hidden state through its unembedding and selecting the top token.This substitutes the target model’s logit-lens projection for the oracle’s remaining layers.Source: §3.3 · p. 4
3.4. Unlearning objective at FRB
- At the estimated boundary, TULIP suppresses hidden-state alignment with the target token’s unembedding vector and uses a retain term to preserve retain-set states.The forget loss gradient reaches transformer layers only up to the estimated boundary; the full objective balances forget and retain losses.Source: §3.4 · p. 4
4. Experiments
Experiments compare TULIP with output- and representation-level baselines on TOFU, PISTOL, and WMDP. TULIP reports strong forget quality while retaining utility, including lower forget accuracy on WMDP.
4.1. Setup
- TOFU contains 4,000 QA pairs about 200 fictitious authors with 1/5/10% forget splits; PISTOL contains 400 synthetic QA pairs from 20 contracts among 24 entities.For PISTOL, the sales contract between companies A and C is unlearned while the rest are retained.Source: §4.1 · p. 4
4.2. Main Results
- TULIP achieves best or near-best FQ in nearly every TOFU and PISTOL setting while maintaining MU, and the lowest forget accuracy on WMDP-Cyber and WMDP-Bio while preserving retain accuracy.The TOFU/PISTOL results are reported in Table 1; the WMDP results indicate unlearning transfers from instance-level to domain-level settings.Source: Table 1 · p. 5, §4.2 · pp. 5–6
4.1. Setup
- FQ is a KS-test p-value and MU is average ROUGE-L recall; comparisons include output-level and representation-level unlearning baselines.WMDP evaluates hazardous biosecurity and cybersecurity knowledge with four-choice questions after unlearning from raw domain documents.Source: §4.1 · p. 5
4.2. Main Results
- On TOFU, output-level methods’ FQ falls near zero on the 5% and 10% splits, while TULIP’s per-input layer selection avoids fixed-layer mismatch across settings.Output-level methods can also lose MU; RMU and LUNAR perform better, but their FQ varies across splits and model scales.Source: §4.2 · p. 5
- On PISTOL, TULIP forgets substantially more while keeping MU competitive, addressing the challenge of forgetting contracts while retaining knowledge of their shared entities.Source: §4.2 · p. 6
- Because WMDP tests independently written questions after unlearning from raw documents, TULIP’s low forget accuracy indicates domain knowledge removal rather than only document-reproduction suppression.The reported result also shows TULIP scales from instance-level to domain-level unlearning.Source: §4.2 · p. 6
5. Analysis
The analysis finds that per-input intervention depth outperforms a fixed layer and that the logit lens is the strongest of three tested boundary estimators, though not established as optimal. TULIP remains robust to paraphrasing and 8-bit quantization, and per-input layer selection improves existing methods.
5.3. Robustness against attacks
- TULIP remains robust to paraphrasing and achieves the highest forget quality among methods after 8-bit quantization, but its robustness declines at 4 bits.Under 4-bit quantization, the authors suggest its single-direction update may be too localized to survive coarse quantization.Source: Table 3 · p. 7, §5.3 · p. 8
5.1. Effect of intervention layer selection
- TULIP’s per-input intervention depth outperforms even the best fixed layer in forget quality.At a fixed layer, deeper updates initially improve forget quality but can also reach general readout layers, conflicting with retention.Source: §5.1 · pp. 6–7
- Intervening at the estimated boundary achieves the highest forget quality while preserving model utility.Earlier updates can leave later layers able to recover the target; later ones modify general readout layers and conflict with retention.Source: §5.1 · p. 7
5.2. Validity of the logit-lens approximation
- Among three tested lenses, the logit lens best separates states before versus at or beyond the oracle boundary by AUROC, but is not established as optimal.The alternatives require training, and future lens-based estimators or boundary-tailored objectives may improve on this approach.Source: Figure 3 · p. 7, §5.2 · pp. 7–8
5.5. Qualitative analysis
- TULIP often gives plausible but incorrect, oracle-like answers, while baselines show distinct failures including retained gold information, degeneration, or refusals.The authors attribute TULIP’s responses to modifying target formation while leaving readout to behave as in the oracle.Source: Figure 4 · p. 8, §5.5 · p. 9
5.4. Per-input layer selection as a plug-in
- Per-input layer selection improves every baseline; GradDiff, NPO, and SimNPO reach MU around 0.94, nearly matching the oracle.Forget quality also improves across baselines, but TULIP still exceeds the combined baselines by a large margin; LUNAR is excluded because its single-layer optimization does not support dynamic selection.Source: Table 4 · p. 8, §5.4 · pp. 8–9
6. Conclusion
The conclusion argues that unlearning should target formation while preserving readout, with a separate boundary for each input. TULIP implements this approach and improves benchmarks and existing unlearning methods.
6. Conclusion
- TULIP targets each input’s formation–readout boundary, removing target alignment there while preserving readout.The boundary is located with the logit lens.Source: §6 · p. 9
- Its gains across benchmarks and model families, and improvements to existing methods, support targeting formation as a direction for representation-level unlearning.Source: §6 · p. 9
- The conclusion says formation ends at different layers for different inputs, so intervention layers should be selected per input rather than fixed.Source: §6 · p. 9
A. Additional Qualitative Results
On the TOFU 5% forget split with Llama-3.1-8B, the qualitative examples compare TULIP’s outputs with the oracle and show failure patterns among some baselines.
A. Additional Qualitative Results
- In Figure 5, TULIP produces the same answer as the oracle model on the TOFU 5% forget split with Llama-3.1-8B.Source: Figure 5 · p. 14
- Some baselines leak the target model’s forget-set answer or produce degenerate text in both qualitative examples.Source: Figure 5 · p. 14, Figure 6 · p. 14
- In Figure 6, TULIP produces an answer close to the oracle model’s on the TOFU 5% forget split with Llama-3.1-8B.Source: Figure 6 · p. 14
B. Hijacking versus Activation Patching
The hijacking experiment distinguishes its recipient from those used in prior activation patching: the oracle has never seen the forget data. This supports the conclusion that the grafted state carries the target and that subsequent readout does not require that knowledge.
B. Hijacking versus Activation Patching
- Because the oracle has never seen the forget data, a recovered target must come from the grafted state, unlike patching where recipient knowledge may be uncontrolled.Prior patching recipients may already know the fact, allowing downstream layers to complete it themselves.Source: B. Hijacking versus Activation Patching · p. 15
- The experiment finds that readout after the boundary works equally well in a model that never learned the fact.Source: B. Hijacking versus Activation Patching · p. 15
- The boundary varies widely across forget inputs, motivating per-token boundary measurement and targeting.The paper contrasts this with prior patching studies that report which layers matter on average.Source: B. Hijacking versus Activation Patching · p. 15
C. Details of Benchmarks
The benchmark section describes the forget and retain data and evaluation measures for TOFU, PISTOL, and WMDP. The WMDP setting uses closely related retain data to assess removal while preserving related knowledge.
C. Details of Benchmarks
- TOFU contains 4,000 synthetic QA pairs about 200 fictitious authors; experiments use 1%, 5%, and 10% author forget splits and evaluate forget quality against an oracle using Truth Ratio distributions.Model utility is average ROUGE-L recall on the retain set, real authors, and world facts.Source: C. Details of Benchmarks · p. 15
- PISTOL Sample Dataset 1 contains 400 QA pairs across 20 contracts among 24 entities, with the 20 sales-contract pairs between companies A and C forgotten.Forget quality uses the TOFU procedure; utility is average ROUGE-L recall on PISTOL retain and control sets.Source: C. Details of Benchmarks · p. 15
- WMDP comprises 3,668 expert-reviewed questions across biosecurity, cybersecurity, and chemistry; experiments forget biosecurity and cybersecurity corpora while retaining subject-specific corpora.The retain corpora are close in content to the forget corpora; WMDP accuracy closer to 25% indicates more effective unlearning, while MMLU measures retained general knowledge.Source: C. Details of Benchmarks · pp. 15–16
D. Model Utility Metric on TOFU
The appendix replaces TOFU’s nine-metric harmonic-mean utility score with a generation-based score because probability perturbations can inflate the original metric without preserving answer quality.
D. Model Utility Metric on TOFU
- TOFU’s original MU is the harmonic mean of nine answer-probability, ROUGE-L recall, and Truth Ratio metrics across the retain set, real authors, and world facts.The probability terms can raise MU even when generated answers stay the same or degrade.Source: D. Model Utility Metric on TOFU · p. 16
- 0.690 MU for an unlearned Llama-3.1-8B exceeds the target model’s 0.627 under the original metric, despite unlearning only removing information.The cited example illustrates how probability perturbations can make the original aggregate misleading.Source: D. Model Utility Metric on TOFU · p. 16
- The revised MU averages ROUGE-L recall over generations on the retain set, real authors, and world facts to reflect whether correct answers are still produced.This removes the probability terms implicated in the original metric’s artifact.Source: D. Model Utility Metric on TOFU · p. 16
E. Details of Baselines
The appendix describes five baseline approaches and clarifies that, unlike full-response baselines, TULIP acts on context–target token pairs.
Conventions
- The appendix writes baseline objectives over full question–answer responses and defines expectations over forget and retain sets; TULIP instead operates on context–target token pairs.The superscript ref denotes quantities computed by the frozen target model before unlearning.Source: E. Details of Baselines · p. 16
GradDiff
- GradDiff applies gradient ascent to forget responses and retain-set NLL minimization, since optimizing forgetting alone tends to degrade retain performance.Its forget objective raises prediction loss and lowers target-response generation probability.Source: E. Details of Baselines · p. 16
RMU
- RMU redirects forget-set hidden states toward a fixed random unit vector at one layer while keeping retain states close to the frozen target model.It updates parameters only from layers ℓ−2 through ℓ, unlike methods that directly modify output probabilities.Source: E. Details of Baselines · pp. 16–17
NPO
- NPO treats forget responses as negative examples, lowering their likelihood relative to the frozen target model while applying retain-set NLL.Its bounded-below loss has adaptive gradients that vanish for already-forgotten samples, slowing progression toward model-utility collapse compared with GA.Source: E. Details of Baselines · p. 17
SimNPO
- SimNPO uses length-normalized forget-response log-likelihood without a frozen target model, with margin δ setting the required likelihood reduction before its loss becomes small.At δ = 0 and β → 0, its forget-set gradient approaches GA with each response weighted by 1/|y_f|.Source: E. Details of Baselines · p. 17
LUNAR
- LUNAR redirects forget-data activations toward a region associated with inability to answer, using prompts where the frozen model already declines as its reference set.It trains forget activations toward shifted original values and keeps retain activations at their original values, updating only the layer’s MLP down-projection weights.Source: E. Details of Baselines · p. 17
F. Hyperparameters
The appendix gives benchmark-specific training settings and a shared hyperparameter-search procedure, with TULIP requiring only a forget-loss weight search.
Training setup
- Across benchmarks, experiments use one RTX PRO 6000 GPU, 32-bit paged AdamW, weight decay 0.01, no warmup, and per-device batch size 4.TOFU uses 10 epochs and effective batch size 16; PISTOL and WMDP use benchmark-specific epoch, learning-rate, and accumulation settings.Source: F. Hyperparameters · pp. 17–18
- TULIP’s intervention layer is determined per input, leaving the forget-loss weight λ as its only tuned hyperparameter while the retain-loss weight stays fixed at 1.The paper assigns λ the role of γ in baseline objectives.Source: F. Hyperparameters · p. 18
Search procedure
- Hyperparameters are tuned separately by benchmark, model, and forget split, selecting on seed 0 and reporting the average across seeds 0–2.TULIP needs no layer search, whereas NPO, RMU, and most LUNAR settings require joint searches over multiple hyperparameters, including intervention layers for RMU and LUNAR.Source: F. Hyperparameters · p. 18