Source-linked AI summary
ALOE: Semantically Addressed Low-Rank Operators for Knowledge Editing
Zeyan Li, Hu Xu, Jianfeng Xu
TL;DR
Knowledge editing must select which hidden states receive a new fact’s residual, but that address is often implicit or external to the edited model. ALOE learns semantic addresses, aligns and calibrates their gates on autoregressive states, and embeds the resulting operator in an MLP. Across three benchmarks and three 7–8B model families, it reports high efficacy and locality, with remaining errors concentrated in paraphrase coverage and write fitting.
Problem
Knowledge editing must update a requested fact, generalize to equivalent wordings, and preserve unrelated behavior, but the hidden-state scope of an update is often left implicit.
Source: §1 · p. 1Method
ALOE learns semantic addresses from paraphrases and same-subject hard negatives, refines and calibrates gates on autoregressive states, and fits gated low-rank writes inside one MLP.
Source: §1 · p. 1, §3.1 · p. 2, §3.3 · p. 3Results
Takeaways & Limitations
Mechanism analyses find that learned addresses separate in-scope states from competing ones beyond what raw hidden states provide, and transfer to an untuned model family.
Source: §5 · p. 4Remaining errors concentrate in paraphrase coverage and write fitting.
Source: §4.3 · p. 4
Abstract
from arXiv · showhide
Knowledge editing changes what a model knows by modifying parameters so that a requested fact updates while unrelated behavior is preserved. This is usually treated as a write problem, but editing also involves an address problem: deciding which hidden states should receive the new residual. An update that activates too narrowly memorizes one prompt, while one that activates too broadly disrupts neighboring knowledge. Parametric editors encode this scope implicitly, whereas memory-based editors make the selection explicit but keep it outside the edited model. We propose ALOE (Addressed Low-rank Operator for Editing), which learns semantic addresses from paraphrases and hard same-subject negatives, aligns them with autoregressive hidden states through rollout refinement and gate calibration, and embeds the resulting gated low-rank operator within one MLP layer, so that the deployed model runs in a single forward pass with no external retriever or auxiliary router. Evaluated on CounterFact, ZSRE, and KnowEdit across three 7--8B model families, ALOE achieves efficacy between 0.955 and 0.999 and locality between 0.981 and 1.000; mechanistic analyses confirm that the learned geometry separates competing edits and that calibration suppresses out-of-scope activation. The remaining errors concentrate in paraphrase coverage and write fitting.
1. INTRODUCTION
Knowledge editing requires both writing a new fact and addressing the hidden states where its residual should act. ALOE learns that selection inside the edited model, with the aim of covering paraphrases while preserving out-of-scope behavior.
1. INTRODUCTION
- Editing has separate write and address problems: specifying a new-fact residual and deciding which hidden states should receive it.An address that is too narrow can miss rewordings, while one that is too broad can affect neighboring knowledge.Source: §1 · p. 1
- Existing low-rank editors leave address factors untrained for separating paraphrases from semantic neighbors, while memory-based selection runs outside edited weights.The passage names ROME, MEMIT, and PMET as parametric approaches and SERAC, GRACE, and WISE as explicit-selection approaches.Source: §1 · p. 1
- ALOE learns semantic addresses from paraphrases and same-subject, different-relation negatives, then refines and calibrates gates before fitting writes inside one MLP layer.At inference, the address is part of the MLP rather than an external selection component.Source: §1 · p. 1
- Efficacy ranges from 0.955 to 0.999 and locality from 0.981 to 1.000 on edit streams of 839 to 1,301 facts across CounterFact, ZSRE, and KnowEdit.The evaluation spans three 7–8B model families.Source: §1 · pp. 1–2
2. RELATED WORK
Prior work either modifies shared parameters, leaving edit scope implicit, or makes access explicit through mechanisms outside the edited weights. ALOE places learned selective access inside an edited MLP.
2.1. Parametric knowledge editing
- Parametric editors construct or constrain weight updates, but the resulting edit scope is induced implicitly by their interaction with hidden states.The section describes learned gradient-to-update methods, direct MLP writes, and preservation- or orthogonal-subspace constraints.Source: §2.1 · p. 2
2.2. Selective access and edit scope
- Explicit-access systems use retrievers, codebooks, routers, or prompts outside edited weights, whereas ALOE gates low-rank writes inside an edited MLP.The section also distinguishes methods that locate where an edit is placed from mechanisms that decide when it fires.Source: §2.2 · p. 2
- Evaluation work measures specificity, propagation to related knowledge, and forgetting, motivating edits that activate on relevant queries but not nearby out-of-scope knowledge.The section cites CounterFact+, RippleEdits, and ReCoE as examples of scope-related evaluations.Source: §2.2 · p. 2
3. METHOD
ALOE gives each edit an address that gates its write direction, learns semantic boundaries from paraphrases and hard negatives, and aligns those boundaries with generation-time states. It then jointly fits writes under the calibrated gates for deployment within an MLP.
3.1. An addressed residual operator
- Each edit has an address that selects hidden states and a write direction that carries the new fact; the address gates the write in a state-dependent MLP operator.The dead-zone sigmoid makes inactive-address gates exactly zero, and Fig. 2 depicts execution in a standard forward pass.Source: §3.1 · p. 2, Figure 2 · p. 3
3.2. Learning the semantic boundary
- The semantic-boundary objective uses request paraphrases and same-subject prompts with different predicates as hard negatives, ranking in-scope queries above wrong-relation alternatives.Orthogonality regularization discourages address-key collapse, and relation-balanced minibatches limit domination by frequent predicates.Source: §3.2 · p. 2
3.3. From construction to generation
- Because generation-time hidden states can drift from construction prompts, ALOE refines addresses on autoregressive rollout states and calibrates gates to separate positive and negative states where possible.After refinement, only the address vectors and gate thresholds and temperatures remain in the edited model; the metric-learning components, labels, and prompts are discarded.Source: §3.3 · pp. 2–3
3.4. Fitting writes jointly
- A preservation-regularized joint ridge solve initializes write directions from target residuals and gate activations, accounting for overlap that can leak one edit’s write through another edit’s gate.The writes are then optimized with the gates frozen; deployment computes the gates and adds their low-rank residual to the MLP projection.Source: §3.4 · p. 3
4. EXPERIMENTS
Across three benchmark families and 7–8B models, ALOE maintains high efficacy and locality over long edit streams. Mechanism analyses show learned addresses outperform raw activation controls, while rollout refinement and calibration address distinct runtime failures.
4.1. Setup
- ALOE is evaluated on CounterFact, ZSRE, and KnowEdit using Llama-2-7B, Llama-3.1-8B-Instruct, and Qwen3-8B, with three-seed means.Five established editors are included; efficacy, generalization, and locality are reported separately.Source: §4.1 · p. 3
4.3. Mechanism analysis
- The mechanism test uses predicate-disjoint train, calibration, and development partitions, with 632 examples paired against 16 same-subject wrong-relation keys.Development subjects and exact surfaces are excluded from training.Source: §4.3 · p. 3
4.2. End-to-end editing
- Efficacy stays between 0.955 and 0.999 and locality between 0.981 and 1.000 across all nine model–benchmark cells, though generalization remains 0.217–0.472.The comparison covers streams of up to 1,301 edits; the supplied table caption identifies the model–benchmark metrics and edit-stream scope.Source: §4.2 · p. 3, Table 1 · p. 4
4.3. Mechanism analysis
- Against raw activation controls, learned addresses reach AUC 0.9976 versus 0.6365, recall 0.9910 versus 0.1804, and same-subject top-1 0.8576 versus 0.3117.Negative FPR is 0.0418; seed standard deviations are below 0.006 on every metric.Source: §4.3 · p. 4
- Full-request keys preserve 632 distinct addresses and raise synthetic-write top-1 from 0.698 to 0.998, while entity-neutral keys collapse edits onto 179 addresses.The construction objective transfers without retuning, reaching AUC 0.997–0.998 and FPR 0.036–0.042 in three other model families.Source: §4.3 · p. 4
- Matched slots dominate 98.4% of original requests, but rephrase gate activation averages 0.057 versus 0.860 for originals; locality-state drift averages 11.54%.The figure reports 96.5% next-token agreement. Rollout refinement adds 55.0 efficacy points, while calibration adds 24.8 efficacy and 96.5 locality points, slightly reducing generalization.Source: §4.3 · p. 4, Figure 3 · p. 4
5. CONCLUSION
ALOE makes edit addressing intrinsic to an MLP and maintains high efficacy and locality across the evaluated benchmarks and model families. Its remaining errors center on paraphrase coverage and write fitting, within an evaluation limited to English factual benchmarks, 7–8B models, and one edited layer.
5. CONCLUSION
- ALOE couples address and write decisions inside an edited MLP, learning scope from paraphrases and same-subject hard negatives before rollout refinement and gate calibration.A joint solve fits write directions under calibrated gates.Source: §5 · p. 4
- Across CounterFact, ZSRE, and KnowEdit on three 7–8B model families, efficacy is 0.955–0.999 and locality is 0.981–1.000 on streams up to 1,301 edits.Mechanism analyses find learned addresses separate in-scope states from competing ones beyond raw hidden states, and transfer to an untuned model family.Source: §5 · p. 4
- Missed paraphrases are address failures, while an open gate followed by an incorrect generation is a write failure; generalization remains 0.217–0.472.The conclusion attributes the generalization limit to calibrated gates rarely opening for rephrasings far from training paraphrases.Source: §5 · p. 4
- Evaluation is limited to three English factual benchmarks, 7–8B models, and one edited layer, with storage growing linearly in the number of edits.The conclusion reports rollout refinement as driving most efficacy improvement and gate calibration as driving most locality improvement.Source: §5 · p. 4