Assessing Knowledge Editing in Language Models via Relation Perspective

• Existing knowledge-editing methods focus on modifying entities, despite factual knowledge being represented as subject–relation–object triples whose relations can change over time.The introduction illustrates this with Parag Agrawal’s changing role at Twitter, from CTO to CEO.Source: §1 · p. 1, Figure 1 · p. 1
• RaKE introduces relation-based knowledge editing to address the gap left by entity-centric approaches.The benchmark extends knowledge-editing evaluation to the relation perspective.Source: §1 · pp. 1–2
• Experiments reveal significant limitations in current methods’ ability to perform relation-based editing.Source: §1 · p. 2
• Causal tracing indicates that relational knowledge is stored in both the FFN and attention layers, whereas existing entity-based methods primarily modify FFN parameters.The paper links this mismatch to underperformance in relation-based editing.Source: §1 · p. 2

2.1 Task Definition

• The task definition includes neighboring subjects, relation paraphrases, the original object oc, and the edited object o* as evaluation variations.Source: §2.1 · p. 2
• Relation-based editing completes the relation r given a subject s and object o, while entity-based editing predicts the object from s and r.The two perspectives are presented as logically equivalent views of the same factual knowledge.Source: §2.1 · p. 2, Figure 2 · p. 3
• The task modifies a base model so pθ(r|s,o*) increases while pθ(r|s,oc) decreases, replacing the original object with the edited object.Source: §2.1 · p. 2

2.2 Evaluation Metrics

• RaKE operationalizes relation editing with deletion and addition operations that remove r from (s,o) or add r between s and o*.Figure 2 presents the equivalence between these relation operations and entity-based editing.Source: Figure 2 · p. 3
• Evaluation measures efficacy, paraphrase generalization, and neighborhood effects through Success, Magnitude, Paraphrase, Neighborhood, and harmonic-mean Score metrics.Success compares post-edit probabilities, while Magnitude measures their average difference.Source: §2.2 · p. 3

3.1 Dataset Construction

• RaKE constructs relation-generalization prompts from 34 Wikidata relations, PARAREL paraphrases, manually designed templates, and GPT3.5turbo coherence checks.Source: §3.1 · p. 3

3.2 Dataset Comparison

• RaKE evaluates the equivalent relation completion (s, o* → r) that existing benchmarks omit while retaining entity-based editing (s, r → o*).Source: Table 1 · p. 4, §3.2 · p. 4

3.1 Dataset Construction

• Relation-based editing is formulated through deleting a relation between s and o and adding it between s and o*, making the task logically equivalent to entity-based editing.Source: §3 · p. 3, §3.1 · p. 4

3.2 Dataset Comparison

• RaKE represents factual knowledge from both perspectives because changing any component of (s, r, o) transforms the associated knowledge, creating mutual dependence between entity and relation views.Source: §3.2 · p. 4

3.3 Dataset Statistics

• The benchmark contains 21,919 editing samples with entity- and relation-based edit, paraphrase, and neighborhood prompts for evaluating edits and unrelated knowledge.Source: §3.3 · p. 4

4.1 Experimental Setup

• The experiments evaluate FT, KN, MEND, ROME, and MEMIT on GPT-2 XL and GPT-J using memory-based and Locate-Then-Edit paradigms.The setup focuses on transformer language models and compares several established editing baselines.Source: §4.1 · pp. 4–5

4.2 Results and Analysis

• Relation-based editing performs substantially worse than entity-based editing on Efficacy, despite the two perspectives representing logically equivalent factual triples.Table 3 reports a significant R-Efficacy decrease relative to E-Efficacy on EM.Source: §4.2 · p. 5, Table 3 · p. 5
• Entity- and relation-based methods improve generalization mainly within their own perspectives, with limited transfer to the other perspective.These differences suggest that entity and relation knowledge may be partly independent and stored in different model components.Source: §4.2 · p. 6, Figure 4 · p. 6

4.3 Casual Tracing

• Causal tracing corrupts relation embeddings, restores hidden states at selected token-layer positions, and measures indirect effects on relation prediction.Average indirect effect is computed across prompts for each hidden state.Source: §4.3 · p. 6
• Causal tracing finds high AIE in later final-token MLP states, earlier relation-token layers, and middle attention layers.Restoring later MLP states recovers much information, while attention effects indicate relation memory is not confined to MLPs.Source: §4.3 · p. 6, Figure 5 · p. 7

4.4 Severed Causal Analysis

• Severed causal analysis links relation-related AIE to attention modules in layers 10 to 20, while confirming important MLP effects in earlier and middle layers.Higher-layer MLP restoration appears independent of MLP activity and may introduce unintended side effects.Source: §4.4 · p. 7

4.5 Discussion

• The discussion argues that editing methods should address both entity and relation perspectives and investigate attention layers alongside FFNs.Current methods do not simultaneously achieve strong editing from both perspectives.Source: §4.5 · p. 7

5.1 Memory in LLMs

• Memory in LLMsFFN layers are described as key-value memories, while prior work also connects attention updates to memory storage and retrieval.Source: §5.1 · pp. 7–8
• Memory in LLMsKnowledge-neuron interventions in FFN layers can alter factual expression, and mid-layer feed-forward modules mediate factual predictions during subject-token processing.Source: §5.1 · p. 8
• Memory in LLMsAttention-head hidden-state memory injection has been reported as an efficient way to improve performance on multi-hop prompts.Source: §5.1 · p. 8

5.2 Memory Editing

• Memory EditingBecause model knowledge becomes outdated or incorrect, editing methods aim to update facts while preserving other valuable knowledge; parameter-efficient fine-tuning and incremental learning can overfit and be computationally costly.Source: §5.2 · p. 8
• Memory EditingExisting model-editing approaches comprise memory-based methods, locate-then-edit methods, and training-free in-context knowledge editing.Source: §5.2 · p. 8

6 Conclusion

• Existing knowledge editing methods show potential difficulty editing relations, while causal analysis finds relational knowledge stored in both FFN and attention layers.The conclusion identifies FFN-only editing as incomplete for relation-based editing, producing suboptimal results.Source: §6 · p. 8
• The paper introduces relation-based knowledge editing and the RaKE benchmark for evaluating it.Source: §6 · p. 8

Limitations

• The current RaKE dataset lacks an assessment of relation specificity performance, which is planned for future versions.Source: Limitations · p. 8
• As the first paper on relation-perspective knowledge editing, the study acknowledges that specific methods for editing relations are not yet provided.The authors describe the work as a preliminary investigation and plan to refine relation-targeted editing methods.Source: Limitations · p. 8