PaperlayerView exact revision on arXiv

Source-linked AI summary

LLM-Align: Utilizing Large Language Models for Entity Alignment in Knowledge Graphs

Xuan Chen, Tong Lu, Zhichun Wang

arXiv:2412.04690v1cs.CLcs.AI

TL;DR

Entity alignment matches corresponding entities across knowledge graphs, but irrelevant triples can distract LLM reasoning and single-round decisions may be unreliable. LLM-Align selects candidates and informative entity triples for sequential attribute- and relation-based reasoning, with multi-round voting. Across three datasets, it improves existing models and achieves the best results when paired with a strong model.

  • Problem

    Entity alignment matches corresponding entities across knowledge graphs, but irrelevant entity triples can distract LLM reasoning and single-round LLM decisions can be unreliable.

    Source: §1 · pp. 2–3

  • Method

    LLM-Align selects candidate alignments with an existing EA model, uses heuristics to select informative attributes and relations for sequential LLM reasoning, and applies multi-round voting.

    Source: §1 · p. 3

  • Results

    LLM-Align enhances both strong and weaker EA models and achieves the best results when combined with strong models.

    Source: §1 · p. 3, §6 · p. 21

  • Takeaways & Limitations

    Across three EA datasets, the reported results support LLM-Align as an enhancement to existing models, with best performance when candidate selection uses a robust EA model.

    Source: §6 · p. 21

Abstract

from arXiv · show

Entity Alignment (EA) seeks to identify and match corresponding entities across different Knowledge Graphs (KGs), playing a crucial role in knowledge fusion and integration. Embedding-based entity alignment (EA) has recently gained considerable attention, resulting in the emergence of many innovative approaches. Initially, these approaches concentrated on learning entity embeddings based on the structural features of knowledge graphs (KGs) as defined by relation triples. Subsequent methods have integrated entities' names and attributes as supplementary information to improve the embeddings used for EA. However, existing methods lack a deep semantic understanding of entity attributes and relations. In this paper, we propose a Large Language Model (LLM) based Entity Alignment method, LLM-Align, which explores the instruction-following and zero-shot capabilities of Large Language Models to infer alignments of entities. LLM-Align uses heuristic methods to select important attributes and relations of entities, and then feeds the selected triples of entities to an LLM to infer the alignment results. To guarantee the quality of alignment results, we design a multi-round voting mechanism to mitigate the hallucination and positional bias issues that occur with LLMs. Experiments on three EA datasets, demonstrating that our approach achieves state-of-the-art performance compared to existing EA methods.

1. Introduction

Entity alignment addresses the heterogeneity of independently developed knowledge graphs, but using all entity triples can distract LLM reasoning and single-round decisions may be unreliable. LLM-Align selects alignment candidates, applies attribute- and relation-based reasoning, and uses multi-round voting; experiments report improved results for existing models.

  • All entity triples may include irrelevant information that disturbs LLM reasoning, while single-round alignment decisions can be unreliable.Source: §1 · p. 3
  • The framework first selects candidate alignments with an existing EA model, then performs attribute-based and relation-based reasoning sequentially.Source: §1 · p. 3
  • Heuristic attribute and relation selection identifies informative entity details for concise, effective LLM prompts.Source: §1 · p. 3
  • Multi-round voting reorders candidates and aggregates repeated LLM votes to mitigate hallucination and positional bias.Source: §1 · p. 3
  • LLM-Align enhances both strong and weaker EA models and achieves the best results when combined with strong models.Source: §1 · p. 3

2. Related Work

Embedding-based EA methods learn entity representations from graph structure and, in some cases, names or attributes. Language-model approaches add semantic representations or LLM reasoning, while candidate-based LLM methods still rely on existing EA techniques and have reported limited LLM-contributed improvements.

2.1. Embedding-based EA

  • Embedding-based methods learn entity representations from graph structure, with some approaches incorporating names, attributes, or other entity features.Examples include neighborhood-aware GNNs and methods that combine structural, semantic, and string-based features.Source: §2.1 · p. 4

2.2. Language Model-based EA

  • PLM-based methods encode entity text and triples, but pairwise input approaches such as BERT-INT and TEA are described as limiting scalability to larger KGs.Source: §2.2 · pp. 4–5
  • LLMEA and ChatEA use existing EA methods to generate candidates for LLM selection, and their LLM-contributed improvements are described as somewhat limited.Source: §2.2 · p. 5

3. Problem Definition

A knowledge graph is represented through entities, relations, attributes, literals, and relational and attributive triples. Entity alignment identifies the equivalent target-graph entity for each source entity.

  • A knowledge graph is represented as sets of entities, relations, attributes, and literals, together with attributive and relational triples.Source: §3 · p. 5
  • Entity alignment identifies the equivalent target entity in the target graph for each source entity.Source: §3 · p. 6

4. Proposed Approach

LLM-Align uses candidate retrieval followed by attribute- and relation-based LLM reasoning to align entities. It selects informative triples for prompts and uses multi-round voting to stabilize reasoning.

4. Proposed Approach

  • LLM-Align retrieves top-k candidate targets with an existing EA model, then applies attribute-based reasoning followed by relation-based reasoning when no candidate meets the voting criterion.Previous similarity scores are ignored during LLM reasoning.Source: §4 · p. 6, §4.1 · p. 6

4.2. EA Prompts for LLMs

  • Entity names, attributes, or relations provide the information in knowledge-driven, attribute-aware, and relation-aware prompts, respectively.Relation triples are included because attributes may not provide sufficient alignment evidence.Source: §4.2 · pp. 7–8

4.3. Heuristic Attribute and Relation Selection

  • Attribute identifiability scores guide selection of the top-k attributes from the source entity and its candidate targets for attribute-aware prompts.Identifiability quantifies how significant an attribute is for distinguishing entities.Source: §4.3 · pp. 8–9
  • Relation identifiability similarly guides selection of the top-k relations for relation-aware prompts.The selected relations are used as LLM input for alignment reasoning.Source: §4 · p. 6, §4.3 · p. 9

4.3.1. Multi-round Voting Mechanism

  • Multi-round voting samples n unique candidate permutations, runs parallel reasoning, and outputs a target only when its vote count reaches ⌊n/2⌋; otherwise, it abstains.The paper reports more stable alignment reasoning than single-round reasoning and proposes voting to mitigate positional bias and hallucinations.Source: §4.3.1 · pp. 9–10

5. Experiments

The experiments use three cross-lingual DBP15K datasets, two candidate-selection models, and two Qwen reasoning models. The comparison includes prior EA methods spanning language-model, graph, and retrieval approaches.

5.1.1. Datasets

  • Experiments use DBP15K Chinese-English, Japanese-English, and French-English cross-lingual entity-alignment datasets derived from multilingual DBpedia subsets.The datasets were created by Sun et al..Source: §5.1.1 · p. 10

5.1.2. Models Settings

  • DERA-R and GCN-Align provide candidate entities, while Qwen1.5-32B-Chat and Qwen1.5-14B-Chat perform LLM reasoning.The candidate models represent textual and structural information, respectively; both Qwen models run with vLLM on a single 80G GPU.Source: §5.1.2 · pp. 10–11

5.1.3. Baselines

  • Baselines include DERA, DERA-R, TEA, BERT-INT, HMAN, AttrGNN, LLMEA, and ChatEA.They cover heterogeneous parsing, language-model methods, graph-based attribute modeling, and candidate retrieval with LLM reasoning.Source: §5.1.3 · pp. 11–12

5.2. Overall Results

On DBP15K, LLM-Align improves Hits@1 over GCN-Align and reaches the highest Hits@1 with DERA-R and Qwen1.5-32B-Chat across all three datasets.

5.2. Overall Results

  • 34.9%, 34.7%, and 38.0% are the Hits@1 gains over GCN-Align with Qwen1.5-32B-Chat on ZH-EN, JA-EN, and FR-EN, respectively.The corresponding Qwen1.5-14B-Chat gains are 32.9%, 34.0%, and 37.3%.Source: §5.2 · p. 12
  • 97.8%, 95.7%, and 99.2% are LLM-Align's Hits@1 scores with DERA-R and Qwen1.5-14B-Chat on ZH-EN, JA-EN, and FR-EN, respectively.With Qwen1.5-32B-Chat, Hits@1 gains over DERA-R are 3.2%, 2.6%, and 0.4% on those datasets.Source: §5.2 · pp. 12–13
  • LLM-Align with DERA-R and Qwen1.5-32B-Chat achieves the highest Hits@1 among baselines on all three DBP15K datasets.Table 2 reports Hits@1 for LLM-Align because it predicts one target entity per source entity.Source: §5.2 · pp. 12–13, Table 2 · p. 13

5.3. Ablation Study

The ablations report that attribute reasoning, relation reasoning, and multi-round voting contribute to LLM-Align, with the complete framework outperforming reduced-module variants.

5.3. Ablation Study

  • Removing Attribute-based Reasoning reduces Hits@1 by 16.1%, 15.3%, and 14.0% for the 14B model, and by 11.1%, 11.1%, and 2.2% for the 32B model.The passage reports these drops across the three datasets.Source: §5.3 · pp. 13–14
  • Removing Relation-based Reasoning lowers performance by an average of 15.7% for the 14B model, versus 1%-3% for the 32B model across the three datasets.Source: §5.3 · p. 14
  • Removing Multi-round Voting drops performance by an average of 4.3% for the 14B model and 1.2% for the 32B model.The module shuffles option order and requires a majority vote across multiple rounds; the authors report that it can recover the correct answer despite an error in one round.Source: §5.3 · p. 14
  • With only Relation-based Reasoning, Hits@1 falls by an average of 22.7% for the 14B model and 12.8% for the 32B model versus the full framework.Source: §5.3 · p. 14
  • With only Attribute-based Reasoning, average Hits@1 is 92.3% for the 14B model and 96.8% for the 32B model.The passage says performance improves as relation reasoning and voting are incrementally added.Source: §5.3 · pp. 14–15
  • The full framework achieves higher performance than any one- or two-module variant, with the modules described as complementary.The authors connect relation reasoning to adding structural context to attributes, and voting to strengthening confidence in useful information.Source: §5.3 · pp. 14–15

5.4. Analysis on the Impact of Candidate Alignment Orders

Candidate-list order affects alignment results: original ordering performs best, random ordering next, and reverse ordering worst for both reasoning-model sizes.

5.4. Analysis on the Impact of Candidate Alignment Orders

  • Original candidate order performs best, random order next, and reverse order worst for both 14B and 32B models.The comparison covers knowledge-driven and attribute-aware prompts; the authors suggest candidate order may carry implicit cues about the correct answer's position.Source: §5.4 · p. 15
  • LLMs prefer candidates ranked near the top, but retaining similarity order may hurt when a weaker candidate selector places the correct entity near the end.The passage particularly notes this risk for candidate-matching models with low Hits@1.Source: §5.4 · pp. 15–16

5.5. Analysis on the Impact of LLM Size

With DERA-R supplying 10 candidates, larger reasoning models show an upward Hits@1 trend across all three datasets. The increase is greater for high-difficulty entities, while low-difficulty entities retain higher overall performance.

5.5. Analysis on the Impact of LLM Size

  • The experiments compare 1.5B, 14B, and 32B reasoning models using DERA-R candidate sets of size 10.The candidate alignments are generated with DERA-R.Source: §5.5 · p. 16
  • Hits@1 rises across all three datasets as reasoning-model size increases from 1.5B to 32B.The 1.5B model scores around 9% across datasets, close to random selection with 10 candidates.Source: §5.5 · pp. 16–17, Figure 4 · p. 17
  • High-difficulty entities gain more from increasing model size than low-difficulty entities, though low-difficulty performance remains higher overall.Difficulty is defined by whether DERA-R ranks the correct entity first; results average three runs on 300 samples from each difficulty set.Source: §5.5 · pp. 17–18

5.6. Analysis on the Impact of Candidate Size

Hits@1 declines as candidate-set size grows for both 14B and 32B models, although 32B generally performs better. On FR-EN, similar-name errors make 32B match or slightly underperform 14B at candidate sizes 20, 40, and 50.

5.6. Analysis on the Impact of Candidate Size

  • The candidate-size experiment removes the framework’s three modules and uses knowledge-driven prompts on samples where the correct entity is in the top-k set.It uses 500 test samples and averages three sampling experiments per candidate-set size.Source: §5.6 · pp. 18–19
  • Hits@1 decreases as candidate-set size increases for both 14B and 32B models.The study varies candidate sets across 10, 20, 30, 40, and 50 entities.Source: §5.6 · pp. 18, 20, Figure 6 · p. 20
  • On FR-EN, 32B matches or slightly underperforms 14B at candidate sizes 20, 40, and 50, with most errors involving similar-name entities.The 32B model attempted to reason between similar entities, while 14B tended to output the similar entity directly.Source: §5.6 · pp. 19–20

6. Conclusions

LLM-Align prompts an LLM with candidate alignments and selected entity information to infer final matches. Experiments on three EA datasets report improved results for existing models and the best compared performance when paired with a robust candidate-selection model.

6. Conclusions

  • LLM-Align uses heuristic selection of key entity attributes and relationships, then incorporates selected triples into LLM prompts.Source: §6 · pp. 20–21
  • A multi-round voting mechanism is designed to mitigate position bias and address LLM hallucinations.Source: §6 · p. 21
  • Across three EA datasets, LLM-Align enhances existing models and achieves the best compared performance with a robust candidate-selection model.Source: §6 · p. 21
Loading 2412.04690v1…