Source-linked AI summary
KARMA: Leveraging Multi-Agent LLMs for Automated Knowledge Graph Enrichment
Yuxing Lu, Wei Wu, Xukai Zhao, Rui Peng, Jinzhuo Wang
TL;DR
Scientific literature is expanding faster than knowledge graphs can incorporate its unstructured findings, and enrichment remains difficult across specialized domains. KARMA uses specialized LLM agents to extract, verify, and integrate knowledge. Experiments across three domains demonstrate high-quality enrichment, while the authors note that LLM-based evaluation does not replace expert validation.
Problem
Rapid literature growth widens the gap between unstructured scientific text and structured knowledge graphs, and enrichment remains challenging across specialized domains.
Source: §1 · pp. 1–2Method
KARMA decomposes knowledge graph enrichment into specialized LLM-agent tasks that extract novel triplets from publications and integrate them into an existing graph.
Source: §1 · p. 2, §3.1 · p. 3Results
Takeaways & Limitations
The reported benefits are especially relevant to domains requiring complex semantic understanding and adherence to structured schemas.
Source: §6 · p. 9Evaluation relies primarily on LLM-based metrics rather than direct human expert validation, and metabolomics QA coherence is lower than in proteomics and genomics.
Source: §7 · p. 12
Abstract
from arXiv · showhide
Maintaining comprehensive and up-to-date knowledge graphs (KGs) is critical for modern AI systems, but manual curation struggles to scale with the rapid growth of scientific literature. This paper presents KARMA, a novel framework employing multi-agent large language models (LLMs) to automate KG enrichment through structured analysis of unstructured text. Our approach employs nine collaborative agents, spanning entity discovery, relation extraction, schema alignment, and conflict resolution that iteratively parse documents, verify extracted knowledge, and integrate it into existing graph structures while adhering to domain-specific schema. Experiments on 1,200 PubMed articles from three different domains demonstrate the effectiveness of KARMA in knowledge graph enrichment, with the identification of up to 38,230 new entities while achieving 83.1\% LLM-verified correctness and reducing conflict edges by 18.6\% through multi-layer assessments.
1 Introduction
Scientific literature is growing faster than knowledge graphs can absorb its unstructured findings, while domain-specific terminology and relationships make reliable enrichment difficult. KARMA proposes specialized LLM agents to extract, verify, and integrate knowledge across scientific domains.
1 Introduction
- KGs structure information for reasoning, but the growth of scientific literature widens the gap between unstructured text and structured graph representations.The paper cites over 7 million articles published annually.Source: §1 · p. 1
- Manual curation is reliable but does not scale, while conventional NLP struggles with specialized terminology and context-dependent relationships.Schema alignment, consistency, and conflict resolution are also required, and inaccuracies can be severe in high-stakes applications.Source: §1 · p. 1
- Applying multi-agent systems to KG enrichment remains challenging because of domain adaptation, systematic verification, and integration into heterogeneous graph structures.Source: §1 · p. 2
- KARMA proposes specialized LLM agents with cross-agent verification, domain-adaptive prompting, and modular design for knowledge extraction and integration.Its experiments span three distinct domains.Source: §1 · p. 2
2 Related Work
Prior KG construction methods face trade-offs among precision, recall, domain specificity, supervision needs, schema flexibility, and verification. KARMA combines specialized agents, domain-adaptive prompting, and LLM-based verification to address these challenges.
2.1 Knowledge Graph Construction
- Earlier KG construction relied on rules, while neural methods improved biomedical NER but required 3–5k labeled examples per relation type and did not generalize beyond predefined schemas.LLM approaches permit open schema learning but raise verification concerns, including hallucinations, schema inconsistency, and quadratic costs for full-text processing.Source: §2.1 · p. 2
2.2 Multi-Agent Systems
- Earlier multi-agent systems used predefined pipelines and handcrafted coordination rules, while recent LLM-based systems use task decomposition to improve reliability.Prior KG work reported improved schema alignment on Wikidata benchmarks while maintaining linear time complexity relative to input text length.Source: §2.2 · pp. 2–3
- KARMA combines modular specialized agents, domain-adaptive prompting, and LLM-based verification to mitigate hallucination and schema inconsistency in KG enrichment.Its architecture includes agents for ingestion, summarization, entity recognition, relationship extraction, conflict resolution, and evaluation.Source: §2.2 · p. 3, Figure 2 · p. 3
3 Methodology
KARMA frames knowledge-graph enrichment as extracting novel triplets from publications and integrating them into an existing graph through modular LLM agents. Candidate triplets are verified before integration, and specialized agents handle distinct pipeline tasks.
3.1 Problem Formulation
- KARMA decomposes enrichment into modular LLM-agent tasks that extract novel triplets from publications and verify candidates before adding them to an existing graph.Agents operate in parallel under a Central Controller.Source: §3.1 · p. 3, §3 · p. 3, §3.2 · p. 4
3.3 Ingestion Agents (IA)
- Ingestion agents normalize documents and metadata, while Reader Agents segment text and discard content below a domain-calibrated relevance threshold.Reader Agents assess segments against the current knowledge graph; surviving segments pass to Summarizer Agents.Source: §3.2 · p. 4, §3.3 · p. 4, §3.4 · p. 4
3.5 Summarizer Agents (SA)
- Summarizer Agents condense each Reader Agent segment while retaining critical entities, relations, and domain-specific terms for downstream extraction.The summaries are intended to give extraction agents high-signal, low-noise inputs.Source: §3.5 · p. 4
3.6 Entity Extraction Agents (EEA)
- LLM-based NER identifies topic-related mentions, then dictionary or ontology filtering removes false positives and normalizes names to canonical forms.For example, the paper maps “acetylsalicylic acid” to “Aspirin.”Source: §3.6 · p. 4
3.7 Relationship Extraction Agents (REA)
- Relationship Extraction Agents score candidate relations for normalized entity pairs, retain those meeting a threshold, and allow multiple relations when implied.The selected relations form entity-relation-entity triplets.Source: §3.7 · p. 5
3.8 Schema Alignment Agents (SAA)
- Schema Alignment Agents classify new entities and relations against domain-specific KG types, flagging unmatched items as candidate additions for review.Entity types include Disease, Drug, and Gene.Source: §3.8 · p. 5
3.9 Conflict Resolution Agents (CRA)
- Conflict Resolution Agents use LLM-based debate for logically incompatible triplets, discarding contradictions or queuing them for expert review according to confidence.The conflict check compares a new triplet with an existing relationship between the same entities.Source: §3.9 · p. 5
3.10 Evaluator Agents (EA)
- Evaluator Agents aggregate confidence, clarity, and relevance signals and use a mean score to finalize triplets; the pipeline can incorporate new ontologies, agents, or prompts.The paper describes the resulting multi-agent pipeline as scalable and accurate.Source: §3.10 · pp. 5–6
4 Experimental Setup
KARMA is evaluated as a biomedical proof of concept across genomics, proteomics, and metabolomics, using LLM-based verification and graph-level metrics. The setup compares three LLM backbones and assesses the augmented graph’s structure, correctness, and utility.
4 Experimental Setup
- The proof-of-concept evaluation spans genomics, proteomics, and metabolomics, using LLM-based verification and graph-level metrics rather than a conventional gold-standard dataset.The framework assesses the quality of generated knowledge graphs and their practical utility.Source: §4 · p. 6
4.1 Data Collection
- The PubMed corpus contains 720 genomics papers, 360 proteomics papers, and 120 metabolomics papers, all processed from PDFs by the Ingestion Agent.Source: §4.1 · p. 6
4.2 LLM Backbones
- Three general-purpose LLMs are evaluated as KARMA backbones; agents share one backbone per experiment, and DeepSeek-v3 is used for LLM-based evaluations.The study varies the backbone while minimally modifying prompts for comparability across models and domains.Source: §4.2 · p. 6
4.3 Metrics
- The evaluation framework groups metrics into core metrics, graph statistics, and quality indicators to assess structural integrity, correctness, and practical utility.Source: §4.3 · p. 6
- Core metrics measure mean confidence, clarity, and relevance across newly added triples, capturing their reliability, unambiguity, and domain significance.Together, these metrics assess the intrinsic quality of added knowledge.Source: §4.3 · pp. 6–7
- Graph statistics track newly introduced entities through Coverage Gain and net node-degree increases through Connectivity Gain.Source: §4.3 · p. 7
- Quality indicators include the conflict ratio, LLM-judged triple correctness, plausible answers to domain questions, and two-expert human scoring.Source: §4.3 · p. 7
5 Results
KARMA outperforms a GLM-4 single-agent baseline, with reported accuracy improvements from automatic evaluation and conflict resolution. Results vary by domain and backbone, and the ablation study reports distinct contributions from summarization, conflict resolution, and evaluation.
5.4 Cost Analysis
- Figure 3 compares prompt tokens, completion tokens, and processing time across domains; genomics has mean completion tokens of 550.64, while proteomics has broader processing-time distributions.Proteomics processing time has mean = 96.58 and std = 46.90; the authors report that it correlates with stronger RLC and CQA scores.Source: Figure 3 · p. 8, §5.4 · p. 8
5.1 Overall Evaluation
- 4.6%–14.4% LLM-based accuracy improvement is reported from automatic knowledge evaluation and conflict resolution; KARMA also outperforms the GLM-4 single-agent baseline.Source: §5.1 · pp. 7–8
5.2 Domain-Level Observations.
- Across domains, DeepSeek-v3 reaches ΔCov = 38, 230 with RLC = 0.831 in genomics; proteomics and metabolomics show different leading backbones across metrics.Proteomics: DeepSeek-v3 leads MCon = 0.845 and ΔCon = 1.468, while GLM-4 leads CQA = 0.617; metabolomics: GLM-4 leads Mcla = 0.790, GPT-4o leads RLC = 0.683, and DeepSeek-v3 reaches ΔCon = 1, 752.Source: §5.2 · p. 8
5.3 Analysis of LLM Backbones
- DeepSeek-v3 exceeds GPT-4o’s coverage gain by 3.9× in genomics and 2.3× in metabolomics, while GPT-4o leads correctness and GLM-4 leads in selected domain metrics.DeepSeek-v3 has RLC = 0.831 versus GPT-4o’s 0.880 in genomics; GPT-4o reaches peak RLC scores of 0.880 in genomics and 0.740 in proteomics, while GLM-4 leads metabolomics clarity and proteomics QA coherence.Source: §5.3 · p. 8
5.5 Ablation Study
- Removing the Summarizer reduces genomics CQA by 22.9% and proteomics RLC by 18.2%; removing Conflict Resolution lowers genomics CQA by 4.9%, and omitting Evaluators lowers metabolomics RLC by 9.7%.The ablations use DeepSeek-v3; Table 2 summarizes RLC and CQA impacts by domain.Source: Table 2 · p. 9, §5.5 · p. 9
6 Conclusion
KARMA uses specialized agents to integrate knowledge from scientific literature into knowledge graphs. Experiments across three omics domains support the value of collaboration for complex semantic understanding and schema adherence.
6 Conclusion
- KARMA decomposes enrichment into entity discovery, relationship validation, and conflict resolution for adaptive knowledge integration.Its modular design uses multi-layer assessments and cross-agent verification to reduce the impact of conflicting edges.Source: §6 · p. 9
- Multi-layer assessments and cross-agent verification reduce the impact of conflicting edges.These mechanisms are part of KARMA’s modular design.Source: §6 · p. 9
- Experiments across genomics, proteomics, and metabolomics demonstrate collaboration can overcome single-agent limitations, especially for complex semantics and structured schemas.Source: §6 · p. 9
7 Limitations
KARMA’s evaluation relies primarily on LLM-based metrics rather than direct expert validation, and performance varies across domains. The authors identify expert review for critical biomedical claims and further technical improvements as future needs.
7 Limitations
- KARMA’s evaluation relies primarily on LLM-based metrics rather than direct human expert validation.The metrics include QA coherence and conflict resolution.Source: §7 · p. 12
- Critical biomedical claims require domain-expert verification before application in clinical settings.Source: §7 · p. 12
- Metabolomics QA coherence is 12.4% lower than proteomics and 11.9% lower than genomics.The authors connect this variation to challenges modeling sparse and rare relationships in metabolomics.Source: §7 · p. 12
- The authors identify hybrid neuro-symbolic approaches and optimized agent coordination protocols as future improvement directions.Source: §7 · p. 12
8 Detailed propmts for KARMA agents
The appendix describes prompt templates and the functions of KARMA’s nine specialized LLM-powered agents, which span document processing, extraction, schema alignment, conflict resolution, and evaluation.
8 Detailed propmts for KARMA agents
- Agent prompts use specialized templates emphasizing confidence, clarity, and domain relevance, with sample inputs, outputs, and negative examples where applicable.Source: §8 · p. 12
8.1 Function summaries of different agents
- Ingestion agents retrieve and normalize documents and metadata, while reader agents segment documents, score relevance using KG context, and filter irrelevant content.Source: §8.1 · p. 12
- Summarizer agents preserve entity relationships; entity extraction agents identify and normalize entities, and relationship extraction agents detect overlapping relations between entity pairs.Source: §8.1 · p. 12
- Schema alignment agents map entities and relations to KG types or flag ontology expansion, while conflict resolution agents use LLM debate and evidence aggregation to address contradictions.Source: §8.1 · p. 12
- Evaluator agents compute integration confidence from weighted confidence, relevance, and clarity signals, then apply threshold-based final approval.Source: §8.1 · p. 12
1. Example-based prompting (few-shot).
Agent prompts can be extended with short input-output examples to provide additional context. The examples are intended to improve response accuracy and consistency.
1. Example-based prompting (few-shot).
- Short input-output examples can extend each agent’s prompt with additional context.Source: §1 · p. 12
- Examples for entity extraction may use drug-disease pairs, while conflict-resolution examples may distinguish partial from direct contradictions.Source: §1 · p. 12
- The authors state that prompt examples can improve response accuracy and consistency.Source: §1 · p. 12
2. Negative Examples and Error Correction.
KARMA uses negative examples and clarifications to improve agent robustness, with examples tailored to errors in summarization and entity identification.
2. Negative Examples and Error Correction.
- Negative examples and clarifications can be supplied to agents to increase robustness.Source: §2 · p. 13
- A Summarizer Agent can be taught not to remove important numerical dosage information.Source: §2 · p. 13
- An Entity Extraction Agent can be shown to ignore location references unrelated to the topic, such as “Paris” as a disease.Source: §2 · p. 13
3. Incremental Fine-Tuning and Updates.
This section describes KARMA’s updateable prompts and agent workflows for document prioritization, knowledge extraction and alignment, conflict resolution, and assessment, including a final relevance-scoring prompt and domain-specific KG examples with score distributions.
3. Incremental Fine-Tuning and Updates.
- Modular prompts let KARMA adapt to emerging entities and complex relationships through periodic retraining or prompt updates without redesigning the pipeline.Collectively, the prompts are described as enabling dynamic, scalable, and accurate knowledge enrichment.Source: §3 · p. 13
8.3 Ingestion Agent (IA) Prompt; 8.4 Reader Agent (RA) Prompt
- The ingestion agent normalizes publications and metadata, while the reader segments text and scores its likely value for downstream extraction.The ingestion output includes metadata and content, and the reader lowers scores for purely methodological segments and incomplete or corrupted sections.Source: §8.3 · p. 14, §8.4 · p. 15
8.5 Summarizer Agent (SA) Prompt
- The summarizer condenses high-relevance segments while retaining domain-specific terms and pertinent numeric data for downstream extraction.It may skip or heavily compress segments below a relevance threshold.Source: §8.5 · p. 16
8.6 Entity Extraction Agent (EEA) Prompt; 8.7 Relationship Extraction Agent (REA) Prompt
- Entity extraction identifies biomedical mentions and maps them to ontology identifiers, while relationship extraction produces entity-linked relations and accounts for negation.When entity matches are ambiguous, the prompt allows candidate matches; relationship extraction can allow multiple candidates for ambiguous text and recognizes negation cues.Source: §8.6 · p. 17, §8.7 · p. 18
8.8 Schema Alignment Agent (SAA) Prompt
- Schema alignment maps unfamiliar entities and relations to existing KG types when possible, or proposes new labels with a confidence score.The agent uses lexical similarity, embedding distance, or domain rules to inform its alignment proposals.Source: §8.8 · p. 19
8.9 Conflict Resolution Agent (CRA) Prompt
- Conflict resolution classifies candidate and existing triplets as contradictory, agreeing, or ambiguous, then chooses whether to discard, review, or cautiously integrate.It considers contextual compatibility, such as different dosages or subpopulations, and can prompt manual verification when both conflicting triplets have high confidence.Source: §8.9 · p. 20
8.10 Evaluator Agent (EA) Prompt for Confidence; 8.11 Evaluator Agent (EA) Prompt for Clarity
- Evaluator prompts combine signals to score triplet confidence and clarity, penalizing contradictions or ambiguous terms.A contradiction can force confidence to 0 absent manual override; clarity may use weighted averaging or a logistic function.Source: §8.10 · p. 21, §8.11 · p. 22
8.12 Evaluator Agent (EA) Prompt for Relevance
- The relevance evaluator combines partial scores and penalizes triplets that mismatch the target KG’s domain.Its inputs can include domain-alignment tags such as domainMatch or domainMismatch.Source: §8.12 · p. 23
- The evaluator prompt combines partial relevance scores and domain tags to produce final relevance scores for candidate triplets.Source: §8.12 · p. 23
9.1 Knowledge graph from Genomics articles (Generate using GPT-4o, Examples)
- Figure 4 presents confidence, relevance, and clarity score distributions for extracted genomics knowledge graph triples generated using GPT-4o.Source: Figure 4 · p. 24
9.2 Knowledge graph from Proteomics articles (Generate using DeepSeek-V3, Examples)
- Figure 5 presents confidence, relevance, and clarity score distributions for extracted proteomics knowledge graph triples generated using DeepSeek-V3.Source: Figure 5 · p. 25
9.3 Knowledge graph from Metabolomics articles (Generate using GLM-4, Examples)
- Figure 6 shows the distributions of confidence, relevance, and clarity scores for extracted metabolomics knowledge graph triples from KARMA.Source: Figure 6 · p. 26
- High-clarity metabolomics relationships (clr ≥0.8) typically involve well-characterized biochemical processes, while lower confidence often marks novel or context-dependent findings needing expert validation.Source: §9.3 · p. 26