Source-linked AI summary
From Extraction to Governed Memory: Multi-Agent Knowledge Graph Construction with Domain-Expert Review
Pranav Bykampadi, Neel Mokaria, Vishesh Narayan, Faizan Wajid, Ashok Agrawala
TL;DR
Agentic knowledge graphs often lack ownership, evidence, and governed admission, limiting their use as controlled memory. MAGG constructs Governed Knowledge Graphs through multi-agent extraction, domain ownership, evidence review, and audited decisions, then reuses that structure for downstream reasoning. Across the reported evaluations, governance improves graph quality and question-answering performance, while the method retains computational, model-transfer, and validation limitations.
Problem
Flat knowledge-graph pipelines lack explicit ownership, evidence, admission paths, and auditability for agentic knowledge construction and use.
Method
MAGG uses multi-agent extraction with open-world schema induction, domain-owned evidence review, governed admission, audit metadata, and domain-routed question answering.
Results
MAGG improves admitted graph quality over flat insertion and improves downstream multi-hop question answering over a graph-based baseline.
Takeaways & Limitations
Governance functions as a graph-level mechanism organizing knowledge creation, update, audit, and downstream reasoning rather than merely as a post-hoc filter.
Takeaways & Limitations
The evaluation uses GPT-5, and transfer to other LLM families remains open; open-world and human-validation results are preliminary.
Abstract
from arXiv · showhide
Knowledge graphs used by agentic systems are often treated as flat stores of extracted triples, with little record of who owns a fact, why it was admitted, or how it should be used downstream. We argue that reliable agentic knowledge systems require governance as an essential component of graph construction to bridge this gap. We propose MAGG, a principled multi-agent framework for constructing Governed Knowledge Graphs that introduces explicit governance decisions for reliable and trustworthy knowledge sharing. A domain classifier first induces entity and relation types directly from document content, enabling operation in open-world settings without fixed schemas. Candidate triples are assigned to domain owners, reviewed against supporting evidence, admitted through governance decisions, and stored with audit metadata. The same ownership structure is reused during question answering, where queries are routed to domain-specific graph experts rather than answered through undifferentiated retrieval. Our evaluation demonstrates MAGG's effectiveness: On SciERC, MAGG improves strict triple F1 by 47% and mapped triple F1 by 51% over flat insertion. A blinded review of 120 triples finds governed-only triples more often source-supported than flat-only ones, and revised triples supported in 100% of cases. Finally, on MuSiQue, MAGG outperforms Microsoft GraphRAG by 9.0 exact-match points and 11.2 token-F1 points.
1 Introduction
MAGG treats knowledge graphs for agentic systems as governed memory rather than flat stores, adding ownership, evidence, review, and auditability. It reuses this organizational structure for domain-aware downstream reasoning and evaluates the approach across construction and question answering.
- Flat knowledge graphs do not record fact ownership, admission evidence, conflict resolution, or which graph portion downstream applications should consult.
- MAGG admits facts through explicit ownership, evidence review, provenance decisions, and LLM-based Domain Expert Agents that control domain-specific memory.
- The framework coordinates creation, governance, and application layers, including deliberative extraction, governed admission, domain-routed question answering, and incremental self-organization.
- MAGG evaluates governed graphs in fixed-schema and open-world settings, with experiments covering extraction quality, governance ablations, human review, and downstream question answering.
2 Related Work
Prior work improved extraction coordination, deliberation, and graph-grounded retrieval, but generally separated these capabilities from construction governance. MAGG addresses the missing organizational layer by preserving ownership, provenance, and admission decisions in memory.
- Earlier knowledge-graph pipelines decomposed extraction into supervised subtasks, while generative architectures increasingly collapsed them into single-pass operations.
- Multi-agent systems introduced deliberation and specialized roles to reduce hallucinations, align schemas, and resolve conflicts during knowledge-graph construction.
- Existing approaches can verify extractions into a global store while discarding domain ownership, extraction provenance, and the governance rules authorizing admission.
- GraphRAG and iterative traversal improve graph-grounded multi-hop retrieval but treat the graph as a static, topology-based artifact without construction provenance.
- Governed graphs trace claims to the responsible domain owner and the governance decision that authorized their admission.
3 Methodology
MAGG constructs a Governed Knowledge Graph by assigning ownership, reviewing evidence, and recording admission decisions instead of directly inserting extracted triples. Its sequential multi-agent pipeline supports open-world schema induction, uncertainty deliberation, evidence grounding, verification, and audited domain routing.
- 3.1 Governed Knowledge Graph: A Governed Knowledge Graph stores candidate and admitted triples together with entity ownership, governance decisions, provenance, and an audit log.
- 3.1 Governed Knowledge Graph: Admission uses a two-phase protocol: propose a triple, obtain a governance decision, and commit only approved, revised, or auto-approved candidates.
- 3.1 Governed Knowledge Graph: The ownership function permits one entity to belong to multiple domains, supporting cross-domain graph organization.
- 3.2 System Overview: The creation pipeline segments documents, induces fixed or open-world schemas, extracts entities and relations, links triples to source evidence, and verifies candidates before admission review.
- 3.3 Deliberation, Blackboard, and Voting: Uncertain or conflicting outputs enter a deliberation path, where hypotheses retain proposed content, confidence, posting agent, and evidence for peer review before governance.
- 3.3 Deliberation, Blackboard, and Voting: The pipeline is sequential by default, uses a short-lived blackboard for hypotheses and vote rationales, and activates deliberation when confidence falls below 0.7.
4 Results
MAGG evaluates governed admission across extraction quality, human validation, and downstream question answering. The results indicate that ownership-aware review improves graph quality and supports stronger multi-hop QA than flat or centralized alternatives.
- Evaluation settings: The evaluation covers extraction quality and governance ablation, blinded human triple review, and downstream multi-hop QA.SciERC evaluates admitted-graph quality, while human review tests source support and MuSiQue tests open-world question answering.
- Extraction Quality and Governance Ablation: Source-grounded flat insertion barely changes strict F1 from 0.106 to 0.107, while a global reviewer raises strict F1 to 0.143.These comparisons show that evidence linking alone is insufficient and centralized review only partially reproduces MAGG’s gain.
- Extraction Quality and Governance Ablation: MAGG achieves strict F1 0.156 and mapped F1 0.290, improving over direct insertion by 47% and 51%, respectively.It admits 1,077 triples and records the strongest precision under both scoring regimes.
- Human Triple Review: Governed-only triples receive support or partial-support judgments in 85.0% of cases versus 62.5% for FlatKG-only triples.Revised triples are supported or partially supported in 100.0% of cases and useful or potentially useful in 95.0%.
- Human Triple Review: Human validation finds governed-only triples more useful or potentially useful for QA than FlatKG-only triples, at 77.5% versus 62.5%.Shared triples show 85.0% support and usefulness, indicating a high-confidence core admitted by both systems.
- Downstream QA on MuSiQue: MAGG reaches 0.510 exact match and 0.646 token F1 on MuSiQue, versus 0.420 and 0.534 for Microsoft GraphRAG.The QA stack is identical in the Flat KG-QA comparison, while the underlying graph changes from flat-admitted to governed.
5 Conclusion
MAGG constructs Governed Knowledge Graphs by admitting triples through ownership, evidence review, governance decisions, and audit records, then reuses ownership for question answering and incremental updates. Across SciERC and MuSiQue, governance improves graph quality and supports downstream graph-based question answering.
- MAGG admits candidate triples through ownership, evidence review, governance decisions, and audit records instead of flat insertion.
- The same ownership structure supports domain-routed question answering and incremental graph update.
- Across SciERC, governance improves admitted triple quality over flat insertion, beyond source filtering or a generic global reviewer alone.
- Blinded human review suggests governed and revised triples are more often source-supported and useful.
- On MuSiQue, MAGG improves over a graph-based question-answering baseline in a supporting-only multi-hop setting.
Limitations
MAGG improves admitted graph quality but does not solve extraction, and its reported validation and transfer evidence remains limited in scope.
- MAGG improves admitted graph quality but does not solve extraction.Strict triple F1 remains modest because exact matching, normalization, relation labeling, and evidence grounding remain difficult.
- The experiments use GPT-5, leaving transfer of governance gains to other LLM families open.
- Open-world and human-validation results remain preliminary, with no separate evaluation of induced domain quality and limited review coverage.The review should be expanded across more annotators, corpora, and domains.
- MAGG introduces additional computational cost because domain-routed governance and domain-local memory require more LLM calls and prompt tokens than flat insertion.
A Token and Runtime Cost
MAGG trades additional governance calls for a cleaner, more auditable graph, while its usage logs provide exact aggregate API-call and token counts but not fully separated extraction costs.
- MAGG trades additional governance calls for a cleaner and more auditable graph.
- Table 6 reports exact API call and token counts from JSONL usage logs.
- Current usage logs do not separate every internal extraction sub-agent into distinct cost buckets.Entity extraction, relation extraction, evidence linking, and extraction deliberation are combined in the extraction-build log, while governance review is separately logged when replayed.
B Shared Memory
MAGG uses shared memory to coordinate document-level extraction and maintain knowledge across documents, separating short-lived pipeline state from persistent graph and audit-related state.
- MAGG uses shared memory both for extraction within a document and for consistency across documents.The separation reflects the need to handle short-lived local uncertainty alongside longer-lived accumulated knowledge.
- Working memory stores active-document state, including candidate entities, triples, hypotheses, and vote requests.
- Semantic memory stores accepted graph structure and canonicalized entities for reuse in later extraction, governance, and question answering.
- Working memory is short-lived, whereas semantic and procedural memory persist across documents and episodic memory is consulted mainly to inspect prior decisions.
C Construction Agent Roles and Prompts
MAGG uses a staged construction pipeline in which domain discovery, extraction, evidence linking, deliberation, verification, and organization govern what enters the knowledge graph. Each stage contributes to auditable, evidence-grounded admission of entities and triples.
- C.1 Pipeline Sequence: The pipeline processes each document through DocumentProcessor, DomainClassifier, EntityExtractor, RelationExtractor, EvidenceLinker, VerificationAgent, and KnowledgeOrganizer.The DeliberationCoordinator operates as a side-loop for sub-threshold-confidence items between stages.
- C.2 DocumentProcessor: DocumentProcessor deterministically normalizes raw text, segments it into overlapping coherent chunks, and registers the document for cross-document entity resolution.It uses Unicode NFC normalization and removes control characters.
- C.3 DomainClassifier: DomainClassifier discovers a document’s domain without a predefined taxonomy and generates entity types, relation types, and text-mined few-shot examples for downstream agents.The resulting domain-specific schema is broadcast to extraction agents.
- C.4 EntityExtractor: EntityExtractor performs four-stage entity extraction with integrated coreference resolution, cross-document alias registration, and deliberation for low-confidence entities.It can discover entity types beyond the initial domain schema.
- C.5 RelationExtractor: RelationExtractor identifies relation types, binds heads, and resolves tails through a three-stage Relation–Head–First procedure.Fixed-schema mode constrains relations to benchmark labels, while open-world mode can propose new relation structures.
- C.6 EvidenceLinker: EvidenceLinker attaches sentence-level support, labels evidence as explicit, implicit, or inferred, and uses prior graph knowledge to adjust confidence and flag weak triples.Its outputs include evidence strength, supporting text, consistency status, and adjusted confidence.
- C.7 DeliberationCoordinator: DeliberationCoordinator conducts multi-agent voting for items with confidence 0.35 ≤ c < 0.7, while accepting items at least 0.7 and rejecting items below 0.35 without a vote.The protocol applies to uncertain entities and triples between extraction and validation.
- C.8 VerificationAgent / C.9 KnowledgeOrganizer: VerificationAgent repairs fixable candidate issues, performs source-grounding checks, and drops unsupported or hallucinated triples before KnowledgeOrganizer integrates approved results.KnowledgeOrganizer deduplicates entities, normalizes relations, resolves canonical IDs, and proposes governed-mode triples for admission.
D QA Agent Roles and Prompts
MAGG reuses its governed domain structure during question answering by decomposing questions, routing sub-questions to domain experts, and synthesizing their evidence-grounded responses. The design keeps retrieval and answer generation tied to owned subgraphs and cross-domain bridge context.
- D QA Agent Roles and Prompts: The QA application layer uses three agents: QAOrchestrator, DomainExpertAgents, and SynthesisAgent.Together they route questions, answer from owned subgraphs, and combine domain responses.
- D QA Agent Roles and Prompts: The QA design preserves the construction graph’s ownership structure by routing each sub-question to agents responsible for relevant domain-specific subgraphs.This connects domain ownership during construction with domain-specific retrieval during answering.
- D.1 QAOrchestrator: QAOrchestrator reads the graph’s domain layout, decomposes questions, routes sub-questions to domain experts, gathers bridge context, and invokes synthesis.Its input includes the persistent governed graph and its domain layout.
- D.2 DomainExpertAgent: Each DomainExpertAgent answers from one governed subgraph by retrieving a query-focused slice and optionally traversing multi-hop paths or entity neighborhoods.It produces an evidence-grounded answer with explicit triple citations.
- D.2 DomainExpertAgent: Domain experts are instructed not to speculate beyond supplied evidence and return answer records containing cited triples, estimated coverage, and confidence.Their prompts also prohibit mentioning domain IDs, routing, or the phrase “knowledge graph.”
- D.3 SynthesisAgent: SynthesisAgent combines domain responses and cross-domain bridge context into a final user-facing answer.It returns both evidence-grounded prose and a minimal short-form answer.
E Human Annotation Instructions
The human annotation protocol evaluates triple support, relation and endpoint correctness, usefulness, and revision quality using only the displayed source evidence. Annotators distinguish direct support from plausible but unsupported claims.
- Annotation criteria: Annotators judge triples only from the displayed evidence and mark plausible but indirectly supported claims as partial or no.Usefulness is assessed by whether a triple helps knowledge-graph search, question answering, or reasoning.
- Triple review: The reviewed triple is iterative_deformation –[Used-for]→ reconstruction_process.The annotation questions separately assess source support, relation correctness, endpoint correctness, and usefulness.
- Source evidence: The source evidence describes iterative deformation of a 3-D surface mesh to minimize an objective function within a stereo reconstruction approach.It also states that geometric constraints address cases where image information alone is insufficient for accurate 3-D shape recovery.
- Revision assessment: Annotators record whether a revision improved, preserved, worsened, or left unclear the triple’s quality, with an optional free-text note.The revision judgment is separate from the four core triple-evaluation questions.