Source-linked AI summary
MissDiag: Diagnostic Evaluation of Incomplete-Knowledge Robustness in KGQA and KG-RAG
Hang Wang, Hang Dong, Lu Liu, Chuanru Ren
TL;DR
Incomplete knowledge makes aggregate robustness scores difficult to interpret because they conflate missing evidence, system behavior, and evaluation protocol. MissDiag applies typed missingness interventions to fixed question–answer support graphs, finding that answer-adjacent evidence loss causes the largest degradation while source-context removal can be neutral or beneficial.
Problem
Aggregate robustness changes conflate missing evidence, system behavior, and evaluation protocol, making incomplete-knowledge degradation difficult to interpret.
Method
MissDiag fixes the question, gold answer, and support-graph nodes while applying severity-controlled typed edge removals and comparing paired system outputs.
Results
Answer-adjacent evidence loss produces the largest and most consistent degradation, whereas source-context removal is often neutral or beneficial across evaluated system families.
Takeaways & Limitations
Typed degradation profiles make cross-system robustness comparisons more interpretable than complete-support accuracy or a single aggregate missing-evidence score.
Takeaways & Limitations
LLM-based comparisons assume a shared backbone, with graph access restricted to the transformed local support graph.
Abstract
from arXiv · showhide
Knowledge graph question answering (KGQA) and knowledge-graph-based retrieval-augmented generation (KG-RAG) aim to ground answers in explicit graph evidence, but real-world knowledge graphs are often sparse, outdated, and incomplete. Existing robustness evaluations usually report aggregate changes in answer quality after evidence is removed or perturbed, which measures sensitivity to incomplete support but leaves the source of degradation under-specified: the same score change can conflate the type of missing evidence, the response of the evaluated system, and the sensitivity of the answer-matching protocol. To address this gap, we propose \textbf{MissDiag}, a diagnostic evaluation framework for incomplete-knowledge robustness in KGQA and KG-RAG. MissDiag keeps the question and gold answer fixed while applying structurally typed missingness interventions to benchmark-provided support graphs, enabling paired comparisons that decompose robustness changes by evidence type, system response, and evaluation protocol rather than reducing them to a single aggregate score drop. Experiments across multiple system families show that incomplete-knowledge robustness is better understood as a typed degradation phenomenon than as a uniform property: answer-adjacent evidence loss produces the largest observed degradation, source-context removal is often neutral and can be beneficial, and semantic answer matching changes absolute scores while preserving the main typed degradation patterns. By transforming aggregate robustness measurement into typed diagnostic attribution, MissDiag provides a more interpretable basis for comparing, diagnosing, and stress-testing KGQA and KG-RAG systems under incomplete knowledge.
Introduction
MissDiag reframes incomplete-knowledge robustness in KGQA and KG-RAG as a typed diagnostic attribution problem rather than a single aggregate score change. It preserves paired questions and gold answers while varying structurally distinct support-graph missingness conditions to separate evidence loss, system behavior, and evaluation sensitivity.
- Motivation: Aggregate score drops alone cannot identify which evidence type was removed, whether answer-local evidence was affected, or whether degradation is systematic.The same QA instance can show similar aggregate degradation under structurally different missingness conditions.
- Method: MissDiag keeps the question and gold answer fixed while transforming each benchmark support graph into structurally distinct incomplete-evidence conditions.Its primary missingness operators include random support loss and source-context removal.
- Framework: MissDiag reports degradation profiles indexed by missingness type, severity, system, and evaluation metric, with structural slices for further analysis.This represents incomplete knowledge as a typed degradation phenomenon rather than a single robustness condition.
- Framework: The framework enables paired comparisons across trained KGQA models, graph-structured prompting methods, iterative KG agents, and direct LLM baselines.These comparisons examine whether degradation reflects genuine evidence loss, system behavior, or evaluation sensitivity.
- Contribution: MissDiag formulates incomplete-knowledge robustness evaluation as diagnostic attribution because aggregate changes conflate evidence availability, system behavior, and evaluation protocol.The contribution argues that aggregate score changes alone cannot support reliable conclusions about reasoning robustness.
Related Work
Prior KGQA research progressed from semantic parsing and query-graph construction toward retrieving and reasoning over local evidence subgraphs. Work on incomplete knowledge emphasizes recovering or maintaining answer quality, while MissDiag instead diagnoses robustness beyond a single aggregate degradation score.
- KGQA foundations: KGQA evolved from mapping questions to executable logical forms or graph queries toward retrieving and reasoning over local evidence subgraphs.This progression makes the support structure part of the answering process.
- Incomplete knowledge: Prior incomplete-knowledge approaches reason over partial graphs, add textual evidence, complete graphs, or use LLM-centered reasoning to preserve answer quality.These approaches primarily ask how to recover or maintain answer quality when knowledge is missing.
- Evaluation limitations: KGQA evaluation conclusions depend on dataset construction, evidence availability, and answer-matching protocols, amid annotation issues, heterogeneous difficulty, and inconsistent reporting.These factors make robustness claims sensitive to evaluation design as well as system behavior.
- MissDiag’s diagnostic gap: MissDiag decomposes incomplete-knowledge evaluation by missingness type, system response, severity, and metric sensitivity rather than relying on a single aggregate degradation score.This framing makes robustness claims interpretable beyond one overall score change.
Method
MissDiag evaluates incomplete-knowledge robustness through paired, severity-controlled interventions on fixed question, answer, system, and support-graph conditions. Its typed degradation profiles separate evidence-loss structure, system response, evaluation metric, and feasibility rather than collapsing robustness into one aggregate score.
- Evaluation design: MissDiag fixes the question, gold answer, and evaluated system while transforming only support edges and comparing outputs with complete support.This produces paired degradation profiles indexed by missingness type, severity, and metric.
- Graph construction: The local support graph converts triples, proof paths, or retrieved subgraphs into a labeled directed graph, with distances computed on its undirected projection.Source entities and graph-aligned gold-answer entities define structural reference sets for the interventions.
- Intervention control: Optional source-anchored context edges are selected deterministically before intervention and remain fixed across conditions, ensuring every operator starts from the same support graph.After intervention, systems receive only retained support evidence; isolated nodes are not exposed as additional answer hints in generative KG-RAG prompts.
- Missingness operators: Four typed operators probe random, source-context, relation-level, and answer-adjacent evidence loss under a shared removal budget.They are not mutually exclusive categories: random removal provides a quantity-matched baseline, while the other operators target source-side, relation-family, or answer-local evidence.
- Diagnostic output: The main output is a typed degradation profile that aggregates paired score changes by missingness type and severity for each evaluation metric.Positive degradation denotes performance loss, negative degradation denotes improvement, and infeasible instances are excluded from the corresponding aggregation.
Experiments
Experiments evaluate MissDiag through paired, fixed-input comparisons of typed missingness across system families, severity levels, metrics, and structural slices. Results show that degradation varies by missing evidence type, system behavior, and evaluation protocol rather than following a uniform pattern.
- Evaluation setup: The evaluation uses 1,050 paired development examples, holding questions, gold answers, source entities, and node inventories fixed while changing retained support edges.The main comparison uses severity α = 0.3 and compares complete support with random, source-context, relation-level, and answer-adjacent missingness.
- Typed degradation profiles: Answer-adjacent removal is the dominant degradation condition for every system, with drops ranging from 10.3 to 21.3 F1 points.Random and relation-level removal usually produce intermediate degradation, while source-context removal is often neutral or beneficial.
- Severity effects: Increasing severity amplifies degradation, with answer-adjacent removal remaining strongest at every tested severity and growing most sharply as α increases.Severity is evaluated at α ∈{0.1, 0.3, 0.5}, while the question, gold answer, metric, and operators remain fixed.
- Metric sensitivity: Semantic matching changes degradation magnitudes but preserves the main typed pattern: answer-adjacent removal remains largest for all four direct LLM baselines.The comparison evaluates exact F1 and semantic F1 at α = 0.3; semantic matching does not reverse the ordering of missingness effects.
- Structural slices: Answer-adjacent degradation is strongest for multi-answer questions and small-support graphs, with gaps over random removal of 19.9 and 14.2 points, respectively.The gap decreases from 14.2 points on small-support graphs to 4.5 points on large-support graphs, and from 19.9 points for multi-answer to 7.6 points for single-answer examples.
Discussion
MissDiag shows that incomplete-knowledge robustness is better understood through typed degradation profiles than a single aggregate score. Answer-adjacent evidence loss dominates degradation, while typed profiles reveal distinct failure mechanisms and clarify cross-system differences.
- Typed degradation: Typed degradation profiles distinguish failure mechanisms that aggregate missing-evidence scores would collapse.The experiments characterize robustness by evidence type rather than a single aggregate change.
- Typed degradation: Answer-adjacent evidence removal consistently produces the largest degradation, indicating strong dependence on evidence near aligned answer entities.This dominant effect is amplified or preserved across severity, metric, and structural analyses.
- Cross-system comparison: Higher complete-support F1 does not necessarily imply stronger robustness across all missingness types.Direct LLM baselines can combine strong complete-support performance with large answer-adjacent degradation, while some trained KGQA models show negative degradation after source-context removal.
Conclusion
MissDiag frames incomplete-knowledge evaluation in KGQA and KG-RAG as a diagnostic problem rather than an uninterpretable score-drop measurement. It controls key benchmark elements while applying severity-controlled, typed missingness to support edges.
- Conclusion: Incomplete-knowledge score drops can conflate missing evidence, system response, and evaluation protocol.This makes aggregate degradation difficult to interpret directly.
- Conclusion: MissDiag keeps the question, gold answer, source entities, and node inventory fixed during evaluation.This enables controlled comparisons while varying the missingness intervention.
- Conclusion: MissDiag applies severity-controlled typed missingness operators to support edges.The framework is designed to diagnose incomplete-knowledge effects across KGQA and KG-RAG evaluation.