Source-linked AI summary
Decoupling Error Attribution in Cloud-Native Graph-RAG: A Data Integrity Diagnostic Framework
Shuai Yan, Yuhang Wu, Xiaodong Huang, Ke Wang
TL;DR
Graph-RAG evaluations often assume pristine knowledge graphs and provide limited separation of query-generation errors from underlying data defects. This paper proposes a decoupled diagnostic framework to quantify data-quality losses. Across the reported defects, data quality is the dominant bottleneck, while PKME can mask degradation from monitoring.
Problem
End-to-end Graph-RAG testing cannot distinguish query-engine failures from KG defects, and many evaluations assume pristine graph data.
Source: §I · p. 1Method
The paper proposes a decoupled data-quality audit framework that orthogonally isolates quantitative losses caused by underlying KG defects.
Source: §I · p. 1Results
At a 20% defect injection rate, accuracy falls from the 0.93 baseline to 0.39–0.62, with structural defects reaching 0.39–0.45 versus 0.58–0.62 for numeric defects.
Source: §VI · p. 6Takeaways & Limitations
PKME can shrink apparent Cypher error by over 70%, increasing the risk of false negatives in automated monitoring scripts.
Source: §VI · p. 6
Abstract
from arXiv · showhide
Graph-RAG systems often assume pristine data quality, overlooking the severe impact of perturbations in cloud-native databases. This paper proposes a three-layer decoupled diagnostic framework to orthogonally attribute system errors to reasoning loss, Knowledge Graph (KG) defects, and Cypher generation errors. Evaluated on a spatio-temporal ecological KG of the Southeastern Tibet region with eight defect types, results reveal that data integrity, rather than algorithmic reasoning, is the dominant performance bottleneck, with structural defects degrading system accuracy from 0.93 to 0.39. Crucially, we observe a masking-like phenomenon termed the Parametric Knowledge Masking Effect (PKME), suggesting LLMs compensate for broken retrieval paths using internal memory. This shrinks apparent query generation errors by over 70 percent, obscuring actual storage deterioration and increasing the risk of false negatives for automated monitoring. This work provides a quantitative foundation for auditing and optimizing data integrity in cloud-based information fusion systems.
I. INTRODUCTION
Graph-RAG evaluation often assumes pristine knowledge graphs and cannot distinguish query-engine errors from damaged graph data, limiting data-quality auditing. This paper proposes a decoupled audit framework and a dynamic ecological KG benchmark, and identifies PKME as an observability challenge.
I. INTRODUCTION
- End-to-end testing cannot distinguish Cypher-generation failures from underlying KG defects, while many studies assume pristine graph data.This entanglement makes it difficult for data engineers to audit quality and optimize cloud-based graph storage.Source: §I · p. 1
- The proposed audit framework orthogonally isolates quantitative system losses caused purely by underlying data defects (εKG).The paper frames this as an early attempt to decouple Graph-RAG error sources from a data-integrity perspective.Source: §I · p. 1
- The benchmark is a dynamic ecological knowledge graph for Southeastern Tibet, built from real-world data for controlled defect-injection analysis.The graph addresses spatio-temporal ecological data integration, for which high-quality graph assets are described as a critical foundation.Source: §I · p. 1
- Non-monotonic error propagation suggests PKME can obscure true data degradation and challenge system observability.The effect is presented as a contribution alongside the decoupled audit framework and benchmark.Source: §I · p. 2
II. RELATED WORK
Prior KG quality management emphasizes static integrity checks, while Graph-RAG evaluation largely emphasizes end-to-end outputs. These approaches leave limited support for dynamically tracing storage defects through retrieval pipelines or attributing errors to graph data versus query and reasoning.
II. RELATED WORK
- Conventional KG quality management focuses on static anomaly detection, schema validation, and logical consistency, but not dynamic defect propagation through cloud retrieval pipelines.These methods provide foundational integrity constraints but do not assess how localized storage defects propagate through complex retrieval.Source: §II · p. 2
- Graph-RAG evaluation predominantly measures end-to-end outputs or hallucinations, leaving DBAs unable to distinguish query limitations from database corruption.Text-oriented frameworks measure generation faithfulness and answer relevance but lack structural graph awareness.Source: §II · p. 2
- Existing evaluation paradigms generally treat KGs as pristine assets, overlooking degraded data conditions in real-world cloud deployments.This limits fine-grained error attribution in cloud-based graph systems.Source: §II · p. 2
A. Spatio-Temporal Knowledge Graph Construction
The study constructs a dense spatio-temporal ecological KG for Southeastern Tibet, pairs it with a manually designed question-answering set, and evaluates it under controlled graph defects. The defect scheme separates topology changes from attribute corruption and uses a 20% injection rate as a stress test, not an estimate of everyday conditions.
A. Spatio-Temporal Knowledge Graph Construction
- The Southeastern Tibet graph fuses NDVI, AGB, and DEM data across 2015 and 2021, producing 1,172 nodes and 3,428 relationships.Its dense topology supports exhaustive, fine-grained logical-integrity auditing.Source: A. Spatio-Temporal Knowledge Graph… · p. 2
B. Question-Answering Dataset Design
- The evaluation set contains 100 questions across four retrieval scenarios, with 54% requiring 3-hop retrieval; gold Cypher queries, answers, and subgraphs are recorded.Qianwen-Max at temperature 0 serves as judge, scoring answers with nine rule categories and binary labels, plus 0.5 for ranking questions.Source: B. Question-Answering Dataset Design · p. 2
C. Controlled Defect Injection Scheme
- Eight independently generated defective graphs each receive one defect type at a fixed 20% rate, selected to stress-test boundary conditions rather than model daily averages.Structural defects alter graph topology to simulate connectivity failures; numeric defects corrupt node attributes to reflect ingestion anomalies.Source: C. Controlled Defect Injection Scheme · p. 2
IV. THREE-LAYER DECOUPLED EVALUATION FRAMEWORK
The framework uses control variables to separate KG defects, Cypher generation, and LLM reasoning in a white-box diagnostic scheme. Its three layers vary graph condition and query source to measure ideal downstream reasoning, graph-defect impact, and whole-system performance under degradation.
A. Design Motivation
- A control-variable strategy fixes parts of the retrieval pipeline while activating one error source at a time, separating graph quality, schema alignment, and LLM reasoning losses.This turns end-to-end benchmarking into structured, white-box error attribution.Source: A. Design Motivation · pp. 2–3
B. Three-Layer Framework Definition
- Oracle Evidence runs gold Cypher on the clean graph and supplies perfect textual evidence to measure the upper bound of downstream reasoning.This controls for data integrity and query generation.Source: B. Three-Layer Framework Definition · p. 3
- Oracle Cypher runs gold Cypher on a defective graph to quantify KG retrieval-path damage, while End-to-End uses LLM-generated Cypher on that graph to audit the full degraded system.The two layers distinguish graph-defect impact from performance when query generation and retrieval both operate under degraded conditions.Source: B. Three-Layer Framework Definition · p. 3
C. Error Attribution Formulas
The section defines an orthogonal decomposition of total error into reasoning, KG-defect, and Cypher-generation components, then describes how defect severity can mask data degradation.
C. Error Attribution Formulas
- OCclean is gold-Cypher accuracy on the clean graph (0.93); OE and OCclean use the clean graph, while OC and E2E use defective graphs.The layer accuracy is defined as the mean binary score over questions.Source: C. Error Attribution Formulas · p. 3
- Total error is decomposed into reasoning loss (OE−OCclean), KG defect error (OCclean−OC), and Cypher generation error (OC−E2E).The components map respectively to the algorithmic bottleneck, storage integrity loss, and schema-alignment failure.Source: C. Error Attribution Formulas · p. 3
- The decomposition assumes intrinsic reasoning is independent of retrieved-context quality, making εreason an invariant reference; εreason and εKG are nonnegative, and εcypher is empirically nonnegative.The latter is observed because E2E does not outperform OC on defective graphs.Source: C. Error Attribution Formulas · p. 3
- 0.18 clean-graph εcypher shrinks to 0.01–0.05 under severe structural defects, while Evidence Gap reaches MI 72.2%, illustrating defect-severity-dependent masking.The authors hypothesize that failed retrieval paths prompt the LLM to use internal parametric knowledge, masking apparent schema misalignment without eliminating εKG.Source: C. Error Attribution Formulas · pp. 3–4
D. Statistical Analysis Methods
The analysis sets a significance threshold, defines the effect-size convention, and specifies the model configuration and evaluation workload.
D. Statistical Analysis Methods
- The statistical significance threshold is α = 0.05.Source: D. Statistical Analysis Methods · p. 4
- Effect size uses Cohen’s d, computed as the standardized mean difference of paired scores; d ≥ 0.8 is large, and positive d favors the first layer.Source: D. Statistical Analysis Methods · p. 4
- The study ran 2,700 inference calls across nine databases, 100 queries, and three layers, using Qianwen models for Cypher generation, answering, and judging.The system used 40 concurrent worker threads; generation temperature was 0.2, and judge temperature was 0.Source: D. Statistical Analysis Methods · p. 4
A. Performance Boundary Analysis
The clean-graph comparison indicates little measured reasoning loss, while defect type strongly differentiates retrieval accuracy and structural defects produce especially small apparent Cypher error.
A. Performance Boundary Analysis
- 0.93 OC clean versus 0.94 OE (p = 0.706, nonsignificant) supports the stated conclusion that data quality may outweigh the algorithmic bottleneck.The comparison is reported for the baseline condition.Source: A. Performance Boundary Analysis · p. 4
- 0.39–0.45 OC accuracy under structural defects contrasts with 0.58–0.62 under numeric defects; structural defects break retrieval connectivity, while numeric defects affect attributes.The reported declines relative to OC clean are more than 50% for structural defects and 33%–38% for numeric defects.Source: A. Performance Boundary Analysis · p. 4
- 0.05 Evidence Gap and 0.04 Structural Sparsity εcypher compare with 0.18 on the clean graph, providing initial evidence for PKME.For Evidence Gap, OC = 0.42 and E2E = 0.37; for Structural Sparsity, OC = 0.45 and E2E = 0.41.Source: A. Performance Boundary Analysis · p. 4
B. Bottleneck Diagnosis and Error Attribution
Across defect scenarios, KG defects vary sharply with structural integrity while reasoning loss remains stable; parametric knowledge can mask broken retrieval paths and obscure data degradation.
B. Bottleneck Diagnosis and Error Attribution
- 0.54 KG-defect loss under Structural Noise and Temporal Drift exceeds the 0.31 loss in numeric defect scenarios, while reasoning loss remains 0.01 across settings.The stacked analysis decomposes error into reasoning, KG-defect, and Cypher-generation components.Source: B. Bottleneck Diagnosis and Error… · pp. 4–5, Figure 3 · p. 5
- For all eight defect types, KG-defect error is significant (p < 0.001), with structural defects showing large effect sizes and numeric defects medium effect sizes.Structural effect sizes range from d = 0.92 to 1.08; numeric effect sizes range from 0.62 to 0.66.Source: B. Bottleneck Diagnosis and Error… · p. 5
- Cypher-generation error falls by more than 70% in Evidence Gap and Structural Sparsity relative to the clean graph.The respective values are εcypher = 0.05 and 0.04, versus 0.18 on the clean graph.Source: B. Bottleneck Diagnosis and Error… · p. 5
- When retrieval paths break and queries return empty results, the LLM may use internal parametric knowledge, shrinking apparent Cypher error while end-to-end accuracy still declines.This compensatory behavior can mask structural defects in the graph.Source: B. Bottleneck Diagnosis and Error… · p. 5
- Conventional monitoring based solely on end-to-end accuracy may miss severe data errors, motivating decoupled diagnostics to reveal storage corruption.The masking effect can make such errors less detectable during routine monitoring.Source: B. Bottleneck Diagnosis and Error… · p. 5
C. System Robustness Evaluation
End-to-end evaluation appears more stable across graph-quality conditions than Oracle Cypher, consistent with parametric buffering masking retrieval instability. Medium- and low-stability questions form a substantial audit target.
C. System Robustness Evaluation
- 25% of E2E questions have Complete Stability, compared with 17% for Oracle Cypher; Low Stability accounts for 35% and 46%, respectively.The figure compares question distributions across four stability tiers for the two evaluation settings.Source: C. System Robustness Evaluation · p. 5, Figure 4 · p. 6
- In the low-score region (≤0.53), fewer E2E questions accumulate in Low Stability than Oracle Cypher questions, consistent with parametric buffering obscuring data deterioration.The cumulative distribution function illustrates this distribution shift.Source: C. System Robustness Evaluation · p. 5
- Medium- and Low-Stability questions together make up 57% of the dataset and vary substantially across defective graphs, making them targets for future audits and robustness optimization.The proposed targets concern cloud-based graph systems.Source: C. System Robustness Evaluation · pp. 5–6
VI. CONCLUSION AND FUTURE WORK
The evaluation identifies data quality as the dominant bottleneck in the tested Graph-RAG pipeline, with structural defects especially damaging and PKME complicating error observability. The framework is intended for database integrity monitoring, while broader model, deployment, and geographic validation remains future work.
VI. CONCLUSION AND FUTURE WORK
- The framework is evaluated across eight data-quality perturbations and is designed to support quantitative integrity monitoring during cloud deployment and large-scale multi-source data fusion.Future work proposes heterogeneous-model verification, real-time defect interception through cloud-platform integration, and larger information-fusion tasks across the Qinghai-Tibet Plateau.Source: §VI · p. 6
- At a 20% defect injection rate, system accuracy falls from 0.93 to 0.39–0.62, and data quality loss is significant across all eight defect types.The maximum effect size is Cohen’s d = 1.08; data asset quality, rather than algorithmic reasoning, is the dominant bottleneck in the evaluated pipeline.Source: §VI · p. 6
- Structural defects reduce accuracy to 0.39–0.45, versus 0.58–0.62 for numeric defects, causing roughly twice the damage.The authors conclude that topological reachability warrants greater priority than numeric node-attribute correction during ingestion and integration.Source: §VI · p. 6
- With reasoning loss insignificant (p = 0.706), severe structural defects shrink εcypher and produce a Masking Index above 70%.The authors warn that PKME increases false-negative risk in automated monitoring and can leave production data degradation undetected without decoupled diagnostics.Source: §VI · p. 6