PaperlayerView exact revision on arXiv

Source-linked AI summary

Propagating construction-time knowledge quality into medical question answering: A framework grounded in clinical guidelines

Jie Hu, Junjie Wang, Shan Lu, Yifang Hu, Gong Cheng, Yun Liu

arXiv:2608.28360v1cs.AI

TL;DR

The paper asks whether construction-time quality information can remain useful for downstream medical QA, addressing the limited connection between KG quality control and inference-time evidence use. It models structural and evidential quality, retains their fused signal and tiers, and propagates them through retrieval and prompting. On Chinese diabetes guidelines, the full framework reduces omission and conflict while improving evidence-grounded and clinician-rated answer quality, although broader generalization remains unestablished.

  • Problem

    Construction-time triple reliability is rarely reused in downstream evidence selection, even though query relevance does not guarantee reliability.

  • Method

    The framework fuses SchemaConf and EvidScore into persistent Q(t) and quality tiers, then uses them for weighted subgraph retrieval, tier-conditioned prompting, and provenance tracing.

  • Results

    The full framework reduces required-knowledge omission from 16.3% to 5.3% and conflicting outputs from 16.3% to 2.7%, with 81.6% evidence-grounded precision and near-zero invalid citations.

  • Takeaways & Limitations

    Quality-weighted retrieval and tier-conditioned prompting provide complementary channels for selecting and presenting guideline evidence in medical QA.

  • Takeaways & Limitations

    Evaluation is limited to Chinese diabetes clinical guidelines, so the results do not establish language- or disease-level generalization and require replication in other clinical domains.

Abstract

from arXiv · show

Large language models have facilitated knowledge graph (KG) construction from clinical guidelines, but extracted triples vary in structural validity and evidential support. Meanwhile, graph-augmented question answering (QA) systems typically optimize query relevance during retrieval, with limited reuse of quality information produced during KG construction. This creates a disconnect between construction-time quality control and inference-time evidence use. We investigate whether construction-time triple quality can serve as a persistent signal for downstream evidence selection and presentation. We propose a quality-aware framework that models structural conformance (SchemaConf) and evidential support (EvidScore) as complementary dimensions and fuses them into a per-triple quality signal, Q(t). Rather than using quality solely for filtering, the framework retains Q(t) and derived quality tiers as graph attributes and propagates them into quality-weighted subgraph retrieval and tier-conditioned evidence prompting, while preserving passage-level provenance. Experiments on Chinese diabetes clinical guidelines show that the utility of the quality signal is distribution dependent. Under cross-version and cross-model shift, the fused Q(t) provides stronger triple-quality discrimination than either component alone (AUC 0.748 vs. 0.703 for EvidScore and 0.645 for SchemaConf). In guideline-grounded QA, propagating construction-time quality reduces required-knowledge omission from 16.3% to 5.3% and conflicting outputs from 16.3% to 2.7%, with an evidence-grounded precision of 81.6% and near-zero invalid citations. Blinded clinician ratings favor the full framework over no retrieval (4.68 vs. 4.21 on a five-point scale) and approach the oracle condition (4.80), while cross-generator experiments show consistent trends.

1. Introduction

The paper addresses the disconnect between unreliable LLM-extracted clinical KG triples and retrieval-focused GraphRAG by preserving construction-time quality for downstream medical QA. It proposes complementary quality modeling and evaluates propagation across construction, retrieval, prompting, and evidence tracing.

  • Clinical guideline-to-KG transformation is difficult because medical text is terminologically dense, semantically complex, heterogeneous in granularity, and often leaves evidence boundaries implicit.
  • LLM-based extraction remains vulnerable to entity, granularity, type, relation, and direction errors.
  • Existing GraphRAG methods use graph structure for relational retrieval and multi-hop reasoning, but commonly optimize query relevance or inference-time confidence.
  • Construction-time triple reliability remains largely disconnected from downstream evidence selection because relevant triples are not necessarily reliable.
  • Quality propagation uses quality-weighted subgraph retrieval and tier-conditioned prompting while preserving passage-level provenance and evaluating conditions where the signal remains informative.
  • The framework models structural conformance and evidential support as complementary dimensions, retaining fused Q(t) and graded tiers for downstream use.

2. Related Work

Related work spans LLM-based clinical KG construction, quality assessment, and graph-augmented QA, but these strands rarely preserve construction-derived reliability for downstream evidence control. The paper positions quality propagation as the connection between offline KG quality assessment and inference-time retrieval and presentation.

  • LLM-based knowledge graph construction: Clinical KG construction has progressed from modular extraction pipelines to joint, unified, domain-pretrained, and prompt-based LLM approaches.
  • LLM-based knowledge graph construction: LLM-based construction still faces inconsistent granularity, schema noncompliance, relation errors, hallucination, and insufficient factual validation, motivating combined quality controls.
  • Knowledge quality assessment: KG quality research covers refinement, error detection, confidence modeling, uncertainty-aware prediction, evidence verification, consensus voting, and human validation.
  • Knowledge quality assessment: Model agreement alone does not establish factual correctness, so factual reliability assessment is combined with supporting evidence and, where appropriate, human validation.
  • Graph-augmented question answering: Text and graph RAG methods support retrieved context, relational paths, graph propagation, chunk expansion, context organization, and graph-neural retrieval for medical or biomedical QA.
  • From quality assessment to quality propagation: Across the literature, construction-time structural validity and evidential support are rarely retained as persistent per-triple signals for downstream retrieval, evidence organization, and prompting.

3. Proposed Framework

The proposed framework creates a construction-to-inference pipeline in which per-triple quality is estimated, retained as graph metadata, and reused to select and present evidence. It combines structural and evidential assessment with quality tiers, relevance-aware retrieval, prompting, and provenance tracing.

  • Overview: The framework unifies construction and QA through a per-triple quality representation used for graded knowledge organization, quality-weighted retrieval, and tier-conditioned evidence prompting.
  • Construction phase: SEEK-GK transforms guideline text into a quality-annotated graph by evaluating candidate triples for schema conformance and evidential reliability.
  • Quality modeling: SchemaConf measures graded compliance with entity, relation, type, and direction constraints, while EvidScore aggregates multi-model agreement, passage-level support, and human verification.
  • Quality tiers: Retained triples receive quality tiers by thresholding Q(t), preserving graded differences rather than applying one hard threshold to all validated triples.
  • Inference phase: At inference, entity-linked seed entities produce a multi-hop candidate subgraph whose triples are ranked using construction-time quality together with normalized query-triple relevance.
  • Inference phase: Tiered evidence contexts combine ranked triples, quality tiers, and guideline passages; prompts prioritize higher-tier evidence and express caution when support is weak or uncertain.
  • Design principle: The framework preserves quality, tier labels, and provenance as persistent graph attributes and reuses them without requiring joint training between construction and inference.

3.2 Guideline knowledge extraction and normalization

The pipeline preprocesses guideline text, normalizes entities and relations, induces schema constraints, and preserves passage identifiers for later evidence assessment and provenance tracing.

  • Guideline documents are cleaned, Unicode-normalized, segmented at semantic boundaries, and length-adjusted before extraction.Non-knowledge material is removed; fragments shorter than 50 characters are merged and those longer than 500 are split at sentence boundaries.
  • The canonical entity library combines human-reviewed extraction outputs with terminology resources, followed by synonym normalization and confirmation.Sources include curated NER outputs and ICD-10/11, MedDRA, and CHPO terminology matched to guideline text.
  • Candidate relation synonyms are ranked using co-occurrence, Jaccard argument-pair similarity, and substring containment before expert review.Surface variants may be merged, while directional or polarity differences remain separate.
  • Domain/Range constraints and inverse-relation pairs are derived and reviewed, while each retained triple keeps its source TextID identifiers.These identifiers support passage-level evidence assessment and answer-to-source provenance tracing.

3.3 Schema-aware triple quality estimation (SchemaConf)

SchemaConf converts structural compliance into a graded per-triple score by combining entity, relation, type, and direction compatibility rather than discarding imperfect triples outright.

  • SchemaConf scores structural conformance continuously, allowing structurally imperfect but potentially valid triples to receive intermediate values.The score is designed as a soft constraint rather than a binary filter.
  • SchemaConf(h, r, o) combines entity, relation, type, and direction sub-scores with weights 0.30, 0.25, 0.35, and 0.10, respectively.The weights sum to 1 and emphasize entity and type compatibility.
  • Entity and relation matching uses five levels spanning exact matches, human-confirmed synonyms, soft suggestions, and less-supported cases.Entity compatibility averages head and tail schema-matching scores.
  • Type constraints score 1.0 when satisfied, 0.5 for unknown entity types, and 0.0 for known violations.When no Domain or Range constraints are available, the type score defaults to 1.0.
  • Direction consistency is checked after canonical normalization, and the weighted dimensions yield a structural-validity score in [0, 1].The dimensions target surface form, relation semantics, type composition, and direction failure modes.
  • EvidScore complements SchemaConf by combining model agreement, passage-level support frequency, and human verification.Its formulation uses normalized evidence signals rather than structural schema checks.
  • EvidScore assigns weights 0.30, 0.20, and 0.50 to model agreement, support frequency, and human verification when all dimensions are informative.Human verification receives the largest contribution, while support frequency is downweighted.
  • When evidence dimensions are constant or unavailable, their weights are redistributed across informative dimensions to preserve effective score variation.For inactive human verification, the example reallocates weights to 0.60 and 0.40 for the remaining dimensions.

3.5 Quality fusion and tiered graph organization

The framework fuses structural validity and evidential reliability into an interpretable quality signal Q(t), then stores Q(t) and quality tiers as persistent graph attributes.

  • Schema conformance and evidential reliability are modeled separately and fused into Q(t) through a transparent weighted formulation.The design keeps each dimension interpretable and adjustable without additional learned parameters.
  • Triples are organized into core, extended, and exploratory tiers using thresholds θcore=0.85 and θext=0.70.Core triples satisfy Q(t) ≥ θcore; extended triples satisfy θext ≤ Q(t) < θcore; exploratory triples fall below θext.
  • Low-scoring triples are tiered rather than removed, preserving potentially useful boundary knowledge while indicating lower confidence.The tiers support downstream evidence selection according to reliability requirements.
  • Human confirmation can raise a triple’s EvidScore and Q(t), potentially promoting it to a higher tier, whereas confirmed incorrect triples are removed.Because quality is estimated per triple, localized verification or guideline revisions require localized recomputation.

3.6 Propagating construction-time quality to downstream question answering

Construction-time quality is retained in the graph and propagated into retrieval and prompting, while complementary guideline passages and TextID links preserve context and provenance.

  • Each retained triple stores Q(t) and its tier label τ(t) as persistent graph attributes for inference-time reuse.This is the mechanism connecting construction-time quality control with downstream question answering.
  • The retrieval interface combines construction-time quality with entity-link relevance when ranking candidate triples for subgraph evidence.Entity-link relevance is the maximum linking confidence among triple entities matched to the question.
  • Question-to-graph linking uses exact, segmentation, and conditional fuzzy matching before up to two-hop subgraph expansion.Exact matches receive confidence 1.0, segmentation matches 0.95, and fuzzy matching is used when earlier stages yield fewer than three candidates.
  • Dense retrieval independently returns the top-5 guideline passages as complementary context for conditions, qualifiers, and explanations omitted by structured triples.The dual-source evidence set retains each triple’s Q(t), tier label, and source TextID(s).
  • Tier-conditioned prompting prioritizes core evidence, uses extended evidence supplementally, and restricts exploratory evidence for high-risk relations.Supporting graph evidence remains traceable through t → TextID → guideline passage.
  • The two evidence channels are merged and presented to the language model through a tier-conditioned prompt.The KG channel performs Q(t)-weighted multi-hop retrieval while the text channel supplies dense-retrieval context.

4. Experimental Setup

The study evaluates construction-time triple quality and its propagation into guideline-grounded medical QA using controlled datasets, baselines, retrieval configurations, and clinician assessment. Evaluation covers quality discrimination, evidence use, robustness across guideline versions and generators, and human-rated answer quality.

  • Evaluation scope: The framework is evaluated at both knowledge-graph construction quality and guideline-based medical question answering.The two evaluations share the same entity library, relation set, and schema constraints, while construction-produced quality scores and tiers are retained for downstream QA.
  • Guideline corpus and graph data: The primary corpus contains 1,905 passages from authoritative Chinese diabetes clinical guidelines, processed with GPT-4, GPT-4o, and GLM-4-Plus.Each passage retains its heading path and fragment identifier.
  • Robustness data: The cross-version test set contains 370 passages and 1,888 expert-labeled triples extracted with Qwen3-Max, GLM-4-Plus, and DeepSeek-V3.It differs from the primary corpus in guideline version and extraction-model composition, supporting distribution-shift evaluation.
  • QA dataset: The QA dataset contains 440 questions across seven types, including single-hop, multi-hop, comparison, constraint, overview, and long-tail scenarios.Of these, 301 questions have predefined required KG paths present in the runtime graph, enabling path-anchored evaluation of retrieval and evidence use.
  • Baselines and ablations: Construction baselines compare tiered retention with increasingly strict schema hard filters and compare Q(t) with SchemaConf and EvidScore for discrimination.QA configurations range from no retrieval and passage-only retrieval to quality-weighted retrieval with tier-conditioned prompting and an oracle KG condition.
  • Metrics and validation: Evaluation reports knowledge preservation, tier precision, ROC-AUC, omission, conflict, invalid citations, unsupported claims, evidence-grounded precision, and blinded clinician ratings.Clinicians rate 100 stratified questions across no_rag, full, and oracle_kg outputs on four five-point dimensions, producing 600 rating instances.

5. Results

Results show that construction-time quality propagation preserves more correct knowledge, discriminates triples more effectively under distribution shift, and improves guideline-grounded QA across evidence quality, clinician ratings, and generators. The gains arise through complementary quality-weighted retrieval and tier-conditioned evidence presentation, while factual validation and contextual representation remain important boundaries.

  • 5.1 Schema-aware knowledge preservation: Approximately twice as many correctly retained triples result from graded SchemaConf retention as from strict schema filtering on the cross-version test set.SEEK-GK retains approximately 6,304 estimated correct triples versus approximately 3,155 for strict filtering.
  • 5.2 Quality-score discrimination under distribution shift: AUC 0.748 for fused Q(t) exceeds EvidScore at 0.703 and SchemaConf at 0.645 under cross-version, cross-model shift.The gains over both components are statistically significant, while in-distribution ranking changes little with the fusion weight.
  • 5.3 Overall guideline-based QA performance: Required-knowledge omission falls from 16.3% to 5.3%, conflict rate from 16.3% to 2.7%, and unsupported-claim rate from 72.1% to 28.8% with the full framework.Evidence-grounded precision reaches 81.6% with a near-zero invalid-citation rate, and supported statements retain triple-to-guideline provenance.
  • 5.4 Ablation of quality-weighted retrieval and tiered prompting: Quality-weighted retrieval lowers omission from 8.3% to 5.0%, while tier-conditioned prompting further lowers conflict numerically from 4.7% to 2.7%.The prompting step alone is not statistically significant, so the two interfaces are best interpreted jointly.
  • 5.5 Clinician evaluation: Clinician ratings order the modes as no_rag 4.21 < full 4.68 < oracle_kg 4.80, with the full framework significantly outperforming no retrieval.The improvement is also observed for accuracy, completeness, and hallucination control, with moderate inter-rater agreement.
  • 5.6 Cross-model robustness: Cross-generator experiments show lower omission and conflict rates for the full framework across GLM-4-Plus, Qwen-Max, and DeepSeek-V3.The results suggest that the benefits are not specific to the primary generator, although the broader synthesis identifies coverage, activation, and reliability as jointly required conditions.

6. Discussion

The discussion presents construction-time quality as actionable inference information, while emphasizing that gains depend on evidence activation, coverage, reliability, and evaluation scope.

  • Construction-time quality as a persistent inference signal: Construction-time quality complements query relevance by preserving reliability information that relevance alone cannot recover for downstream evidence selection.Q(t) captures schema conformance, support frequency, model agreement, and human verification independently of future queries.
  • Construction-time quality as a persistent inference signal: Quality-aware ranking and tier-conditioned prompting integrate Q(t) into existing pipelines without joint training or query-time quality re-estimation.The signal is stored as a graph attribute, with provenance linking graph evidence to guideline passages.
  • Conditions limiting quality propagation gains: 15 of 16 remaining KOR failures occurred during activation or retrieval, showing that quality scoring cannot compensate when relevant evidence is not surfaced.This evaluation guaranteed graph coverage because questions were constructed from existing KG evidence paths.
  • Conditions limiting quality propagation gains: Broader deployment may expose coverage gaps because questions are not necessarily constrained by known KG paths, requiring additional sources, extraction, review, and graph maintenance.No retrieval or quality-ranking strategy can recover evidence absent from the graph.
  • Scope and interpretation: SchemaConf adds limited discrimination in-distribution but contributes more under cross-version, cross-model shift, where fused Q(t) reaches AUC 0.748.The discussion reports Q(t) outperforming EvidScore at 0.703 and SchemaConf at 0.645 on that shifted test set.
  • Scope and interpretation: The conclusions are bounded by a Chinese diabetes, single-language setting, a self-constructed QA set, limited generator evaluation, and one distribution-shift instance.The paper also states that its quality-to-answer link is indirect and lacks sufficient evidence-quality variation for reliable monotonic estimation.

7. Conclusion

The conclusion presents a lightweight framework that carries fused triple quality from KG construction into retrieval and prompting. Experiments report improved evidence-grounded QA, while the central claim remains scoped to the evaluated setting.

  • Conclusion: The framework stores fused Q(t) and reliability tiers during construction, then reuses them for evidence retrieval and tier-conditioned prompting without joint training.Quality attributes stored with the graph support integration into existing guideline pipelines.
  • Conclusion: 16.3% → 5.3% required-knowledge omission and 16.3% → 2.7% conflicting outputs were reported with 81.6% evidence-grounded precision and near-zero invalid citations.Clinician ratings followed no-retrieval 4.21 < full 4.68 < oracle-evidence 4.80.
  • Conclusion: The framework’s scoped claim is that construction-time quality can operationally improve downstream evidence selection and presentation, not guarantee correct answers or provide a universally optimal quality measure.The claim applies within a single clinical domain and language under KG-anchored evaluation with required evidence present.

Data Availability

The paper provides an anonymized review package containing data, labels, scripts, results, documentation, and integrity checksums through a private peer-review link.

  • Data Availability: An anonymized package supports independent recomputation without redistributing copyrighted guideline text or clinician free-text comments.The package includes de-identified construction and evaluation data, frozen labels, analysis scripts, machine-readable results, and checksums.
  • Data Availability: The study collected or used no patient data.
  • Data Availability: The authors used ChatGPT for language editing and clarity improvements, then reviewed and edited the manuscript and accepted responsibility for its content.

Funding

The work was supported by Chinese national, institutional, and provincial medical research funding, with funders reported as having no role in study design, data collection, or analysis interpretation.

  • Funding: Funding came from national, institutional, and provincial medical research programs, including grants 2023YFC3605800, 82300918, and NM202408.
Loading 2608.28360v1…