PaperlayerView exact revision on arXiv

Source-linked AI summary

EGT-KG: Evidence-Grounded Typed KG Retrieval for Practical Scientific QA with Small Language Models

Muran Yu, Jiechao Gao, Yuandong Pan, Barney H. Miao, Andrew C. Lesh, Kincho H. Law, Jie Wang, Michael D. Lepech

arXiv:2609.00479v1cs.AI

TL;DR

Scientific QA with local SLMs is constrained by fragmented evidence, limited context, and weaker reasoning, motivating retrieval methods that better preserve scientific support. EGT-KG combines typed relation guidance with provenance-aware evidence nodes and improves answer quality over vanilla RAG in most tested settings. The study finds particular gains in factual correctness, reasoning soundness, completeness, and relevance, while identifying evidence localization, one-hop expansion, and BSC-centered validation as limitations.

  • Problem

    Local SLMs struggle with specialized scientific QA because their reasoning and context handling are limited while emerging-domain evidence is fragmented.

  • Method

    EGT-KG combines typed relation-guided retrieval with provenance-aware evidence nodes that retain links to original supporting text.

  • Results

    EGT-KG improves answer quality over vanilla RAG in most settings, especially for factual correctness, reasoning soundness, completeness, and relevance.

  • Takeaways & Limitations

    EGT-KG is especially relevant for local SLM scientific QA over specialized corpora with fragmented evidence and limited model capacity.

  • Takeaways & Limitations

    The study is limited by high fallback rates, one-hop query expansion, and validation on a BSC-centered benchmark.

Abstract

from arXiv · show

For emerging scientific research domains, local Small Language Models (SLMs) are becoming more attractive, as they offer stronger privacy control and more stable deployment pipelines than Large Language Models. However, in practice, scientific question-answering on SLMs often operates under inevitable constraints: small literature collections, fragmented evidence, limited context window and reasoning abilities. We propose the Evidence-Grounded Typed Knowledge Graph (EGT-KG), a retrieval framework to improve information retrieval with local SLMs. We assessed three question-answering settings: a vanilla Retrieval-Augmented Generation (RAG) workflow and two EGT-KG workflows: an automatically generated relation schema (AS) and an expert-defined relation schema (ES). Our experiments were evaluated with a six-dimensional evaluation framework (S3CRF: Soundness, Correctness, Completeness, Conciseness, Relevance, Fluency) on a Biopolymer-bound Soil Composite literature benchmark, showing that EGT-KG outperforms the vanilla RAG method in most settings, with the best improvement from llama3:8b: a Final Score of 70.37 (+14.67%) and 68.82 (+12.14%) by AS/ES EGT-KG variants.

1 Introduction

Specialized scientific QA remains difficult for local SLMs because evidence is fragmented and conventional retrieval may miss the relations and details needed for faithful answers. EGT-KG addresses this by combining typed relation-guided retrieval with provenance-preserving evidence, evaluated on a compact BSC literature benchmark.

  • SLMs face specialized scientific QA challenges because of limited internal knowledge, weaker reasoning, and fragmented domain evidence.
  • Dense RAG can retrieve semantically relevant passages while missing the specific relational evidence required for faithful scientific answers.
  • EGT-KG uses graph structures to guide retrieval without replacing original evidence, combining evidence nodes with classified relations.
  • The framework was evaluated on a 30-paper Biopolymer-bound Soil Composite benchmark representing a compact, specialized literature collection.
  • The study compares vanilla RAG with automatically generated and expert-designed EGT-KG schemas using multidimensional answer-quality and retrieval analysis.

2 Related Works

Related work establishes RAG and graph-based retrieval as useful foundations for scientific QA, while highlighting risks from irrelevant retrieval, information loss, and the costs of relation-classification methods for local SLMs.

  • Retrieval: RAG supplies external knowledge for answer generation but dense representations can overlook structured dependencies, conditions, and procedures.
  • Graph-RAG: Graph-RAG supports global corpus descriptions and multi-hop evidence discovery, but compressing original text can lose information.
  • Relation Classification: Relation-classification methods assign semantic relations to entity pairs, but applying existing approaches can require additional internet corpora and computational cost.

3 Methodology

EGT-KG combines typed, provenance-aware graph construction with two-step retrieval so local SLM answers remain grounded in original evidence. The methodology evaluates this workflow on a structured corpus using shared answer generation and six-dimensional assessment.

  • Corpus and evaluation: The benchmark contains 30 papers organized across directly relevant, related, similar-material, and general-reference groups, yielding 884 metadata-preserving chunks.The corpus focuses on Biopolymer-bound Soil Composite and related emerging biopolymer-based materials; triple extraction is performed once and reused across EGT-KG variants.
  • Graph construction: EGT-KG preserves supporting evidence spans as provenance-linked evidence nodes rather than replacing source text with processed graph information.Raw triples retain head and tail entities, free-form relations, localized evidence spans, and source-chunk links for graph construction.
  • Typed relation classification: The framework maps extracted relation labels into either an automatically generated schema (AS) or an expert-designed schema (ES) before constructing the typed graph.Identical relation labels receive identical types, while noisy, inconsistent, and irrelevant triples are trimmed for retrieval.
  • Two-step retrieval: Two-step retrieval first expands queries around seed entities to select top-k chunk candidates, then re-ranks them using evidence-level relevance.The first score combines original-query and expanded-query relevance; the second compares the query with evidence-node content and supports localized-window or full-chunk context.
  • Evidence-grounded answer generation: The final context is assembled from top-k evidence windows or full-chunk fallbacks, and all workflows use the same evidence-based answer-generation policy.The answer model is instructed to avoid unsupported inference and acknowledge insufficient reference information.
  • Corpus and evaluation: Answers are evaluated with S3CRF across soundness, correctness, completeness, conciseness, relevance, and fluency using a unified LLM-as-judge workflow.The six dimensions are combined into a weighted Final Score, with domain experts involved in validating the benchmark and criteria.

4 Results

Across the benchmark, EGT-KG generally improved answer quality over vanilla RAG, but gains depended on the local answer model. The strongest relative improvement occurred for llama3:8b, while retrieval analyses linked better outcomes to evidence-window utilization and lower fallback rates.

  • Overall performance: EGT-KG enhanced answer quality in most cases, with model-dependent improvements across the compared workflows.The experiments used the same top-k retrieval setting and evaluated five local answer models with two judge models.
  • Overall performance: 14.67%: llama3:8b’s Final Score increased from 61.37 with vanilla RAG to 70.37 with EGT-KG(AS) under gpt-4o.The EGT-KG(ES) variant reached 68.82 (+12.14%) for the same model and evaluation condition.
  • Overall performance: 86.02: llama3.1:8b achieved the highest reported Final Score with the ES schema under gpt-4o.Its corresponding score under gemini-2.5-pro was 81.72.
  • Overall performance: Gemma2:9b showed the most stable improvements, whereas mistral:7b and qwen2.5:7b had smaller or less stable gains.The benefit of finer schema granularity varied across answer models.
  • Generalization checks: QASPER mean Answer F1 improved by 5.9% on average, with gains for four of five base models in the cross-paper setting.On HotpotQA, the outcome depended on the size of the candidate pool searched by retrieval.
  • Retrieval analysis: Lower fallback rates and higher evidence-window utilization characterized the strongest EGT-KG cases, while mistral:7b had fallback rates of 0.78-0.79.Mistral’s similar Jaccard index but higher fallback rate suggests difficulty extracting useful evidence windows after re-ranking.
  • Retrieval analysis: Qwen2.5:7b expanded to 25.05 neighbor entities per question under ES with the lowest expansion efficiency.More neighbor entities therefore did not guarantee better results.

5 Discussion

The discussion attributes EGT-KG’s value mainly to improving the precision and localization of textual evidence passed to the answer model. It also identifies a trade-off: finer schemas can improve semantic precision but may reduce retrieval robustness for some local SLMs.

  • Retrieval evidence: EGT-KG’s main value lies in improving the precision of textual evidence passed to the answer model.The authors attribute gains over vanilla RAG more plausibly to retrieval quality and evidence localization than to prompt engineering.
  • Retrieval evidence: The reified graph identifies relevant entities, expands queries through structured neighbors, and re-ranks chunk candidates using evidence-level matching.The framework is characterized as a lightweight retrieval booster for SLMs in limited-resource settings.
  • Schema granularity: Finer-grained schemas can provide more informative query alignment, but may introduce sparsity and classification noise for local SLMs.Coarse schemas are less specific yet can provide a more stable retrieval path for models that cannot interpret additional injected knowledge.
  • Evidence localization: Evidence localization, rather than the number of retrieved chunk candidates, is identified as the main retrieval-quality bottleneck.Although one-hop expansion replaced 30% of original candidates after re-ranking, response quality depended on locating a clear evidence window.
  • Evidence localization: More graph expansion does not automatically improve results when answer models fall back to passing full chunks instead of using localized evidence windows.The discussion prioritizes the model’s ability to convert chunk information into grounded evidence when judging whether expansion helps or adds noise.

6 Conclusion

The paper concludes that EGT-KG improves scientific question-answering over vanilla RAG in most settings, especially on factuality-related evaluation dimensions. It presents the framework as relevant to specialized corpora with fragmented evidence and limited local-model capacity.

  • Conclusion: EGT-KG improves answer quality over vanilla RAG in most settings for practical scientific question-answering with local SLMs.The framework combines typed knowledge-graph retrieval with evidence grounding.
  • Conclusion: The strongest reported improvements concern factual correctness, reasoning soundness, completeness, and relevance.These dimensions are part of the paper’s six-dimensional evaluation framework.
  • Conclusion: The findings are especially relevant when scientific QA systems use specialized corpora with fragmented evidence and limited model capacity.This conclusion stays within the paper’s stated scope for local SLM-based scientific QA.

7 Limitations

The study’s limitations concern evidence localization, restricted one-hop expansion, and validation primarily on a BSC-centered benchmark. These constraints limit exploitation of provenance-aware retrieval, complex-query handling, and broader domain generalization.

  • Evidence localization: High fallback rates leave a substantial portion of retrieved candidates without clear evidence windows, so provenance-aware retrieval is not fully exploited.The authors identify improved evidence localization for SLMs as a route to better results.
  • Query expansion: One-hop neighbor expansion limits handling of queries requiring multi-hop reasoning or cross-document inference.The choice was made partly to manage noise and accommodate SLM reasoning limitations.
  • Generalization: Validation on a BSC-centered benchmark means the results should not be generalized broadly without further domain validation.The QASPER evaluation partially examined generalization, but its questions often use pronouns instead of precise terminology in cross-paper settings.

A.1 Additional performance analysis

EGT-KG improves evidence localization most reliably for simple factual questions, while its benefits are less stable for complex questions requiring multi-hop reasoning or cross-document inference.

  • EGT-KG variants outperform vanilla RAG in 9 of 10 simple-question comparisons, with ES variants achieving the best result in 8.
  • Benefits are less stable for complex questions involving multi-hop reasoning and global evidence aggregation.The one-hop query expansion limits processing of complex queries requiring multi-hop reasoning or cross-document inference.
  • Plain SLM question answering achieves relatively higher Final Scores on complex than simple factual questions.The paper suggests broader, discussion-oriented complex questions allow generally relevant responses to receive partial credit.

A.2 Component ablation

Component ablations show that query expansion and reranking provide stable gains, while the full pipeline performs best by combining retrieval improvements with evidence-window compression.

  • Reranking alone changes sign across judges, contributing +2.03 under gemini-2.5-pro but -0.66 under gpt-4o.
  • The full framework is the highest-scoring configuration under both judges, improving over vanilla RAG by +4.29 with gpt-4o and +3.86 with gemini-2.5-pro.
  • Query expansion plus reranking consistently improves over vanilla RAG by +2.61 with gpt-4o and +3.81 with gemini-2.5-pro.Reranking operates on the candidate chunks supplied by query expansion.
  • The evidence window cuts mean context from 8,196 to 6,399 characters without reducing score once reranking places gold evidence in the final top-k.Used alone, the evidence window contributes essentially nothing when retrieved candidates omit the gold evidence.

A.3 Supplementary QASPER Evaluation

The supplementary evaluation tests EGT-KG beyond the BSC corpus using cross-paper scientific QA and additional retrieval settings. Results show gains when retrieval must localize evidence across larger candidate pools, while benefits diminish when dense retrieval already finds the gold passages.

  • Evaluation setting: QASPER evaluates scientific question answering over NLP research papers, with each question answered using evidence from its associated source paper.
  • Evaluation setting: The cross-paper setup withholds the gold paper ID, requiring retrieval across 5,253 chunks from 281 papers for all 1,005 development questions.
  • Compared workflows: Vanilla cross-paper RAG retrieves chunks from the full corpus using dense retrieval, whereas EGT-KG adds KG-expanded query terms, evidence-aware reranking, and evidence-window construction.
  • QASPER results: EGT-KG raises mean QASPER Answer F1 from vanilla RAG’s 0.1587 to 0.1681, a 5.9% relative gain with improvements for four of five models.
  • HotpotQA results: On HotpotQA, EGT-KG adds +0.016 in the ten-candidate distractor setting but +0.142 when the same questions are pooled into 1,977 paragraphs.The reported outcome depends on the size of the candidate pool retrieval must search.
  • Practical scope: The framework is designed for practical deployment on a compact 30-paper corpus, using a lightweight typed graph to provide useful and explainable retrieval.
Loading 2609.00479v1…