PaperlayerView exact revision on arXiv

Source-linked AI summary

RAGU: A Multi-Step GraphRAG Engine with a Compact Domain-Adapted LLM

Mikhail Komarov, Ivan Bondarenko, Stanislav Shtuka, Oleg Sedukhin, Roman Shuvalov, Yana Dementyeva, Matvey Solovyov, Nikolay O. Nikitin

arXiv:2607.11683v1cs.CLcs.AI

TL;DR

GraphRAG systems often rely on single-pass extraction and expensive models, leading to noisy graphs and limiting engineering maturity. RAGU separates extraction from consolidation with a compact skill-oriented extractor, retrieving the most complete context across factoid levels and outperforming HippoRAG2 on synthesis tasks.

  • Problem

    Existing GraphRAG systems use single-pass extraction and often depend on expensive LLMs, while open-source frameworks remain difficult to install, test, or deploy.

  • Method

    RAGU uses a modular multi-step pipeline that separates typed extraction from deduplication, summarization, relation consolidation, and community detection, alongside a 7B skill-oriented extractor.

  • Results

    RAGU retrieves the most complete context across GraphRAG-Bench factoid levels and overtakes HippoRAG2 on synthesis tasks, while Meno-Lite-0.1 outperforms Qwen2.5-32B on KG construction.

  • Takeaways & Limitations

    RAGU is preferable for broad-context synthesis under a single-GPU budget, whereas chain-traversal systems are preferable for precise multi-hop fact lookup.

  • Takeaways & Limitations

    RAGU’s default NetworkX backend does not scale to massive corpora, and weak extraction models can leave structural noise that consolidation cannot fully remove.

Abstract

from arXiv · show

Graph retrieval-augmented generation (GraphRAG) enhances large language models with structured knowledge, yet existing systems construct knowledge graphs in a single extraction pass, producing noisy entities and brittle retrieval. RAGU, an open-source modular GraphRAG engine, addresses this by separating extraction from consolidation: entities and relations pass through two-stage typed extraction, DBSCAN-backed deduplication, LLM summarization, and Leiden community detection. A key insight motivates a compact extractor: the skills an in-pipeline LLM needs - comprehension, extraction, reasoning over context - are language skills that grow only weakly with model size, unlike factual world knowledge. Accordingly, we train Meno-Lite-0.1, a 7B model optimized for language skills, which outperforms Qwen2.5-32B on knowledge-graph construction (+12.5% relative harmonic mean) and matches it on English GraphRAG tasks. On GraphRAG-Bench (Medical), RAGU retrieves the most complete context at every factoid level (evidence recall up to 0.84 vs. $\leq$0.76) and overtakes HippoRAG2 on synthesis tasks; on multi-hop factoid QA, the apparent HippoRAG2 advantage is shown to be largely an answer-format artifact. RAGU is installable via $\texttt{pip install graph_ragu}$, runs on a single GPU, and is released under MIT. The source code is publicly available at https://github.com/RaguTeam/RAGU, and the Meno-Lite-0.1 model can be obtained from https://huggingface.co/bond005/meno-lite-0.1.

1 Introduction

The introduction identifies noisy single-pass extraction, costly large-model dependence, and engineering immaturity as barriers to practical GraphRAG adoption. RAGU addresses these obstacles through a compact skill-oriented extractor and explicit multi-step consolidation targeting engineerable deployment.

  • Motivation: Single-pass graph extraction produces noisy, duplicated entities and fails to consolidate information across chunks.This is identified as the first practical obstacle facing current GraphRAG systems.
  • Motivation: GraphRAG pipelines need comprehension, extraction, and contextual reasoning, which are language skills that grow weakly with model size rather than factual recall.This hypothesis motivates using a compact, skill-oriented model for extraction.
  • Motivation: 21.1×: CheGeKa F1 grows from 0.5 B to 72 B, whereas MultiQ F1 grows only 4× across Qwen2.5-Instruct models.The comparison motivates distinguishing world knowledge from language skills in GraphRAG pipelines.
  • Contributions: Meno-Lite-0.1 is a 7 B model fine-tuned for RAG-oriented language skills and outperforms Qwen2.5-32B on knowledge-graph construction.It is fine-tuned from RuadaptQwen2.5-7B over the NEREL schema.
  • Contributions: RAGU introduces an explicit multi-step consolidation stage while targeting installation, testing, deployment, and cost-effective hardware.These design goals respond to single-pass extraction and engineering immaturity.

2 System Description

RAGU is a configurable six-stage GraphRAG system that separates constrained two-stage extraction from consolidation, community organization, and refinement. It provides multiple retrieval engines, swappable storage, validated structured outputs, incremental updates, and single-GPU deployment with a 7B extraction model.

  • Indexing pipeline: RAGU processes documents through six configurable stages: chunking, two-stage extraction, consolidation, community detection, summarization, and refinement.Chunking supports fixed-size overlap, embedding-based split points, and cross-encoder reranking; later stages use hierarchical Leiden clustering and structured community reports.
  • Indexing pipeline: Two-stage extraction validates entities against the NEREL schema before constraining relation endpoints to validated entity names, eliminating spurious entity–relation mismatches.The schema contains 29 entity types and 49 relation types, with optional in-context-learning examples at both stages.
  • Indexing pipeline: Consolidation groups entities by name and type, applies DBSCAN and LLM summarization to duplicate mentions, and similarly summarizes relations before community detection.This noise-reduction step produces a cleaner graph and is absent from single-pass systems such as LightRAG.
  • Retrieval engines: RAGU offers LocalSearch, GlobalSearch, NaiveSearch, MixSearch, and QueryPlanEngine, with cross-encoder reranking and hybrid dense+sparse retrieval via Qdrant.QueryPlanEngine performs DAG decomposition, while LocalSearch expands entity retrieval to relations and chunks and GlobalSearch uses LLM-rated community summarization.
  • Implementation and deployment: The production-oriented package supports three-tier storage, asynchronous bounded-concurrency execution, Pydantic v2 validation, deterministic incremental updates, consistency auditing, and single-GPU deployment with a 7B extraction model.Storage backends can be swapped across graph, key-value, and vector tiers; deterministic mock LLM support enables CI without API keys.

3 Evaluation

RAGU is evaluated across GraphRAG-Bench, BioASQ, MuSiQue, and 2WikiMulti-HopQA with controlled answer generation and varying graph-construction models. It achieves stronger context retrieval and synthesis-oriented performance, while apparent factoid-QA disadvantages are reduced by controlling answer format.

  • Evaluation setup: The evaluation spans four benchmarks, holding answer generation at gpt-4o-mini while varying the graph-construction LLM.Metrics include Answer Correctness, ROUGE-L, Coverage, Faithfulness, Evidence Recall, and Context Relevancy.
  • GraphRAG-Bench: On GraphRAG-Bench, HippoRAG 2 leads Fact Retrieval by 18.2 percentage points, reaches parity on Contextual Summarize, and trails RAGU on Creative Generation.AC is 72.4 vs. 54.2 on Fact Retrieval, −0.9 pp on Contextual Summarize, and 59.0 vs. 56.9 on Creative Generation.
  • GraphRAG-Bench: 84% Evidence Recall is RAGU’s result at every factoid level, versus ≤76% for competitors, despite HippoRAG 2’s higher single-fact accuracy.The reported ablations show that ICL and validation each shift AC by at most ∼1.5 pp.
  • Multi-hop QA: Under verbose prompts, HippoRAG 2 dominates multi-hop metrics, but controlled terse prompts raise RAGU to 72.9 vs. 72.4 AC on BioASQ and narrow the 2Wiki-MultiHopQA gap to 5.5 percentage points.HippoRAG 2 retains a lead on MuSiQue, scoring 54.4 vs. 40.1.
  • Extractor evaluation: 12.5% relative is Meno-Lite-0.1’s harmonic-mean advantage over Qwen2.5-32B on knowledge-graph construction, driven by relation extraction F1 of 0.347 vs. 0.239.Its standalone extraction edge compresses to ≤1 pp on end-to-end GraphRAG-Bench QA, while delivering 7B extraction-model cost.

4 Demonstration

The demonstration applies RAGU to a Dennis Ritchie passage, showing typed extraction, community detection, structured summarization, and multi-hop question answering. It also documents the system’s installation, licensing, and public resources.

  • Extraction: The two-stage extractor identifies 9 typed entities and 8 relations constrained to the validated entity set.Entities follow the NEREL schema, and relations are extracted in a second stage.
  • Community detection: Leiden clustering partitions the 9-entity graph into two communities: Ritchie’s creations and a Bell Laboratories geographic-professional cluster.The communities contain 5 entities and 4 relations, and 4 entities and 3 relations, respectively; an LLM summarizes each.
  • Multi-hop retrieval: RAGU’s LocalSearchEngine answers multi-hop questions by traversing the built graph.The examples identify Alistair E. Ritchie’s workplace and Dennis Ritchie’s creations from chained clues.
  • Availability: RAGU is installable with pip install graph_ragu and provides API documentation, examples, source code, and a demonstration video.The source code is hosted on GitHub, and the system is released under the MIT license.
  • Availability: Meno-Lite-0.1 is distributed under the Apache 2.0 license through Hugging Face.The passage identifies the model’s public distribution location.

5 Conclusion

The conclusion argues that pipeline LLMs primarily need language skills rather than world knowledge, and that these skills scale weakly with model size. RAGU operationalizes this insight through a modular multi-step pipeline that excels in context completeness and synthesis, while retrieval precision and some multi-hop results favor HippoRAG2.

  • Core insight: Pipeline LLMs need language skills—not world knowledge—and these skills scale weakly with model size.This principle motivates RAGU’s design.
  • RAGU’s strengths: RAGU retrieves the most complete context at every factoid level of GraphRAG-Bench and overtakes HippoRAG 2 on synthesis tasks.The synthesis tasks named are Creative Generation AC and Coverage.
  • Comparative results: HippoRAG 2 excels at retrieval precision, leading single-fact AC and chain-following multi-hop reasoning on MuSiQue.These results contrast with RAGU’s context-completeness and synthesis advantages.
  • Comparative results: The wider multi-hop gap under verbose prompts is largely an answer-format artifact.The conclusion qualifies the apparent multi-hop difference between the systems.

Limitations

The paper’s limitations concern the scope of its scaling evidence, Meno-Lite-0.1’s factual recall and long-context reasoning, benchmark distributional overlap, and deployment constraints. RAGU also remains sensitive to graph backend scalability and extraction-model quality.

  • Evidence scope: Scaling evidence covers one model family and selected tasks, so its conclusions remain a supported hypothesis rather than a universal theorem.The evidence is reported as robust across six Qwen2.5 model sizes.
  • Model limitations: Meno-Lite-0.1 sacrifices parametric factual recall, is unsuitable as a standalone knowledge base, and degrades in multi-hop reasoning beyond 32K tokens.The limitation is typical of 7B-class models.
  • Evaluation caveat: The IE benchmark has a distributional-overlap caveat because training uses NEREL train and validation splits, while evaluation uses the held-out test split with different instruction wording.This caveat concerns Meno-Lite-0.1’s supervised fine-tuning and the benchmark’s split design.
  • System limitations: RAGU’s default NetworkX backend does not scale to million-node corpora, requiring a dedicated graph-database adapter, while weak extraction LLMs can introduce irreparable structural noise.The backend uses a swappable BaseGraphStorage adapter, but final graph quality remains extraction-model-sensitive.

Ethics Statement

The paper documents open licensing, public-data provenance, and privacy safeguards, while addressing accessibility, misuse, schema bias, and reproducibility. RAGU is designed to run on a single consumer GPU and includes safeguards against unsafe model outputs.

  • Data Provenance and Licensing: RAGU is MIT-licensed, while Meno-Lite-0.1 is distributed under Apache 2.0 and trained exclusively on cited publicly available datasets.The training sources include educational web corpora, Russian academic texts, information-extraction and multi-hop QA datasets, and synthetic GPT-4o-mini instructions.
  • Data Provenance and Licensing: No personally identifiable information was included in any training or evaluation corpus.The IE benchmark is a test-only derivative of the human-annotated NEREL corpus and is released under an MIT license.
  • Environmental Impact and Democratization: A 7 B model suffices for extraction, enabling RAGU+Meno-Lite-0.1 to run on a single consumer GPU instead of frontier-model multi-GPU clusters.The paper presents this as an experimentally confirmed consequence of its language-knowledge hypothesis and a means of broadening GraphRAG access.
  • Potential Misuse: RAGU may propagate corpus prejudice or factual errors, while Meno-Lite-0.1’s context-grounded design reduces confident hallucination risk but increases dependence on corpus quality.The model trades parametric factual recall for context-grounded skills and should not be used as a standalone knowledge base.
  • Potential Misuse: RAGU validates all LLM outputs with Pydantic models rather than executing raw responses, eliminating a class of code-injection attacks from adversarial output.This engineering safeguard is contrasted with systems that use eval().
  • Bias and Fairness: The Russian-news NEREL schema may require adaptation for other languages or domains, with performance potentially degrading outside its design context.The associated IE benchmark also reflects Russian-language characteristics, including heavy inflection addressed by Snowball stemming.

A Engineering Comparison

RAGU and HippoRAG 2 are both open-source systems supporting incremental indexing, but they differ substantially in production failure handling, migration flexibility, and testing practicality. RAGU emphasizes storage abstraction and mock-LLM support, while the comparison identifies safety and robustness risks in HippoRAG 2.

  • A Engineering Comparison: Both RAGU and HippoRAG 2 are open-source and support incremental indexing, but differ in handling failure, migration, and change.The comparison is organized around production risks encountered while benchmarking open-source GraphRAG frameworks.
  • A Engineering Comparison: HippoRAG 2 uses eval() on regex-filtered raw LLM output, creating arbitrary-code-execution exposure and opaque exceptions when syntax deviates.The passage identifies this parsing path as a production failure mode.
  • A Engineering Comparison: HippoRAG 2 also uses assert for control flow; assert False can let offline indexing proceed without the online vLLM server under python -O.The cited implementation anchors assert False at HippoRAG.py:216, while the surrounding passage describes the resulting indexing behavior.
  • A Engineering Comparison: The HippoRAG 2 comparison is reproducible against commit d437bfb1, corresponding to hipporag 2.0.0-alpha.4 at the analysis snapshot.The appendix pins file-and-line references to a fixed repository commit rather than a moving target.
  • A Engineering Comparison: RAGU migration from NetworkX and NanoVDB to Neo4j and Qdrant with hybrid retrieval requires changing two constructor arguments.The same migration in a system without storage abstractions requires reimplementing the indexing pipeline.
  • A Engineering Comparison: RAGU’s mock LLM server reduces full CI runs from dollars of API calls to seconds of CPU time, making continuous regression testing affordable.This benefit is presented as part of RAGU’s production engineering design.

B GraphRAG-Bench Ablation Summary

The GraphRAG-Bench Medical ablation finds that model size, in-context learning, and validation have limited effects on generation accuracy. Meno-Lite-0.1 and Qwen2.5-7B perform within 1 percentage point across configurations.

  • Model and configuration effects: <1 pp is contributed by either ICL or validation, while Meno-Lite-0.1 and Qwen2.5-7B remain within 1 pp in every configuration.ICL and validation each contribute less than 1 percentage point, and the two models are within 1 percentage point throughout.

C Cost Analysis

The cost analysis separates one-time graph-construction costs from recurring answer-generation costs. Because all systems use gpt-4o-mini for answering, query-time cost is a common baseline and is omitted from Table 8.

  • Cost scope: Graph construction is a one-time per-document indexing operation, whereas answer generation occurs repeatedly at query time.Table 8 analyzes these phases separately.
  • Cost scope: All systems use gpt-4o-mini via API for answer generation, making query-time cost a common baseline.This common query-time cost is omitted from Table 8.
  • Construction models: MS-GraphRAG uses commercial gpt-4o for graph construction, while LightRAG, HippoRAG 2, and RAGU run local models under vLLM.MS-GraphRAG uses gpt-4o at $2.50/M input tokens; LightRAG and HippoRAG 2 use gpt-oss-20b, while RAGU uses Meno-L…
Loading 2607.11683v1…