PaperlayerView exact revision on arXiv

Source-linked AI summary

Knowledge-Graph Based Augmentation versus Retrieval Augmented Generation for Cultural-Related Question Answering

Pablo Poulenard, Yannis Karmim, Valentin Barrière

arXiv:2609.18317v2cs.CLcs.AI

TL;DR

Culturally specific facts about underrepresented regions are difficult for LLMs to memorize reliably. This paper builds Spanish Wikipedia knowledge graphs with KGGen and compares graph-based retrieval with standard RAG on LatamQA. G-Retriever reduces base-LLM error, while benchmark-aware extraction narrows its gap with RAG and its trained projection transfers zero-shot to Portuguese.

  • Problem

    Culturally specific facts, especially those about underrepresented regions, appear too rarely in pretraining corpora for LLMs to memorize reliably.

    Source: §1 · p. 1

  • Method

    The paper constructs category-level knowledge graphs from LatamQA-linked Spanish Wikipedia articles with KGGen and evaluates G-Retriever against text and triple retrieval baselines.

    Source: §2 · p. 2, §1 · p. 2

  • Results

    G-Retriever reduces base LLM error by 74% on a 3.2× shorter context, and a benchmark-aware KG narrows the residual gap with RAG to 2 pp.

    Source: §5 · p. 4

  • Takeaways & Limitations

    Benchmark-aware extraction narrows the gap with RAG, and the trained projection transfers zero-shot to Portuguese.

    Source: §5 · p. 4

    Because LatamQA and the graphs derive from Spanish Wikipedia, the benchmark reflects its uneven coverage and underrepresents orally transmitted or artisanal domains.

    Source: §6 · p. 5

Abstract

from arXiv · show

Large language models (LLMs) suffer from a long-tail deficit: culturally specific facts, particularly those concerning underrepresented regions such as Latin America, appear too rarely in pretraining corpora to be reliably memorized. Retrieval-Augmented Generation (RAG) addresses this by grounding generation in external text, but structured alternatives such as Knowledge Graphs (KGs) offer tighter control over what enters the context, along with potential gains in explainability and updatability. We benchmark Graph-RAG against standard RAG on LatamQA, a culturally grounded multiple-choice dataset spanning eight thematic categories. The graphs are built end-to-end from Wikipedia articles with KGGen, a recent open-domain extractor, without manual curation in our main setting. G-Retriever is competitive with RAG and reduces the error of the base LLM by 72\% with a standard KG and 78\% with a benchmark-aware variant, the gap to RAG narrowing further as the graph is oriented toward task-relevant content. The trained projection transfers zero-shot to Portuguese without target-language fine-tuning, indicating multilingual reach.

1 Introduction and Related Work

The paper frames culturally specific Latin American facts as a long-tail knowledge gap and compares text retrieval with graph-based augmentation in a deliberately difficult QA setting. Its contribution is an end-to-end KGGen benchmark study on LatamQA, including ablations and zero-shot Portuguese transfer.

1 Introduction and Related Work

  • Underrepresented cultural facts appear too rarely in pretraining to be reliably memorized, with the gap manifesting as confabulation rather than abstention.The introduction identifies Latin American culture as a case where LLMs answer Iberian Spanish questions more accurately than equivalent Latin American questions.Source: §1 · p. 1
  • Knowledge graphs offer explicit relational triples that are compact, inspectable, updateable, and traceable to sources, while RAG grounds generation in external text but fills the context window.The introduction contrasts structured control over prompt contents with RAG's token cost.Source: §1 · p. 1
  • Existing graph QA evaluations use small, clean, dedicated graphs and multi-hop questions, whereas this study uses one large, noisy KG per category and single-hop questions.The authors describe this regime as naturally advantaging RAG and exposing the cost of converting text into triples.Source: §1 · p. 1
  • The study benchmarks Graph-RAG against standard RAG on LatamQA, a culturally grounded multiple-choice dataset spanning eight Latin American thematic categories.Its pipeline applies KGGen to 5,848 Wikipedia articles and evaluates downstream QA rather than intrinsic extraction metrics.Source: §1 · pp. 1–2
  • The contribution includes comparing RAG, Top-k triples, and G-Retriever, with ablations of graph structure and extraction quality and zero-shot transfer of the trained projection to Portuguese.Figure 1 depicts the pipeline: KGGen builds domain graphs, G-Retriever selects a PCST subgraph, and the same triples are used as text for RAG and Top-k baselines.Source: §1 · p. 2, Figure 1 · p. 2

2 Proposed experimental pipeline

The experimental pipeline pairs LatamQA questions with Wikipedia articles and constructs Spanish category-level knowledge graphs using KGGen. It also specifies shared query encoding and a benchmark-aware graph variant directed toward task-relevant content.

2 Proposed experimental pipeline

  • The dataset contains 5,848 Wikipedia articles collected from eight thematic categories by recursively scraping country-theme categories to depth two and intersecting them with LatamQA.Each article supports one question with one correct answer and three content-grounded distractors, so no question requires composition across articles.Source: §2 · p. 2
  • KGGen extracts entities and relations from Spanish Wikipedia, aggregates article graphs by category, resolves entities and edges, and produces one unified KG per category.The pipeline applies KGGen to 50M characters, with each article split into 5,000-character segments; triples retain provenance for retrieval ground truth.Source: §2 · p. 2
  • A benchmark-aware variant injects question-derived entity and relation hints into extraction prompts to steer each category graph toward content relevant to the downstream task.Entity and edge resolution otherwise follows the base pipeline unchanged.Source: §2 · p. 2
  • Qwen2.5-3B-Instruct scores 60.17 zero-shot accuracy, leaving headroom to measure gains from external knowledge.The setup selects a model that has not already saturated the benchmark with parametric knowledge.Source: §2 · pp. 2–3
  • The retrieval query concatenates the question and its four options, giving the retriever a symmetric signal without privileging any option.All four systems use the same multilingual jina embeddings encoder and query format.Source: §2 · p. 3

3 Compared retrieval methods

The compared systems use the same language model and embedding model but differ in the retrieved context. They range from raw-text retrieval and unordered triples to a trained subgraph encoder that supplies a soft prompt.

3 Compared retrieval methods

  • All systems share the same language model and embedding model and differ only in retrieved context, providing a common comparison across raw text and graph-based methods.The graph methods increase their use of graph structure from unordered triples to a trained subgraph encoder.Source: §3 · p. 3
  • RAG retrieves the five most similar article chunks, each at most 512 tokens with 64-token overlap, from the question's corresponding category.The chunk and retrieval settings were selected by grid search.Source: §3 · p. 3
  • Top-k triples ranks each triple independently by cosine similarity to the query and verbalizes the highest-ranked triples, treating the graph as an unordered set.The baseline omits entity linking because open extraction does not guarantee exact correspondence between question and graph entities.Source: §3 · p. 3
  • G-Retriever selects a subgraph with PCST, encodes it as one vector, and projects that vector into the language model's embedding space as a trainable soft prompt.The graph encoder and projection are trained end-to-end on a training split; a mean-pooling variant retrieves nodes by topology without encoding topology.Source: §3 · p. 3

4 Results and Analysis

RAG leads the overall LatamQA comparison, while graph-based methods improve on zero-shot performance and remain competitive. Ablations, Portuguese transfer, and inference costs show how extraction quality, the trained projection, and compact retrieved context shape the results.

4.1 Main comparison

  • RAG reaches 93.38 accuracy versus G-Retriever’s 89.71; G-Retriever exceeds RAG in Gastronomía, 93.04 versus 87.37.Across categories, the training-free Top-k triples baseline improves by 15.4 pp over zero-shot accuracy of 60.17.Source: §4.1 · pp. 3–4, Table 2 · p. 4

4.2 Ablation Studies

  • Removing the trained projection drops PCST-as-plain-text to 73.10; Linear matches or exceeds GraphEnc at 88.90 vs. 86.13 without text and 89.50 vs. 89.71 with it.The authors attribute the linear projection’s performance to the single-hop task, where message passing is unnecessarily expressive.Source: §4.2 · p. 4, Table 3 · p. 4
  • Benchmark-aware extraction improves Top-k triples by 2.3 pp and G-Retriever by 1.7 pp to 91.41, but remains below RAG’s 93.38.The authors identify extraction, rather than retrieval, as the main bottleneck.Source: §4.2 · p. 4, Table 3 · p. 4

4.3 Multilingual transfer

  • Without further training, Spanish-fold checkpoints reach 91.18 on Portuguese Literatura, above their 89.44 on Spanish Literatura.The zero-shot baseline falls from 60.06 to 53.40; transfer is conditioned on a multilingual retrieval backbone.Source: Table 4 · p. 4, §4.3 · p. 4

4.4 Inference efficiency

  • Graph-RAG reduces average inference context from 2,814 to 875 tokens, while KG extraction costs 72.26 EUR and graph-encoder training takes about 9,942 seconds per fold.The compute and API costs move offline.Source: Table 5 · p. 4, §4.4 · p. 4

5 Conclusion

On a culturally grounded, single-hop multiple-choice benchmark spanning eight Latin American categories, G-Retriever reduces base-LLM error and approaches RAG with a benchmark-aware KG. The trained projection also transfers zero-shot to Portuguese, while Spanish Wikipedia coverage remains a boundary of the study.

5 Conclusion

  • G-Retriever reduces base LLM error by 74% on a 3.2× shorter context, while a benchmark-aware KG narrows the residual gap with RAG to 2 pp.The authors identify extraction quality, rather than retrieval, as the bottleneck.Source: §5 · p. 4
  • The trained projection transfers zero-shot to Portuguese.Source: §5 · p. 4
  • Spanish Wikipedia coverage skews toward documented rather than orally transmitted knowledge, and the authors say addressing this bias will require participatory sources.Source: §5 · pp. 4–5

6 Limitations

The evaluation and evidence are bounded by multiple-choice scoring, a single reader, narrow cross-lingual testing, and Spanish Wikipedia’s coverage.

6 Limitations

  • Four-way multiple-choice scoring allows success by elimination, so extraction-induced information loss is penalized only when discriminative content is removed.The authors therefore consider the measured gap between flat retrieval and graph augmentation likely a lower bound on the true cost of converting text into triples.Source: §6 · p. 5
  • All results use Qwen2.5-3B-Instruct, and the projection is aligned to that frozen reader without testing transfer across backbones.Source: §6 · p. 5
  • Portuguese transfer covers one category and a language close to Spanish, using a graph an order of magnitude smaller than its Spanish counterpart.The authors note that broader claims would require matched-size graphs and typologically distant, lower-resource languages.Source: §6 · p. 5
  • LatamQA and the graphs derive from Spanish Wikipedia, whose uneven category coverage reflects editorial attention and underrepresents orally transmitted or artisanal domains.Source: §6 · p. 5

A Implementation Details

Retrieval uses a shared multilingual encoder and queries that combine each question with all four answer options. The encoder’s long context window permits variation in retrieval granularity while keeping the encoder fixed.

A.1 Embedding and Query

  • The retrieval encoder is jinaai/jina-embeddings-v3, a multilingual model with an 8,192-token context window and task-specific LoRA adapters for queries and passages.Source: §A.1 · p. 6
  • Queries concatenate the question and all four answer options, enriching retrieval cues without privileging an option.The format also matches the reader’s inference-time input, which includes the question and all options.Source: §A.1 · p. 6

A.2 Retrieval Method Hyperparameters Optimization

The implementation tunes retrieval on a held-out question subset, constructs G-Retriever’s graph-conditioned prompt, evaluates with cross-validation, and builds category-level graphs from article-level extractions.

A.2 Retrieval Method Hyperparameters Optimization

  • Retrieval hyperparameters were grid-searched on 500 uniformly sampled held-out questions; the Graph-RAG search omitted the trained GNN projection for cost, assuming retrieval rankings remain approximately separable from projection training.The grid varied RAG chunk size and k, triple count, and Graph-RAG node and edge budgets and PCST edge cost.Source: §A.2 · p. 6

A.3 G-Retriever

  • G-Retriever retrieves a query-relevant connected subgraph with PCST, encodes its structure, and projects the resulting graph token into the frozen LM’s embedding space as a trained soft prompt.Source: §A.3 · p. 7

A.4 Cross-Validation

  • Experiments use full-dataset k-fold train-validation-test cross-validation.Source: §A.4 · p. 7

B.1 Base pipeline

  • KGGen extracts entities and relations from article chunks, then aggregates per-article graphs by category and resolves entities and edges into a unified category graph.Triple provenance is retained to recover which article produced each triple for retrieval evaluation.Source: §B.1 · p. 7

B.2 Benchmark-aware knowledge graph

The benchmark-aware graph steers extraction toward entities and relations useful for distinguishing LatamQA answer choices, while the prompt figures show the vanilla and triplet-enhanced formats.

B.2 Benchmark-aware knowledge graph

  • The benchmark-aware graph injects question- and answer-option-derived entity and relation hints into extraction to target task-relevant content.It is an upper-bound diagnostic, not deployable, because it requires evaluation questions during graph construction.Source: §B.2 · p. 7

C.1 Vanilla Prompt

C.2 Triplet-enhanced Prompt

  • The triplet-enhanced prompt uses a context_block containing extracted triplets retrieved with Top-k triplet or PCST.Source: Figure 3 · p. 8, §C.2 · p. 8

D Retrieval Oracle Analysis

The oracle analysis restricts retrieval to each question’s gold article and finds that this benefits RAG most, widening its accuracy gap over graph-based methods.

D Retrieval Oracle Analysis

E Inference Cost Details

RAG uses longer inference contexts, whereas Graph-RAG incurs offline graph-construction and encoder-training costs.

E Inference Cost Details

  • 2,814.3 tokens is RAG’s average inference context, versus 874.8 tokens for Graph-RAG.The difference reflects retrieved passages versus compact subgraph triples; the figures depend on retrieval hyperparameters.Source: E Inference Cost Details · p. 8, Table 7 · p. 8
  • The systems distribute costs differently across the lifecycle: RAG incurs inference-time context costs, while Graph-RAG front-loads offline construction costs.Source: E Inference Cost Details · p. 8
  • 72.26 EUR in API calls and 9,942 s per fold are Graph-RAG’s reported KG extraction and graph-encoder training costs.KGGen extraction used a 51M-character dataset.Source: E Inference Cost Details · p. 8, Table 7 · p. 8

Related papers

Loading 2609.18317v2…