PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards a Learnable Retriever for Textual Graph Question Answering

Kaifeng Jin, Yuchen Yan, Zhichen Zeng, Xiao Lin, Lihui Liu, Yinglong Xia, Hanghang Tong

arXiv:2610.33511v1cs.CLcs.LG

TL;DR

Existing textual GraphQA systems often leave the retriever untrained as an LLM and rarely use gold answers to supervise evidence selection. TaLenT combines a learnable LLM retriever and fine-tuned generator, using weighted pseudo-labels to prioritize gold answers and retain answer-related nodes while excluding irrelevant noise. Across three GraphQA benchmarks, it consistently improves downstream QA performance, outperforming the best competitor by at least 3.18%.

  • Problem

    Existing GraphRAG methods typically fine-tune the generator without making the retriever a learnable LLM, and rarely use gold answers to supervise evidence selection.

    Source: §1 · p. 1

  • Method

    TaLenT combines a learnable LLM-based retriever and fine-tuned LLM generator, training retrieval with weighted pseudo-labels from gold-answer signals.

    Source: §1 · p. 2

  • Results

    TaLenT consistently improves downstream QA performance across three GraphQA benchmarks and outperforms the best competitor by at least 3.18%.

    Source: §1 · p. 2, §7 · p. 7

  • Takeaways & Limitations

    The framework uses gold answers to prioritize answer coverage, retain answer-related context, and suppress irrelevant noise in evidence selection.

    Source: §1 · p. 2, §7 · p. 7

Abstract

from arXiv · show

Textual graph question answering (GraphQA) and graph-based retrieval-augmented generation (GraphRAG) have gained increasing attention as a way to ground large language models (LLMs) on structured knowledge. Despite extensive efforts, existing methods still face two key gaps. The first is an architectural gap: existing methods typically fine-tune an LLM-based generator, but do not also make the retriever itself a learnable LLM for graph evidence selection within one end-to-end retrieve-then-generate pipeline. The second is a retrieval-supervision gap: existing retrievers rarely use gold answer signals to supervise evidence selection. As a result, they may fail to recall evidence nodes required for reasoning, resulting in insufficient coverage of the gold answer set; meanwhile, naive attempts to boost recall often introduce excessive irrelevant information that compromises compactness and overwhelms the LLM. To address these gaps, we propose TaLenT, an LLM-based end-to-end retriever-generator framework for textual GraphQA. To bridge the architectural gap, TaLenT pairs a learnable LLM-based retriever for evidence selection with a fine-tuned LLM-based generator within one unified retrieve-then-generate pipeline. To bridge the retrieval-supervision gap, we leverage gold answer signals as retriever supervision. Specifically, we stratify graph nodes into gold answers, answer-related nodes, and irrelevant noise. We then adopt a weighted pseudo-label loss to prioritize gold answers and preserve answer-related nodes for coverage, while discarding irrelevant noise for compactness. Experimental results across three GraphQA datasets demonstrate that TaLenT consistently improves downstream performance.

1 Introduction

Textual GraphQA uses retrieved graph evidence to ground LLM answers, but existing systems leave both retriever learning and gold-answer supervision gaps. TaLenT pairs a learnable LLM retriever with a fine-tuned generator and trains evidence selection using gold-answer signals.

1 Introduction

  • LLMs can hallucinate or omit critical evidence when queries require precise, up-to-date, or domain-specific knowledge.Their outputs are bounded by information encoded in model parameters, trained on corpora collected before a fixed cutoff date.Source: §1 · p. 1
  • Existing GraphRAG pipelines often fine-tune an LLM generator but leave evidence selection to non-parametric, GNN-based, or LM-assisted procedures.Thus, the retriever itself is typically not a learnable LLM within the end-to-end pipeline.Source: §1 · p. 1
  • Query-similarity retrievers may miss gold answers while including irrelevant nodes that match query terms.The Newton example illustrates partial gold-answer coverage alongside distractor noise in the retrieved subgraph.Source: §1 · p. 1, Figure 1 · p. 2
  • Existing retrievers rarely use gold answers to supervise evidence selection, although gold answers can indicate what to cover and distinguish related evidence from noise.TaLenT uses a weighted pseudo-label objective that prioritizes gold answers, retains answer-related nodes, and ignores irrelevant noise.Source: §1 · p. 2
  • TaLenT pairs a learnable LLM retriever with a fine-tuned LLM generator in one retrieve-then-generate pipeline.Its retriever selects graph evidence before the generator answers.Source: §1 · p. 2
  • TaLenT outperforms the best competitor by at least 3.18% across experiments on three GraphQA benchmarks.The experiments evaluate retrieval effectiveness and end-task QA performance.Source: §1 · p. 2

2 Preliminaries

The preliminaries define textual graphs and the retrieve-then-generate setup. Because full graphs can exceed the LLM context budget, the candidate-subgraph procedure uses similarity-ranked nodes and edges and PCST to construct a connected subgraph balancing prize coverage and compactness.

2 Preliminaries

  • A textual graph G = (V, E) contains text-attributed nodes and relational edges, while S denotes a subgraph and V⋆ the nodes appearing in the gold answer.The retriever and generator are denoted Rθ and Gϕ, respectively.Source: §2 · p. 2

2.1 Problem Setup

  • The retrieve-then-generate setup extracts a query-conditioned subgraph S from graph G, then predicts answer a from query q and S.Directly serializing a large graph into an LLM prompt is impractical.Source: §2.1 · pp. 2–3

2.2 Candidate Connected Subgraph

  • The candidate-subgraph procedure embeds the query and graph text, then retrieves top-k nodes and edges by cosine similarity.The selected candidates receive rank-based prizes, and PCST constructs a connected subgraph trading off prize coverage and compactness; c controls the size-coverage trade-off.Source: §2.2 · p. 3

3 Method

TaLenT combines PCST preprocessing, a learnable LLM retriever trained with weighted pseudo-labels, and an SFT-tuned LLM generator. Its inference pipeline passes retrieved graph evidence to the generator to produce answers.

3.1 Retriever Training

  • Gold answers are strong positives, other nodes in the candidate subgraph are weak positives, and nodes outside it are treated as irrelevant noise.Weighted pseudo-label supervision prioritizes strong positives while retaining weak positives as a learning signal.Source: §3.1 · pp. 3–4

3 Method

  • TaLenT uses PCST to form a candidate subgraph, trains a learnable LLM retriever to select answer-relevant nodes, and fine-tunes an LLM generator to produce answers.The overview describes these as preprocessing, retriever training, and generator training stages.Source: §3 · p. 3, Figure 2 · p. 4

3.2 Generator Training

  • The generator receives the query, retrieved nodes, and associated edges, then uses standard SFT to generate the gold answer.The serialized graph context may contain disconnected components, avoiding bridge nodes or edges included only to preserve connectivity.Source: §3.2 · pp. 4–5

3.3 Inference and Evaluation

  • At test time, PCST forms a candidate subgraph, the learned retriever selects nodes, and the generator answers from the resulting textual graph context.Task metrics are computed from generator outputs on the test split.Source: §3.3 · p. 5

4 Experiments

TaLenT is evaluated on three textual GraphQA datasets, where it outperforms the strongest competitors. On WebQSP, its retriever balances answer coverage and compactness and also adds useful evidence beyond the PCST candidate subgraph.

4.1 Experiment Setup

  • Experiments use ExplaGraphs, WebQSP, and CWQ, with Accuracy for ExplaGraphs and Hit@1 for WebQSP and CWQ.For WebQSP and CWQ, the pipeline first forms a PCST candidate subgraph with kv=ke=10; the backbone is LLaMA2-Chat-7B for both retriever and generator.Source: §4.1 · p. 5

4.2 Benchmark results

  • 91.79% ExplaGraphs Accuracy, 76.97% WebQSP Hit@1, and 55.71% CWQ Hit@1 exceed the strongest baselines by 4.74%, 3.18%, and 6.52%, respectively.TaLenT also improves over the strongest inference-only baseline, StructGPT, by 13.45%, 4.37%, and 23.14% on those datasets, respectively.Source: §4.2 · pp. 5–6

4.3 Coverage-Compactness Tradeoff

  • On WebQSP, TaLenT averages 12.79 retrieved nodes versus 31.75 for PCST(kv=ke=10), with comparable Hit@1 and Recall but higher precision and F1.Precision rises from 9.26% to 24.55% and F1 from 12.22% to 26.63%.Source: §4.3 · p. 6
  • At a similar node budget, TaLenT improves WebQSP Hit@1 from 61.92% to 80.28% and Recall from 51.76% to 67.84% over PCST(kv=ke=3).Figure 3 compares gold-answer coverage for the two PCST subgraphs and TaLenT’s selected nodes; TaLenT keeps coverage close to PCST(kv=ke=10) using far fewer nodes.Source: §4.3 · p. 6, Figure 3 · p. 6

4.4 Retrieval Beyond PCST

  • Beyond-PCST nodes alone achieve 10.26% Hit@1 and 5.46% Recall, and adding them raises WebQSP Hit@1 from 75.31% to 80.28% and Recall from 64.19% to 67.84%.The analysis attributes the added gold-node coverage to parametric-knowledge nodes retrieved by the trained LLM.Source: §4.4 · p. 7

5 Ablation Study

WebQSP ablations show that pseudo-label training improves retrieval quality and downstream QA, while removing the learnable retriever also reduces QA performance.

5.1 On the Effect of Pseudo-label Training

  • Training the retriever with pseudo-labels raises WebQSP QA Hit@1 from 68.37% to 76.97% versus a frozen retriever.The trained retriever also improves retrieval Hit@1, precision, and recall over the frozen version: 80.28%, 24.55%, and 67.84% versus 36.61%, 14.97%, and 26.26%.Source: §5.1 · p. 7

5.2 On the Effect of Learnable Retriever

  • Removing the learnable retriever lowers WebQSP QA Hit@1 from 76.97% to 72.50%; using a frozen retriever yields 68.37%.The frozen retriever performs worse than feeding the PCST candidate subgraph directly to the generator.Source: §5.2 · p. 7

6 Related Work

The related work surveys retrieval-augmented generation and graph-based methods that combine graph information with LLMs, including GraphRAG and GraphQA.

  • RAG work includes query transformation, expansion, and rewriting before retrieval, reranking afterward, and LLM-based iterative refinement of queries or evidence selection.The review also covers work combining graph-structured information with LLMs, including graph reasoning, node prediction, graph-level prediction, and GraphRAG/GraphQA.Source: §6 · p. 7

7 Conclusion

TaLenT combines a learnable LLM retriever and a fine-tuned LLM generator, training evidence selection with gold-answer-guided weighted pseudo-labels to improve coverage while suppressing noise. The paper evaluates it on ExplaGraphs, WebQSP, and CWQ, and reports consistency checks for ExplaGraphs pseudo-gold reasoning-node annotations.

7 Conclusion

  • TaLenT addresses architectural and retrieval-supervision gaps with a learnable LLM retriever, fine-tuned LLM generator, and gold-answer-guided weighted pseudo-label loss.The loss prioritizes answer-related signals for coverage while suppressing noise for compactness; experiments on three GraphQA benchmarks show consistent downstream QA improvements.Source: §7 · p. 7

A.1 Datasets

  • ExplaGraphs supplies only binary stance labels, so LLaMA2-Chat-7B generates pseudo-gold reasoning-node sets for retriever training; inference requires no node annotations.The generator remains fine-tuned on the original support-versus-counter task supervision.Source: §A.1 · p. 10, §A.2 · p. 10

A.2 Pseudo-Gold Nodes Annotation for ExplaGraphs

  • Two ExplaGraphs annotators share at least one selected node in 80.44% of instances, and GPT-4o-mini covers 83.75% of LLaMA-2-7b-chat nodes on average.Manual inspection also found the selected nodes broadly consistent with human judgment.Source: §A.2 · p. 10

B Implementation Details

Implementation uses dataset-specific candidate contexts, then fine-tunes the retriever and generator with LoRA. The retriever’s node targets distinguish gold-answer nodes from other candidates.

B Implementation Details

  • For WebQSP and CWQ, PCST constructs candidate subgraphs with kv=ke=10; ExplaGraphs uses the full graph as its candidate context.Source: B Implementation Details · p. 10
  • Retriever fine-tuning uses LoRA and token-weighted cross-entropy, assigning gold-answer-overlapping nodes weight 1.0 and other candidates λ=0.001 for WebQSP or λ=0.01 for ExplaGraphs and CWQ.Targets list strong positives before weak positives, separated by |.Source: B Implementation Details · p. 11
  • The generator is fine-tuned with LoRA using supervised training on retrieved graph contexts and gold targets; both models use rank r=8, α=16, and dropout 0.05.Retriever training uses three epochs, a 5 × 10−5 learning rate, and gradient accumulation of 8.Source: B Implementation Details · p. 11

C Sensitivity Analysis and Ablation of the Weighted Objective

On WebQSP, smaller weak-positive weights favor compact retrieval without sacrificing downstream performance, while larger PCST candidate pools improve Hit@1. Equal weighting weak and strong positives lowers Hit@1 and expands retrieved sets.

Sensitivity to the weak-positive weight λ

Sensitivity to PCST candidate subgraph size

D Discussion of Answer-Aware or Answer-Related Methods and Additional Recent Baselines

TaLenT combines a learnable LLM retriever, a fine-tuned generator, and weighted pseudo-label supervision; the discussion contrasts this design with methods aimed at different retrieval targets or task settings. The prompts specify node selection and answer generation, while the generator context can include isolated retrieved nodes.

D Discussion of Answer-Aware or Answer-Related Methods and Additional Recent Baselines

  • QA-GNN scores candidate answers in multiple-choice QA, while GNN-RAG uses answer-candidate scoring and path retrieval for KGQA; neither uses TaLenT’s trainable LLM retriever and coverage-compactness objective.GNN-RAG is not directly applicable to label-prediction tasks such as ExplaGraphs.Source: D Discussion of Answer-Aware or… · p. 12
  • RoG retrieves KG reasoning paths from generated plans, whereas CG-RAG retrieves chunks over citation graphs and SentGraph retrieves sentences over document graphs; these differ from node-level selection on a given QA graph.RoG is designed for entity-answer KGQA; SentGraph differs in graph granularity, retrieval target, and benchmark formulation.Source: D Discussion of Answer-Aware or… · p. 12
  • TaLenT’s distinguishing combination is a unified LLM-based retriever-generator pipeline with weighted pseudo-label supervision, targeting coverage of answer-relevant evidence while suppressing noise across KGQA and label-prediction GraphQA.Source: D Discussion of Answer-Aware or… · p. 12

E Concrete Prompt Templates and Graph Serialization

E.1 Retriever Prompt

  • The retriever prompt supplies the question and initial PCST subgraph and requires a single lowercase, pipe-separated line of selected candidate entities.Source: §E.1 · p. 13

E.2 Generator Prompt and Graph Serialization

  • For answer generation, nodes are serialized before their associated edges; retrieved nodes outside the initial candidate subgraph remain in context as isolated nodes.The generator prompt asks for final answer entities in a single lowercase, pipe-separated line.Source: §E.2 · p. 14

F Groundedness and Hallucination Analysis of Out-of-PCST Retrieval

The analysis evaluates whether retrieved nodes outside the initial PCST subgraph introduce ungrounded evidence, using a hit-aware proxy on WebQSP. The reported proxy rate is low, alongside complementary coverage from out-of-PCST retrieval.

F Groundedness and Hallucination Analysis of Out-of-PCST Retrieval

  • 4.97%: the WebQSP test-set hallucination rate under the authors’ hit-aware proxy.The proxy counts out-of-PCST retrieved nodes that are not gold answers only when the final prediction is incorrect.Source: F Groundedness and Hallucination… · p. 14
  • TaLenT’s learned retriever maintains comparable gold-answer coverage with substantially fewer nodes than the kv=ke=10 PCST candidate, and higher coverage than the size-matched kv=ke=5 PCST subgraph.Figure 4 compares answer-set coverage for the learned retriever and the two PCST references on WebQSP with Llama-3-8B.Source: F Groundedness and Hallucination… · p. 15, Figure 4 · p. 15
  • The authors interpret out-of-PCST retrieval as providing useful complementary coverage while maintaining a low level of hallucinated evidence.Source: F Groundedness and Hallucination… · pp. 14–15

G Backbone Ablation with Llama-3-8B

With Llama-3-8B on WebQSP, TaLenT improves over G-Retriever w/ LoRA and maintains a favorable retrieval coverage–compactness tradeoff. The related work section also situates GraphRAG/GraphQA among graph–LLM applications and describes PEFT methods used in graph+LLM settings.

G Backbone Ablation with Llama-3-8B

  • 83.05% Hit@1 versus 79.36% for G-Retriever w/ LoRA: TaLenT remains effective with the Llama-3-8B backbone.The authors characterize its downstream QA performance as strong with this backbone.Source: G Backbone Ablation with Llama-3-8B · p. 15
  • 31.75 to 19.60 nodes: TaLenT produces a more compact set with comparable gold-answer coverage against the initial PCST subgraph.Hit@1 rises from 78.93% to 81.45%, Recall from 69.76% to 70.40%, Precision from 9.26% to 22.16%, and F1 from 12.22% to 23.54%.Source: G Backbone Ablation with Llama-3-8B · p. 15
  • 8.29% Hit@1 and 4.28% Recall come from nodes outside PCST; including them raises Hit@1 from 76.90% to 81.45% and Recall from 67.35% to 70.40%.Table 10 reports macro-averaged percentages for gold-answer coverage on the WebQSP test set with Llama-3-8B.Source: G Backbone Ablation with Llama-3-8B · p. 15, Table 10 · p. 15

H More Related Works

  • Related work spans general graph modeling, multimodal graph-like architectures, graph reasoning and prediction, and GraphRAG/GraphQA; PEFT methods have also been explored in graph+LLM settings.Examples include prompt tuning, prefix tuning, LoRA, GraphLLM, GraphToken, and GNP.Source: H More Related Works · pp. 15–16

Related papers

Loading 2610.33511v1…